Microsoft Research publishes phi-1 and "Textbooks Are All You Need"

Background

Before this, many code models were trained on very large amounts of raw code and web text. Microsoft Research asked whether a much smaller model could match them if the training data were chosen more carefully.

What happened

On 20 June 2023 Microsoft researchers posted "Textbooks Are All You Need" to arXiv. The paper introduced phi-1, a 1.3-billion-parameter model for code.^[1][2]

The model was trained on about 7 billion tokens of "textbook quality" data. This included filtered web code and textbooks and exercises generated with GPT-3.5.^[1]

Training took four days on eight A100 GPUs.^[1]

Key details

phi-1 reached 50.6% pass@1 on HumanEval and 55.5% on MBPP, beating far larger open models.^[1]

The paper argued that high-quality data can change the shape of scaling laws. Under this view, small models can match much larger ones.^[1]

Weights were later published on Hugging Face under the MIT licence.^[2]

Impact

The paper set out the "textbook" data approach that Microsoft carried into the Phi family. It shifted attention in small-model research from size alone to the quality of training data, and phi-1.5 followed in September 2023.^[1][2]

What "textbook quality" means

"Textbook quality" data here means text that is clear, well-organised and explains concepts step by step, in the manner of a teaching book, rather than raw text scraped from the web, which may be repetitive, incomplete or wrong. The authors filtered web code for educational value and wrote synthetic lessons and exercises for the rest. The idea is that a model learns more efficiently from examples chosen for clarity.^[1][2]

Why the result drew attention

A model with 1.3 billion parameters is small compared with the systems that dominated attention in 2023. Showing that it could beat larger open code models on two standard benchmarks suggested that the amount of training data and parameters was not the only path to better code generation. The paper's claim about scaling laws, that the shape of the relation between size and performance could change with data quality, was the more ambitious point.^[1]

Practical use

Because the weights were released openly, researchers and students could download and run the model themselves, study its behaviour and compare it with other code models. This openness is one reason the paper became a reference point in later small-model research.^[2]