Phi-1

What it is

Phi-1 is a 1.3-billion-parameter Transformer language model built by Microsoft Research for writing Python code^[1]. It was described in a paper titled "Textbooks Are All You Need" (arXiv:2306.11644), posted on 20 June 2023, and its weights were published on Hugging Face under the MIT licence^[2]. Its scope is deliberately narrow: it is aimed at Python rather than at programming languages in general^[2]. A model of just over a billion parameters is small next to general-purpose chat models; what made Phi-1 notable was its argument about how such a small model should be trained.

Where it came from: data as the variable

The paper's central claim is about training data rather than architecture. Code models of the time were typically trained on large web crawls — a mixture of documentation, forum threads, tutorials, half-finished snippets and noise. The authors argued instead for "textbook quality" data: text written like a careful lesson that introduces an idea and then exercises it. Their conclusion was that data quality can substitute for model and data scale^[1], that is, a small model fed very clean material can compete with far larger models fed far more material.

The recipe mixed two sources. About 6 billion tokens came from filtered code-and-language web data, drawn from The Stack and from StackOverflow. About 1 billion tokens were synthetic: Python "textbooks" and exercises generated with GPT-3.5^[1][2]. The synthetic part is what gives the paper its title — a larger model was used to write teaching material for a smaller one.

The token accounting is easy to misread, because two different numbers appear. The model card lists 54 billion training tokens in total, of which 7 billion are unique^[2]: the model went over the same material several times, so the total counts repeated passes. Training took about 4 days on 8 A100 GPUs^[1].

Results

The standard way to measure a code model on HumanEval is to give it a function signature and docstring and let it write the body; the generated code is then run against unit tests, and pass@1 is the share of problems solved by the first attempt. MBPP works similarly with short standalone problems. Phi-1 scored 50.6% pass@1 on HumanEval and 55.5% on MBPP^[1]. The authors reported that it outperformed other open code models on both benchmarks, with only GPT-4 better overall^[1].

A second result made the same point about scale. A 350-million-parameter variant, phi-1-small, trained with the same pipeline, reached 45% on HumanEval^[1] — a quarter of the parameters, a score in the same range.

Limits

Microsoft's model card describes Phi-1 as best suited to code-format prompts and warns that generated code should be treated as a starting point rather than a definitive solution^[2]. Direct adoption for production coding tasks was declared out of scope, and the model was not tested for production-level code^[2]. The Python-only focus follows from the training data and narrows the range of tasks further.

Place in the Phi family

Phi-1 is the first model of the Phi family of small language models, and it was succeeded by phi-1.5 in September 2023^[3]. It was integrated into version 4.37.0 of the transformers library^[2], which put it in front of developers already using that library. The model's main legacy is the argument in its paper: that curating what a model reads, including generating teaching material for it, can matter as much as making the model bigger.