Phi-2

What Phi-2 is

Phi-2 is a 2.7-billion-parameter language model that Microsoft Research announced on 12 December 2023, in a blog post by Mojan Javaheripi and Sébastien Bubeck^[1]. Parameters are the learned numbers inside a neural network; at 2.7 billion, the model sits far below the largest systems of its time, small enough to be run and fine-tuned on modest hardware. Phi-2 is a base model: it was not put through reinforcement learning from human feedback (RLHF), the alignment step that turns a raw text predictor into a chat assistant, so it continues text rather than following instructions out of the box^[1].

Where it came from

The model belongs to Microsoft's line of deliberately small language models trained on carefully chosen data rather than on sheer volume. Microsoft said its earlier 1.3-billion-parameter Phi-1 had reached state-of-the-art Python coding performance among small language models on the HumanEval and MBPP benchmarks^[1]. Phi-2 was built by scaling up: the knowledge of the 1.3-billion-parameter Phi-1.5 was embedded into the larger 2.7-billion-parameter model^[1]. Satya Nadella announced Phi-2 at Microsoft Ignite 2023^[1].

How it was trained

The training mixture combined synthetic datasets for common sense and general knowledge with web data that had been filtered for educational value^[1][2]. The training corpus was 250 billion tokens, and the model saw 1.4 trillion tokens over the course of training — more tokens than the corpus itself contains^[1][2]. Training ran for 14 days on 96 A100-80G GPUs, and the model's context length was 2,048 tokens^[2]. The emphasis on filtered and generated data, rather than on a very large crawl of raw web text, is the central design choice behind the model: the claim being tested was that data quality can substitute for some of the scale that larger models buy with compute.

Results

Microsoft said Phi-2 matched or outperformed models up to 25 times larger on complex benchmarks, and described it as the best base model under 13 billion parameters^[1]. On aggregated benchmarks, Microsoft reported that it surpassed Mistral 7B and Llama 2 7B and 13B^[1]. Because it is not aligned, questions about how it behaves in open conversation are separate from its benchmark scores: Microsoft said it nonetheless showed better toxicity and bias behaviour than comparable aligned open models^[1].

Release and positioning

The model was released through the Azure AI Studio model catalog and on Hugging Face, and the Hugging Face model card lists the MIT licence^[1][2]. Microsoft positioned Phi-2 as an ideal playground for researchers working on mechanistic interpretability, safety improvements and fine-tuning experiments^[1] — that is, as a small, permissive, easily inspectable model on which experiments are cheap, rather than as a product to be deployed as an assistant.

Place in the line

The recipe did not stop at 2.7 billion parameters. Its training data was scaled up again to build phi-3-mini in April 2024^[3]. Phi-2 therefore marks the point in the series where the small-model approach, first tried on coding with Phi-1, was shown to generalise to broad language benchmarks and to attract attention outside Microsoft.