Microsoft releases Phi-2

Background

Microsoft's Phi series is a line of small language models, with far fewer parameters than the largest systems. Phi-2 came out of earlier work on phi-1.5, a 1.3-billion-parameter model whose knowledge Microsoft carried into the larger Phi-2.^[1]

What happened

On 12 December 2023 Microsoft Research announced Phi-2, a 2.7-billion-parameter language model.^[1] Microsoft said Phi-2 matched or outperformed models up to 25 times larger and presented it as a playground for researchers who want to study how small models behave.^[1] The company made it available in the Azure AI Studio model catalog, so developers could use it through Microsoft's cloud service.^[1]

Key details

Phi-2 was trained on 1.4 trillion tokens of synthetic data and filtered web data. It took 14 days on 96 A100 GPUs.^[2] A token is a chunk of text, often part of a word, and the GPU is the kind of chip that does most of the arithmetic in training a model.

Microsoft reported that Phi-2 beat Mistral 7B and Llama 2 13B on several aggregated benchmarks, which combine the scores from multiple tests into one comparison.^[1] The Hugging Face release carries the MIT licence, which allows others to use, change and redistribute the model.^[2]

Impact

Phi-2 showed that a model with 2.7 billion parameters could be placed in front of researchers and developers as a serious comparison point against models several times larger. Its public release on Azure and Hugging Face made it available for others to test directly.^[1][2]

Terms used here

Azure AI Studio is Microsoft's cloud platform for building AI applications, and its model catalog lists models that developers can select and deploy from one place. Hugging Face is an online hub where researchers and developers publish model weights, code and documentation. Each model page there is called a model card, and it describes what the model is for, how it was trained and how it was evaluated. The MIT licence is a short permissive open-source licence: anyone may use, copy, modify and share the software as long as the licence text and copyright notice are kept.