Phi-1.5

What it is

Phi-1.5 is a 1.3-billion-parameter language model from Microsoft Research, described in the technical report "Textbooks Are All You Need II: phi-1.5 technical report" (arXiv:2309.05463), submitted on 11 September 2023^[1]. Its authors are Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar and Yin Tat Lee^[1]. At 1.3 billion parameters it sits at the small end of the scale, and the report's central argument is precisely that a model of this size can hold its own on everyday language tasks.

Where the idea came from

Phi-1.5 continues an approach first tried on code. Its predecessor phi-1 was trained on text that had been selected and generated to resemble textbook material, and phi-1.5 extends that textbook-data approach from code to common-sense reasoning in natural language^[1]. The bet behind it is about data quality: carefully written synthetic prose can carry more useful signal per token than raw web text, so a small model can be trained on fewer but better tokens.

How it was trained

The training set was a 30-billion-token dataset built from phi-1's sources together with new synthetic natural-language texts, with generic web-crawl sources such as Common Crawl left out^[2]. Over the course of training the model saw 150 billion tokens^[2]. The run took 8 days on 32 A100-40G GPUs, using PyTorch, DeepSpeed and Flash-Attention^[2].

Excluding the web crawls is the distinctive part of the recipe. The model's picture of the world comes from curated and generated text rather than from a broad scrape of the internet^[2].

What the report claims

The report claims natural-language performance comparable to models five times larger^[1]. The model card describes nearly state-of-the-art performance among models under 10 billion parameters on benchmarks for common sense, language understanding and logical reasoning^[2]. The authors also say the model shows traits usually associated with bigger systems, such as step-by-step reasoning^[1].

The point of the exercise is less a leaderboard record than a question about where ability comes from: how much of it is scale, and how much is the text a model reads. If a 1.3-billion-parameter model trained on curated synthetic text can reach the level of models five times larger, then data design becomes a lever that can partly stand in for parameter count^[1].

Limitations and design choices

The same authors note that the model hallucinates and can produce toxic or biased output, and they say this is reduced by the absence of web data^[1]. Phi-1.5 was not fine-tuned for instruction following, and no reinforcement learning from human feedback was applied; according to the model card this was deliberate, so that researchers would have an unrestricted small model to work with on safety questions^[2]. Because it is a base model rather than a tuned assistant, it continues text in the manner of its prompt rather than being optimised to answer requests.

Release and place in the series

Phi-1.5 was released openly on Hugging Face, later under the MIT licence^[1][2]. It was integrated into version 4.37.0 of the transformers library, and its model card lists poem writing, emails, stories, summarisation and Python code among its uses^[2]. It is the successor of phi-1 and the predecessor of phi-2, which arrived in December 2023^[1].