Microsoft releases phi-1.5

Background

phi-1 had shown in June 2023 that a small model trained on textbook-quality data could do well on code. The next question was whether the same approach worked for natural language.

What happened

On 11 September 2023 Microsoft researchers posted the phi-1.5 technical report, "Textbooks Are All You Need II". The report described phi-1.5 as a 1.3-billion-parameter model, the same size as phi-1, but aimed at common-sense reasoning in natural language.^[1]

The model was trained mostly on synthetic "textbook" data, for 150 billion tokens on 32 A100 GPUs over 8 days.^[2]

Key details

The report said phi-1.5 matched natural-language models five times its size.^[1]

Microsoft open-sourced the model to promote further research on issues such as hallucination, toxicity and bias in small models.^[1]

The model is available on Hugging Face under the MIT licence.^[2]

Impact

phi-1.5 showed that the textbook-data approach was not limited to code. By releasing it openly, Microsoft gave other researchers a small model to study the problems of hallucination, toxicity and bias.^[1][2]

Background in context

By 2023 the question of how much data a language model needs, and of what kind, was central to the field. phi-1 had shown that careful data selection could produce a strong code model. A natural question was whether the same method could help a model with ordinary language, which is the domain most users care about. The phi-1.5 report was an attempt to answer this question at a small scale.^[1]

What a technical report is

A technical report is a document in which a research group describes a model, how it was trained and how it performs, usually without the peer-review process of a journal. It is a primary source for the claims it makes, and readers should treat its results as the authors' own measurements. The phi-1.5 report is cited in this entry for that reason.^[1]

Why open release matters

Releasing the weights openly allows independent researchers to test the claims, reproduce the behaviour and study the problems the authors identified. In this case the authors named hallucination, toxicity and bias as the issues they wanted others to examine in small models.^[1][2]