Microsoft introduces Phi-4

Background

Phi-4 belongs to Microsoft's Phi line of small language models, and its release focused on complex reasoning.^[1][2]

What happened

On 12 December 2024 Microsoft released Phi-4, a 14-billion-parameter small language model specialised in complex reasoning such as mathematics.^[1][2] The model was first available on Azure AI Foundry. Microsoft then released the weights on Hugging Face on 8 January 2025 under the MIT licence.^[3]

Key details

The technical report credits synthetic data and careful data curation for scores above GPT-4o on the GPQA and MATH benchmarks.^[2] The model was trained on 9.8 trillion tokens using 1,920 H100 GPUs over 21 days.^[1] Its model card reports 84.8 on MMLU and 80.4 on MATH.^[1] It has a 16K-token context and is a dense decoder-only Transformer, meaning all of its parameters are used for each token.^[1]

GPQA is a set of graduate-level science questions, and MATH is a set of competition-style maths problems, so the comparison with GPT-4o concerns the two tests where the report said Phi-4 was ahead.^[2]

Impact

Phi-4 showed that a 14-billion-parameter model could be offered as a reasoning-focused open model, with its weights later published for download.

Terms used here

A dense model uses all of its parameters for every token, whereas a mixture-of-experts model activates only part of its network. GPQA is a set of hard, graduate-level science questions. MATH is a collection of competition-style mathematics problems. MMLU covers many school and professional subjects. Azure AI Foundry is Microsoft's platform for building and running AI applications and models. Open weights means that the trained parameters are published so that anyone can download the model and run it on their own hardware, subject to the licence terms.

The term small language model refers to a model with far fewer parameters than the largest systems, which can be run at lower cost. Benchmarks are standard tests that score models on the same questions so that they can be compared.