Phi-4-mini and Phi-4-multimodal

What was released

On 26 February 2025, two small models joined the Phi-4 family: Phi-4-mini, a 3.8B-parameter text model, and Phi-4-multimodal, a 5.6B-parameter model.^[1][3] The multimodal version extends Phi-4-mini so that a single model handles text, images and speech or audio, rather than separating those jobs into different systems.^[1][3]

Phi-4-mini takes the place of Phi-3.5-mini in the small tier of the family.^[2]

Architecture and training

Phi-4-mini is a dense decoder-only Transformer. Compared with Phi-3.5-mini it adds a vocabulary of 200,064 tokens — the "200K" vocabulary the model card highlights — together with grouped-query attention and shared input and output embeddings.^[2] Grouped-query attention is an arrangement in which several query heads share one set of key and value projections, which reduces the memory and computation spent on attention at long sequence lengths. The larger vocabulary is the change most directly tied to language coverage: the model supports 23 languages.^[2]

The context window is 128K tokens.^[2] Training used 5 trillion tokens on 512 A100-80G GPUs over 21 days in November–December 2024, with a public data cutoff of June 2024.^[2]

After pre-training the model was enhanced with supervised fine-tuning and direct preference optimisation for instruction adherence and safety.^[2] Direct preference optimisation is a post-training method that adjusts a model toward responses people preferred over alternatives. The release notes describe three changes relative to the previous small model: a new architecture for efficiency, a larger vocabulary for multilingual support, and improved post-training.^[2]

Benchmarks

On the numbers published with the model card, Phi-4-mini scores 67.3 on MMLU, 88.6 on GSM8K and 64.0 on MATH; the same card lists GPT-4o-mini at 77.2 on MMLU.^[2] MMLU is a broad multiple-choice test of knowledge across subjects, GSM8K a set of grade-school maths word problems, and MATH a harder competition-style problem set. The pattern is a model that is deliberately small and, in these measurements, behind a larger competitor on broad knowledge while posting high scores on the maths sets.

The model card states the intended uses: memory- and compute-constrained environments, latency-bound scenarios, and tasks that need strong reasoning, especially in maths and logic.^[2]

Availability

Both models were released on Hugging Face under the MIT licence, and offered through Azure AI Foundry, the NVIDIA API Catalog and GitHub Models.^[3][4] The MIT licence allows reuse and modification with few restrictions, which matters for a model aimed at developers who want to run it themselves on modest hardware.

Where it sits

Phi-4-mini and Phi-4-multimodal sit at the small end of a family whose larger members carry the Phi-4 name. The design choices point in one direction: a compact dense model with a long context window, a large multilingual vocabulary and heavier post-training, paired with a multimodal variant built on top of it. For developers, the pair covers two common needs at once — text-only inference where memory and latency are tight, and a single model that can take an image or a spoken input without adding separate components to the pipeline.^[1][2][3]