Phi-3.5

What Phi-3.5 is

Phi-3.5 is an update to the Phi family of small language models from Microsoft, released in August 2024. It is not one model but a set of three — Phi-3.5-mini, Phi-3.5-MoE and Phi-3.5-vision — and their documentation was added to the existing Phi-3 technical report rather than published as a separate paper.^[1][2][3]

The three cover different needs. Phi-3.5-mini is the general-purpose small member, with 3.8 billion parameters and a 128K-token context window. Phi-3.5-vision is derived from Phi-3.5-mini, has 4.2 billion parameters, and handles prompts containing a single image or several images.^[1][3] Phi-3.5-MoE is the largest and the most unusual of the three: it is Microsoft's first mixture-of-experts model in the Phi line, built from 16 experts of 3.8 billion parameters each, for 42 billion parameters in total, of which 2 experts — 6.6 billion parameters — are active per token.^[1][2]

Why mixture-of-experts

A mixture-of-experts (MoE) model replaces a single feed-forward block with several parallel "experts" plus a router that decides, token by token, which experts should handle it. Only the selected experts are computed, so the amount of arithmetic per token stays close to that of a much smaller dense model, while the total number of stored parameters — and the capacity those parameters give — is much larger. Phi-3.5-MoE follows this pattern: out of 16 experts, 2 are active for each token, so 42 billion parameters are stored and 6.6 billion are used per token.^[2] In practice this is a way to buy capacity without paying the full compute cost of a dense model of the same total size.

Training and languages

The Phi-3.5-MoE model card gives unusually specific training details: the model was trained on 4.9 trillion tokens, about 10% of them multilingual, on 512 H100-80G GPUs over 23 days between April and August 2024. It has a 128K-token context window and a 32,064-token vocabulary, and supports 23 languages, among them Arabic, Chinese, Japanese, Russian and Spanish.^[2]

Benchmarks

The published scores for Phi-3.5-MoE are 78.9 on MMLU (5-shot), 88.7 on GSM8K, 59.5 on MATH and 70.7 on HumanEval.^[2] The technical report states that the model beats similar-scale open models such as Llama 3.1 and Mixtral on reasoning, mathematics and code, and is on par with Gemini 1.5 Flash and GPT-4o-mini.^[1] Those are the authors' own comparisons, made against models of roughly the same scale; they describe where the model sits relative to its contemporaries rather than a general ranking.

Tuning, prompt format and availability

According to the model card, Phi-3.5-MoE was enhanced with supervised fine-tuning, proximal policy optimisation (PPO) and direct preference optimisation (DPO) for instruction adherence and safety.^[2] It is best suited to chat-format prompts built from system, user and assistant roles.^[2] The instruct version was integrated into the transformers library from version 4.46.0 and is also offered in Azure AI Studio.^[2] All three Phi-3.5 models were released under the MIT licence, which permits commercial and non-commercial reuse.^[2][3]

Where it sits in the Phi line

Phi-3.5 is the successor of Phi-3 and was itself succeeded by Phi-4 in December 2024, a cadence of roughly a few months between generations.^[1] Its interest for readers is mostly architectural and practical: a small family that pairs compact dense models, one of them able to take images, with an MoE model whose active parameter count is a fraction of its stored total, all shipped openly enough to be run and inspected.^[1][2][3]