Microsoft releases Phi-3.5 mini, MoE and vision

Background

Phi-3.5 was an update to Microsoft's Phi family of small language models. The release added three models and documented them in the Phi-3 technical report.^[1][3]

What happened

In August 2024 Microsoft released three Phi-3.5 models: Phi-3.5-mini-instruct, Phi-3.5-MoE-instruct and Phi-3.5-vision-instruct.^[1][2] The "instruct" suffix marks versions tuned to follow instructions. All three support a 128K-token context, and all three were released under the MIT licence.^[1][2]

Key details

Phi-3.5-MoE combines 16 experts for a total of 42 billion parameters, with 6.6 billion active for each token.^[2] A mixture-of-experts model routes each token to a small number of specialised sub-networks, so only part of the model runs for any one token.

Phi-3.5-MoE was trained on 4.9 trillion tokens on 512 H100 GPUs over 23 days.^[2] Microsoft compared the model with Gemini 1.5 Flash and GPT-4o-mini.^[3]

Phi-3.5-vision is a 4.2-billion-parameter model for understanding both single images and multiple images in one prompt.^[3]

Impact

The release gave developers three new options in one step: a larger model that uses experts to keep per-token computation low, a small text model, and a vision model that reads several images at once. All three shared the MIT licence and the 128K context.^[1][2][3]

Terms used here

An instruct model is a version of a base model that has been tuned to follow instructions and answer questions, rather than only to continue text. A mixture-of-experts model splits its capacity across several expert sub-networks and activates only some of them for each token, so the computation per token is lower than the total parameter count suggests. A context of 128K tokens means the model can take in very long inputs, such as a book-length document, in one request. Vision models read images as well as text. The MIT licence allows free use and redistribution of the models.