Mistral AI releases Mixtral 8x7B, an open mixture-of-experts model

Background

Most large language models of the time were dense: every parameter in the network took part in processing every token of text. Making a dense model more capable generally meant making it bigger, and running it became more expensive too.

A mixture-of-experts (MoE) design offers another route. Instead of one large block of computation, the model contains several smaller specialist sub-networks, called experts, and a small router decides which experts should handle each piece of input. The total number of parameters can be large while the work done for any single token stays much smaller. The idea was not new; what this release added was its use in a language model whose weights were openly available.

What happened

On 11 December 2023 Mistral AI released Mixtral 8x7B, a sparse mixture-of-experts (SMoE) model with open weights.^[1] "Sparse" means that only part of the model is active for any given token.

Key details

In Mixtral, each layer contains eight groups of parameters, the experts. For every token, at each layer, a router network picks two of them to process it, and their outputs are combined.^[1] The model has 46.7 billion parameters in total but uses only 12.9 billion of them per token, so its per-token compute resembles a much smaller model while its total capacity is larger.^[1]

Mixtral handles a context of 32k tokens, meaning the amount of text it can take into account at once, and works in English, French, Italian, German and Spanish.^[1] It is licensed under Apache 2.0, a permissive open-source license that allows broad reuse, including commercial use.^[1]

Mistral said Mixtral outperforms Llama 2 70B on most benchmarks while running inference six times faster, and that it matches or outperforms GPT-3.5 on most standard benchmarks.^[1] These are the company's own comparisons, and independent testing is needed to confirm them in other settings.

The instruction-tuned version, Mixtral 8x7B Instruct, scored 8.30 on MT-Bench, a test of multi-turn conversation quality. Mistral called this the best score for an open-source model at the time.^[1]

Mixtral also reached real products. It powered the "mistral-small" endpoint of Mistral's beta platform, and Mistral contributed changes to the open-source vLLM project to make inference more efficient.^[1]

Impact

Mixtral showed that a mixture-of-experts model could be released with open weights and a permissive license, and reach the level of GPT-3.5 by the company's own measures. Its practical appeal was that only a fraction of its parameters run for each token, so in principle it can offer high quality at lower serving cost than a dense model of similar total size.

The release also served as a public example of how a sparse design can be packaged for developers: an open license, a hosted endpoint, and a contribution to shared inference software. Readers should treat the performance comparisons as Mistral's reported results rather than settled rankings.