Mixtral 8x7B

What it is

Mixtral 8x7B is an open-weight language model from Mistral AI built on a sparse mixture-of-experts (SMoE) design. Rather than passing every token through the whole network, it sends each token to a small subset of its parameters, so the computation per token is far smaller than the total size of the model would suggest.^[1]

The weights were first shared through a BitTorrent link on 9 December 2023, and the model was formally announced on 11 December 2023.^[1][3]

How it works

The idea behind mixture-of-experts models is to keep several alternative sub-networks, called experts, inside the network and use only a few of them for each input. That raises the number of parameters without raising the amount of arithmetic per input by the same factor.

Mixtral 8x7B follows the architecture of Mistral 7B with one change: every layer holds 8 feed-forward experts instead of a single feed-forward block.^[2] A router network chooses two of those experts for each token, and their outputs are combined additively.^[1] Because the choice is made separately at every layer, the same token can be handled by a different pair of experts from layer to layer.^[2] In Mistral's own description, the feed-forward block picks from 8 distinct groups of parameters and the router selects two of them per token.^[1]

This separates total size from per-token cost. Mixtral 8x7B has 46.7 billion parameters in total, but only 12.9 billion are used for any single token.^[1] All the weights still have to be stored; what shrinks is the arithmetic done for each token. Mistral reported that this made inference about 6 times faster than Llama 2 70B.^[1]

What it can do

The model handles a context of 32,000 tokens and works in English, French, Italian, German and Spanish.^[1]

On quality, Mistral reported that Mixtral 8x7B outperforms Llama 2 70B on most benchmarks and matches or beats GPT-3.5 on most standard benchmarks.^[1] The instruction-tuned variant, Mixtral 8x7B Instruct, scored 8.30 on MT-Bench, which Mistral called the best result for an open-source model at the time.^[1]

Release and licensing

Both the base model and the Instruct model were released under the Apache 2.0 licence.^[1] Mistral served the model through its mistral-small API endpoint and contributed support for it to the vLLM inference project, integrating Megablocks CUDA kernels for efficient inference.^[1]

The paper and afterwards

A technical paper, "Mixtral of Experts", was posted to arXiv on 8 January 2024. It lists 26 authors, among them Albert Q. Jiang as first author and Arthur Mensch as fourth, with Alexandre Sablayrolles and Antoine Roux also among the authors.^[2]

A larger version of the same idea, Mixtral 8x22B, followed in April 2024.^[4] The hosted API model open-mixtral-8x7b was deprecated on 30 November 2024 and retired on 30 March 2025.^[4] The base and Instruct weights remain published under Apache 2.0.^[1]