Mistral AI releases Mixtral 8x22B

What was released

On 17 April 2024, Mistral AI announced Mixtral 8x22B, a large language model built on a sparse mixture-of-experts (MoE) architecture. It has 141 billion parameters in total, of which 39 billion are active at any one time.^[1] The public record carries two slightly different dates: the announcement is dated 17 April 2024, while Wikipedia records the release of the model weights as 10 April 2024, a week earlier.^[2]

How a sparse mixture of experts works

A mixture-of-experts model is not one solid block of computation. The network is divided into many separate expert sub-networks, and for each token a routing step selects only a small number of them to run; the rest stay idle. Because only part of the network fires, the total parameter count — the model's overall capacity — can be much larger than the amount of computation spent on any single token. The active-parameter figure is the one that governs cost and speed: 39 billion active parameters means each token is processed with roughly the arithmetic of a dense model of that size, not of a 141-billion-parameter one. A name like "8x22B" describes how the experts are arranged, not how much compute one token consumes. Mistral said the model's sparse activation patterns make it faster than any dense 70B model.^[1]

Context, tools and licence

Mixtral 8x22B comes with a 64K context window, so a long document can be passed in at once rather than chopped into fragments.^[1] It also has native function calling: the model can be given descriptions of external functions or tools and return structured calls to them, which an application then wires to search, databases or other software.^[1] The weights were released under the Apache 2.0 licence,^[1] a permissive open-source licence that allows use, modification and redistribution, including in commercial products, provided the copyright and licence notices are kept.

Benchmark results

For the instructed version, Mistral reported 90.8% on GSM8K with maj@8 and 44.6% on MATH with maj@4.^[1] Both are mathematics reasoning benchmarks, and the maj@k notation means the model is sampled several times on the same question and the majority answer is taken — a self-consistency decoding trick that usually lifts scores. Mistral said the model strongly outperforms LLaMA 2 70B on HellaSwag, ARC Challenge and MMLU, three widely used benchmarks,^[1] and described it as setting a new standard for performance and efficiency within the AI community.^[1] These figures and comparisons are the publisher's own reported numbers rather than a third-party evaluation.

Why it mattered

The release put three things together: openly licensed weights, a total parameter count far above the dense models commonly run at the time, and a much smaller active count that keeps inference cheap. On Mistral's account, that combination bought better scores than a dense 70B model on several benchmarks while running faster.^[1] For developers outside the company, Apache 2.0 meant the weights could be downloaded, modified and deployed in commercial products without a separate agreement, and the 141-billion-total / 39-billion-active split offered a worked example of how a sparse mixture-of-experts model can scale capacity while holding per-token compute down.^[1]