Mixtral of Experts

The problem: capacity versus compute

A dense language model spends the same amount of computation on every token it processes: every parameter takes part in every forward pass, so capability and cost rise together, and a bigger model is also a slower and more expensive one. A sparse mixture-of-experts (MoE) design loosens that link. The network can hold far more parameters than it uses on any single token, because only a subset of its blocks runs for each token.

How Mixtral 8x7B is built

Mixtral 8x7B keeps the architecture of Mistral 7B and changes one component. Each layer is composed of 8 feedforward blocks, called experts, plus a router network that selects two experts per token at that layer^[1]. Only the selected experts run, so each token has access to 47B parameters in total, while inference uses 13B active parameters^[1]. Training was done with a context size of 32k tokens^[1].

To picture the routing, imagine a passage that mixes a line of code, a formula and words in two languages. The router judges each token separately, so the token for a variable name and the token for a verb need not be sent to the same pair of experts; over training, different experts come to handle different kinds of input. That is the appeal of the design: capacity grows with the number of experts, while the computation per token grows only with the number of experts actually selected. The trade-off is that all the parameters still have to be stored, even though only a fraction of them are used at any one step.

What the paper reports

Across all the benchmarks the authors evaluated, Mixtral outperforms or matches Llama 2 70B and GPT-3.5, and it vastly outperforms Llama 2 70B on mathematics, code generation and multilingual benchmarks^[1]. The comparison is notable because Mixtral does this with 13B active parameters per token^[1]. The instruction-following version, Mixtral 8x7B - Instruct, surpasses GPT-3.5 Turbo, Claude-2.1, Gemini Pro and Llama 2 70B chat on human benchmarks^[1].

Release

Both the base model and the instruct model were released under the Apache 2.0 license^[1], a permissive open-source license that allows reuse and modification. Releasing under that license means the models described in the paper are not only reported numbers: readers can obtain them, re-check the comparisons and build on them.

Authors and record

"Mixtral of Experts" was submitted to arXiv on January 8, 2024 by Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux and 23 other authors at Mistral AI^[1].

What to take away

Mixtral's concrete claim is that a model can hold 47B parameters while paying the inference cost of a 13B one^[1], and that on the benchmarks the authors chose this trade-off is competitive with a much larger model^[1]. For a reader following the field, the paper is also useful as a fixed reference point: it states what was measured, against which systems, and under which license^[1], so later work can be compared against it.