Mixture of experts (MoE)

What it is

A mixture of experts (MoE) is a neural network architecture in which a layer contains many parallel sub-networks, called experts. A small router network decides which few experts should handle each token. The other experts do not run for that token.^[1]

How it works

Picture a team of specialists with a receptionist at the door. The receptionist reads each incoming request and sends it to the few specialists best suited to it. Different requests reach different specialists, so over many requests the whole team is used, yet each request costs only a fraction of the total work. In an MoE layer, the router plays the receptionist and the tokens are the requests.

Only the chosen experts run. A model can therefore have a very large total number of parameters while the compute for each token stays close to that of a much smaller model.^[1]

The router is itself learned during training. It has to decide, token by token, which experts fit best, and those decisions change as the experts change.

Where it came from

In January 2017, Shazeer and colleagues at Google introduced the sparsely-gated MoE layer for deep networks. The idea was to keep the capacity of a large model while activating only a small part of it at a time.^[1]

Where it is used today

Mistral AI's Mixtral 8x7B, released in December 2023, has 46.7 billion total parameters but uses 12.9 billion per token.^[2]

DeepSeek-V3, from December 2024, has 671 billion total parameters, with 37 billion activated per token.^[3] Both models follow the same pattern: the stored capacity is large, but the work done for each token is modest. The two differ in size and in the companies that built them, but the design principle is the same.

Why the gap matters

A dense model, in which every parameter takes part in every computation, pays for its size at every step. An MoE model can hold more parameters than it uses at any one moment, so its total size no longer determines the cost of each token in the same way. This is the main appeal of the design. In other words, storing a model and running it are different costs, and MoE mainly reduces the second.

Limits

All experts must still fit in memory, even though only a few run for any given token. Routing also has to stay balanced. If the router sends too many tokens to a few experts, those experts become overloaded while others sit idle, and the efficiency gain is lost. Engineers must also decide how many experts to include and how many to use for each token. Those choices set the trade-off between capacity and cost.