Ai2 releases OLMoE, an open mixture-of-experts model

What a mixture of experts is

A standard language model uses all of its parameters for every token it reads or writes. A mixture-of-experts (MoE) model divides its parameters into separate blocks called experts and adds a router that decides, token by token, which few experts should handle it. Only those experts are run, so the model can hold far more parameters in total than it spends computation on for any single token. The trade-off is that the routing has to work: if the router sends tokens to experts that do not help, the extra capacity buys little. This is the general idea behind the design; OLMoE is one concrete implementation of it.

What Ai2 released

In September 2024, Ai2 released OLMoE ^[1]. The accompanying paper was posted on 3 September ^[1]. The model has 7B total parameters but only 1B active parameters per token ^[1], which places it in the sparse mixture-of-experts family: a large pool of parameters, a small slice of them used for each token. It was trained on 5 trillion tokens ^[1]. Ai2 developed the model with Contextual AI ^[2].

Open weights, data, code and logs

The release was not just a set of weights. Ai2 open-sourced the model weights, the training data, the code and the training logs ^[1]. That combination matters for how such a model can be used and checked. Weights alone let people run the model; data, code and logs let them retrace how it was built and rerun or modify parts of the process. For a mixture-of-experts model, where the behaviour of the router is part of the design, having the training record available makes the internals easier to study rather than only the outputs.

Reported results

The paper reported that OLMoE outperformed all available models with similar active parameters, and that it even surpassed Llama2-13B-Chat and DeepSeekMoE-16B ^[3]. Those comparisons are stated in terms of active parameters, which is the relevant measure for a sparse model: OLMoE spends roughly the compute of a 1B-parameter model per token while holding 7B parameters in total, and the comparison is against larger models on that per-token basis. The authors also said that their routing analysis showed high specialization among the experts ^[3] — that is, different experts tended to take on different kinds of work rather than all behaving the same way. That finding speaks directly to the question an MoE design raises: whether the router actually makes useful distinctions.

Why it matters

OLMoE is interesting less for any single number than for the combination it puts in public: a sparse mixture-of-experts model together with the data, code and logs behind it. Mixture-of-experts architectures are widely used because they separate total capacity from per-token cost, but the details of training them — how the router behaves, how the experts divide up the work — are usually not visible from outside. A release that includes the training record gives researchers and students a case they can inspect end to end, and gives anyone building on the architecture a documented starting point. The reported specialization of the experts is the kind of finding that only becomes checkable when the routing behaviour and the training setup are open to examination.