OLMoE

What OLMoE is

OLMoE is a fully open sparse mixture-of-experts (MoE) language model from the Allen Institute for AI (Ai2), described in a paper posted to arXiv on 3 September 2024.^[1] Its total parameter count is 7 billion, of which 1 billion are active for each input token.^[1] The paper introduced two models: the base model OLMoE-1B-7B and an instruction-tuned variant, OLMoE-1B-7B-Instruct.^[5]

What "sparse mixture of experts" means

A dense language model applies all of its parameters to every token it reads. A mixture-of-experts model instead splits part of the network into many parallel "expert" sub-networks and adds a small router that decides, token by token, which experts to use. Because only a few experts run for any given token, the model can hold far more parameters than it uses at once. That is the sense in which OLMoE is sparse: 7 billion parameters sit in the model, but a single token passes through only 1 billion of them.^[1] The attraction is arithmetic — per-token compute stays close to that of a much smaller model, while the parameter count, and with it the room the model has to store what it learns, can grow.

Training and results

OLMoE was pretrained on 5 trillion tokens.^[1] According to the paper, it outperforms models with a similar number of active parameters, and it also surpasses larger models such as Llama2-13B-Chat and DeepSeekMoE-16B.^[1] This is the comparison that MoE designs are built for: at inference time what a model spends is its active parameters, so a sparse model is measured against models that cost far more to run per token.

What the authors found inside

Beyond benchmark numbers, the paper examined the routing behaviour of the trained model. The authors analysed how tokens are assigned to experts and reported high specialization among experts.^[5] In a mixture-of-experts layer the router is the only part that decides where a token goes, so whether the experts end up doing genuinely different work — rather than converging on the same work and wasting parameters — is a central question for the design.

Openness

OLMoE was released as a fully open model: weights, training data, code and training logs were all made public, and the weights carry an Apache 2.0 licence.^[1][4] The model was built with Contextual AI.^[2] The paper was first posted on 3 September 2024 and revised on 3 March 2025.^[5] Releasing the data, code and logs alongside the weights lets others reproduce the training run and study how a sparse model behaves, rather than only downloading a finished checkpoint.

Where it sits

OLMoE went on to serve as the language backbone of MolmoE-1B, a multimodal model from Ai2.^[3] That reuse is a small illustration of what an openly released base model is for: it can be taken up as a component in systems its original authors did not build, while the paper and the released material document how it was trained and how its experts divide the work.