Olmo Hybrid

What Olmo Hybrid is

Olmo Hybrid is a fully open language model with 7 billion parameters, released by the Allen Institute for AI (Ai2) on 5 March 2026. What sets it apart is the layer stack: rather than building every layer out of Transformer attention, it places attention and linear recurrent layers side by side, an arrangement Ai2 calls a hybrid architecture.^[1]

How the layers are arranged

The model interleaves Gated DeltaNet linear-RNN layers with attention in a 3:1 ratio: three DeltaNet sublayers, then one multi-head attention sublayer, repeated through the network. These layers take the place of the sliding-window attention layers used in Olmo 3.^[1]

The two kinds of layer do different jobs. A linear recurrent layer carries a fixed-size state forward from token to token and updates it as it reads, so the cost of handling each token stays roughly constant and the state does not grow with the length of the text. Gated DeltaNet belongs to this family: the delta rule decides how the state is corrected by each new input, and a gate controls how much is written into the state at each step. Attention works the other way round: every token can look directly at the tokens before it, which is expressive but gets more expensive as the context grows, because the work scales with sequence length. Putting an attention layer in every fourth position aims to keep that ability to look back directly, while most of the depth runs at recurrent cost.^[1]

Training

Olmo Hybrid was trained on 6 trillion tokens using the improved data mix developed for Olmo 3 32B. The run used 512 GPUs, which switched from H100s to B200s about halfway through.^[1]

Results

Ai2 reports that the model matches Olmo 3 7B's MMLU accuracy while using 49% fewer tokens. After mid-training it led Olmo 3 7B on every primary evaluation domain. On the RULER long-context test at 64K tokens it scored 85.0 with DRoPE, against 70.9 for Olmo 3 7B with YaRN.^[1]

Why Ai2 argues it matters

Beyond the single model, Ai2 presented the release as evidence about architecture in general. The institute argued that hybrid architectures are fundamentally more expressive than pure transformers or pure linear RNNs, and supported that claim with theoretical analysis and with scaling experiments showing more efficient pretraining.^[2] It also projected that the token savings grow with model size, from about 1.3x at 1 billion parameters to about 1.9x at 70 billion.^[1]

Where it sits

Olmo Hybrid sits in the Olmo line of openly released models, as a 7B sibling of Olmo 3 in which part of the attention stack is replaced by linear recurrence. Because both the hybrid model and the model it is measured against are open, the token-efficiency comparison Ai2 makes is one that outside researchers can examine and build on.^[1] The savings quoted for 1 billion and 70 billion parameters are projections rather than measurements of models trained at those sizes.^[1]