Ai2 releases OLMo 2

Background

A language model is, at bottom, a very large set of numerical parameters together with the code and data used to produce them. Many public releases publish only the first part: the trained weights can be downloaded and run, but the training corpus, the training code and the intermediate states passed through during training stay private. "Fully open" is the term used for releases that include those things as well, so that someone outside the lab can not only use the model but also retrace how it was built.

What happened

On 26 November 2024, Ai2 released OLMo 2 in two sizes, 7B and 13B, where the number refers to billions of parameters ^[1]. The models were trained on up to 5 trillion tokens, a token being the small chunk of text — often a word fragment — that a model reads and predicts during training ^[1]. Alongside the weights, Ai2 published the data, the code, the training recipes and the checkpoints ^[1]. The organisation described the two models as the best fully open language models to date ^[1], a characterisation made by the publisher about its own release.

Key details

Ai2 reported that the 7B model beat Llama-3.1 8B and that the 13B model beat Qwen 2.5 7B ^[1]. Comparisons of this kind are usually made on a fixed set of benchmark tasks; such a claim is a statement about measured scores, not about how the models behave in every situation.

The organisation credited training-stability techniques with keeping its long pretraining runs healthy ^[2]. Pretraining over trillions of tokens means a very large number of optimisation steps, and a run that becomes numerically unstable partway through can degrade or fail, taking the compute already spent with it. Keeping such a run healthy is an engineering problem in its own right, and Ai2 treated it as part of what it was reporting ^[2].

The release deliberately included recipes and intermediate checkpoints, not just the final weights; in Ai2's framing, fully open science requires more than open weights ^[2]. A checkpoint is a snapshot of the model's parameters taken at some point during training. Publishing a series of them lets an outside researcher compare early and late snapshots and watch particular abilities appear as training proceeds, rather than only inspecting the finished model.

Why it matters

Open weights let people run and adapt a model. A full release additionally lets them check how it was produced: which data went in, which code processed it, which settings were chosen, and what the model looked like along the way. For researchers studying how language models acquire abilities during training, for engineers who need to reproduce a result rather than approximate it, and for teachers who want to explain where a model's behaviour comes from, that extra material is the difference between a usable artefact and an inspectable one.

It also sets a comparison point. A release that ships data, code, recipes and checkpoints makes it possible to ask of later releases which of these parts are actually included, so that the phrase "fully open" has something concrete to refer to.