Ai2 releases OLMo, a fully open language model

Background

A language model is the product of a long training run: a very large amount of text goes in, and a set of weights comes out. What is usually published under an "open" label is those final weights alone. The corpus the model was trained on, the code that ran the training, and the intermediate states the model passed through on the way are typically not part of the release. Without them, an outside researcher can measure a finished model, but cannot easily reproduce it, audit what went into it, or watch how its behaviour emerged step by step.

OLMo was built to close that gap, and the release covered much more than the final weights.

What was released

On 1 February 2024, the Allen Institute for AI (Ai2) released OLMo in two sizes, 1B and 7B ^[1]. The release included Dolma, the pretraining dataset, the training code, the training logs and more than 500 checkpoints for each model ^[1]. Everything was published under the Apache 2.0 licence ^[1].

Ai2 described the work as a collaboration, naming AMD, CSC, Databricks, Harvard's Kempner Institute and the University of Washington as partners ^[1].

Checkpoints and data tooling

A checkpoint is a snapshot of a model's weights saved partway through training, so that training can be paused, resumed or inspected. OLMo's checkpoints were saved every 1,000 training steps, which produced over 500 of them for each base model; on Hugging Face each one is available as a separate revision of the model repository ^[2].

Alongside the Dolma dataset and the code used to generate it, the release included WIMBD, a tool for analysing pretraining data ^[2]. Data analysis matters here because the contents of a pretraining corpus shape what a model knows and what it can be asked about; a tool that lets outsiders search and count what is in the corpus makes the dataset itself examinable rather than a fixed input taken on trust.

The checkpoint trail is the other side of the same idea. A model that only exists as a final set of weights can be tested but not followed. A model that exists as hundreds of ordered snapshots can be studied as a process: when a particular ability appears, what it looks like in the middle of training, and how the model changes between one stage and the next. The same snapshots also make it possible to restart training from an intermediate point instead of from scratch.

How it compared

Ai2 said that OLMo 7B was on par with Llama 2 on many tasks, and slightly behind it on MMLU ^[1]. Llama 2 was therefore the reference point the release was measured against: the claim was not that OLMo was the strongest model available, but that a fully open model of that size could stand alongside a widely used one on a broad set of tasks.

Why it matters

The argument for releasing data, code, logs and checkpoints together is that each answers a different question. The data says what the model was taught. The code says how. The logs say what happened during the run. The checkpoints say what the model looked like at each stage. A release that contains all four can be retrained, audited and studied; a release that contains only the weights can only be used. OLMo was presented as an attempt to make the first kind of release normal, and to give researchers outside the labs that train large models something they can reproduce and examine rather than merely query.