OLMo

What it is

OLMo, short for Open Language Model, is a language model from the Allen Institute for AI (Ai2). It was released on 1 February 2024 in 1B and 7B sizes. The project had been announced on 11 May 2023 ^[1][4]. Ai2 described OLMo as "truly open". Alongside the model weights, it published the Dolma pretraining dataset and the code that builds that dataset, the training code, the training logs, Weights & Biases metrics and the evaluation code ^[1][2]. Weights & Biases is a service that records training metrics, and the metrics show how the training run progressed.

Origins

Ai2 announced the OLMo project on 11 May 2023 and released the models nine months later, on 1 February 2024 ^[1][4]. The release was accompanied by an arXiv paper, "OLMo: Accelerating the Science of Language Models", written by Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia and 39 others. It was posted to arXiv on 1 February 2024 ^[2].

Several partners helped with the project. They included AMD, CSC in Finland, MosaicML and Databricks, Harvard's Kempner Institute and the University of Washington ^[1].

Main features

Each base model was trained on at least 2 trillion tokens. OLMo 7B was trained on 2.5 trillion tokens of Dolma, which makes about 3 trillion tokens in total ^[1][3][4]. A token is a small unit of text, such as a word or part of a word.

OLMo 7B has 32 layers, a hidden size of 4096, 32 attention heads and a context window of 2,048 tokens ^[3]. The hidden size and attention heads describe the internal design of the network, and the context window is the amount of text the model can consider at one time.

Each base model has more than 500 checkpoints, saved every 1,000 training steps ^[1]. A checkpoint is a copy of the model saved during training, so a researcher can see how it changed over time. OLMo 7B was trained on A100-40GB GPUs provided by MosaicML. A twin model was trained on AMD MI250X GPUs on LUMI, a supercomputer in Finland ^[3].

All code, weights and intermediate checkpoints were released under the Apache 2.0 license, which allows reuse, modification and redistribution ^[1]. Ai2 also released instruction-tuned versions, OLMo-7B-Instruct and OLMo-7B-SFT, together with the Open Instruct code ^[1].

Performance

Ai2 reported that OLMo 7B is on par with Llama 2 on many generative and reading-comprehension tasks. It was slightly behind on MMLU and Big-Bench Hard, two benchmark suites that test a wide range of knowledge and reasoning ^[1]. These are Ai2's own results, so readers should look at the evaluation details before comparing the models closely.

Place in the field

OLMo was released with its training data, logs and evaluation code alongside the weights. This makes the training process something that other researchers can inspect. OLMo was succeeded by OLMo 2 in November 2024 ^[4]. The OLMo name then became the name of a whole family of models, and the later members are described in their own entries.