Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research

What Dolma is

Dolma is an open English corpus of three trillion tokens, assembled from a diverse mixture of web content, scientific papers, code, public-domain books, social media and encyclopedic materials^[1]. The paper describing it was submitted to arXiv on January 31, 2024 by Luca Soldaini, Rodney Kinney, Akshita Bhagia and 34 other authors, most of them affiliated with the Allen Institute for AI^[1].

A token is the unit a language model actually reads: text is broken into pieces — often fragments of words rather than whole words — and each piece is mapped to a number before it enters the model. "Three trillion tokens" is therefore a statement about scale, and about how much text is available for pretraining before a model begins to see the same material over and over.

Where the text comes from

Dolma draws on six kinds of source: Common Crawl web text, papers from Semantic Scholar, code from GitHub, public-domain books, Reddit posts and Wikipedia^[2]. Each contributes something different. Web pages give breadth and everyday language; papers and encyclopedic text give expository prose; code gives structured, formal patterns; books give long-form writing; social media gives a conversational register. Mixing them is a way of keeping the corpus from carrying only one voice.

The work of building the corpus was mostly the work of removing things. The team curated roughly 200 TB of raw text down to an 11 TB dataset^[2]. Token counts were computed with the LLaMA tokenizer^[2], which means the three-trillion figure is tied to one particular way of splitting text into tokens; a different tokenizer would give a different number.

What the paper documents

The paper is as much a record of process as a description of a dataset. The authors extensively document Dolma, including its design principles, details about its construction, and a summary of its contents^[1]. Beyond the finished corpus, they present analyses and experimental results on intermediate states of Dolma, so that what the team learned about data curation practices along the way is shared rather than discarded^[1]. In effect this means looking at the corpus partway through its processing pipeline, not only at the packaged final version, and asking what each stage did to the text and what that might mean downstream.

Alongside the corpus, the team open-sourced its data curation toolkit, both to enable reproduction of the work and to support further research in large-scale data curation^[1].

Why it matters

For anyone trying to study how pretraining data shapes a model, the practical obstacle is often that the corpus cannot be inspected: the text exists, but its contents, its filtering decisions and the code that produced it are not available to outside researchers. Dolma is an attempt to remove that obstacle by publishing the corpus together with the record of how it was made and the tools used to make it^[1].

Because the documentation reaches down to intermediate stages, other researchers can ask which filtering and cleaning choices changed the data in which ways, and which of those changes are worth repeating. A corpus released this way is not only something to train on; it is also a worked example of the decisions that go into preparing text at this scale.