The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text

Pretraining a large language model requires a corpus of text far larger than the model itself, and that corpus is usually assembled by crawling the web. Much of the resulting text has no clear license attached to it. The Common Pile v0.1, submitted to arXiv on June 5, 2025, is a dataset built to sidestep that problem: it is assembled entirely from openly licensed text and released for others to use ^[1].

What is in the collection

The Common Pile v0.1 is an eight terabyte collection of openly licensed text designed for pretraining large language models ^[1]. It comprises content from 30 sources, spanning research papers, code, books, encyclopedias, educational materials, and audio transcripts ^[1].

Those categories are not incidental. A pretraining corpus has to expose a model to many kinds of text at once — formal prose, technical notation, reference-style definitions, dialogue — because a model learns statistical structure from whatever it sees. A corpus that is cleanly licensed but narrow would produce a model that is good at a few things and poor at the rest. Drawing on 30 sources spread across these broad categories is the authors' answer to that concern.

Testing whether the data is good enough

A pretraining dataset is only credible if a model trained on it actually works. The authors validated the collection by training two 7 billion parameter language models on Common Pile text: Comma v0.1-1T and Comma v0.1-2T, trained on 1 trillion and 2 trillion tokens respectively ^[1]. Both models attain competitive performance to LLMs trained on unlicensed text with similar computational budgets, such as Llama 1 and Llama 2 7B ^[1].

The comparison is the substance of the paper's argument. Publishing a corpus only establishes that the text exists and carries a license; it does not establish that the text is sufficient to train a useful model. Training the Comma models and placing them next to models of the same size trained on conventional, unlicensed data turns the dataset into a testable claim rather than an assertion.

What was released

Along with the dataset, the authors released the code used in its creation as well as the training mixture and checkpoints for the Comma v0.1 models ^[1]. The training mixture matters because a corpus assembled from 30 sources still has to be sampled in some proportion; publishing the mixture makes the recipe reproducible instead of leaving the proportions a private detail. The checkpoints let others evaluate the resulting models directly rather than retraining them from scratch.

The paper has 26 authors. Nikhil Kandpal, Brian Lester, Colin Raffel, Sebastian Majstorovic and Stella Biderman are joined by 21 other authors ^[1].

Why it matters

Most of the public debate about training data has been about whether scraped text can be used, not about what a model trained without it would look like. The Common Pile addresses the second question directly: it provides a large, licensed corpus and a demonstration that models trained on it reach the level of models trained on unlicensed text at comparable computational budgets ^[1]. For anyone who wants to train a model and cannot or will not rely on scraped data, that is a starting point rather than a promise.

The released code, mixture and checkpoints also make the result checkable. Reproducibility is often the weak point of dataset papers: a corpus is announced, but the filtering decisions, the sampling weights and the resulting models stay out of reach. Here the pieces needed to rebuild the pipeline and re-evaluate the models are published together with the data ^[1].

The validation reported in the paper covers 7-billion-parameter models trained on 1 and 2 trillion tokens, compared against models trained on unlicensed text at similar computational budgets ^[1]. What a corpus of this kind can support at larger scales, and how far the performance gap narrows or widens as compute grows, are questions the released models and data leave open to further work.