EleutherAI releases Common Pile v0.1, an openly licensed 8 TB dataset

Background

Large language models (LLMs) learn by reading very large amounts of text. This training text, called pretraining data, is usually gathered from the web and other sources, and the rights attached to each piece are not always clear. Some researchers have tried to build training collections from material that anyone may legally reuse. The Common Pile is one such effort. It is the successor to The Pile, a collection released in 2020 ^[3].

What happened

In June 2025 EleutherAI and its partners released Common Pile v0.1, an 8 TB collection of public-domain and openly licensed text for LLM pretraining ^[1][2]. Public-domain text is material with no remaining copyright, while openly licensed text carries a license that allows reuse, such as a Creative Commons license.

The collection draws on 30 sources. They include research papers, code, books, encyclopedias and transcripts, and code makes up more than half of the total ^[3].

Key details

Only content that meets the Open Knowledge Foundation's Open Definition 2.1 was included. The definition sets out what counts as open, and licenses such as CC BY, CC0 and MIT qualify ^[3]. A license such as CC BY lets others reuse a work as long as they credit its author, while CC0 waives those requirements altogether.

Collaborators included the University of Toronto, Hugging Face and the Allen Institute for AI ^[3].

To test the data, the authors trained two 7B models, named Comma v0.1-1T and Comma v0.1-2T. They were trained on 1 and 2 trillion tokens respectively ^[2]. A token is a small unit of text, such as a word or part of a word, and the token count describes how much text a model has read during training. The authors report that the two models perform competitively with Llama 1 and 2 7B ^[2]. In this context, competitive means that the scores are close to those models, not that they are better on every test.

Impact

The project shows that a large training collection can be assembled from text whose reuse terms are stated, so that other researchers can inspect what a model was trained on ^[3]. Because the collection is published openly, it can be examined, reused and checked by others. It also forms a link in a line of datasets that began with The Pile in 2020 ^[3].