The Pile: An 800GB Dataset of Diverse Text for Language Modeling

What problem it addresses

Large language models learn from very large collections of text. By 2020 much of that text came from web crawls, which are automatic copies of web pages. The Pile was built to give open researchers a large and diverse pretraining corpus that goes beyond web crawls. ^[1]

The title of the paper is "The Pile: An 800GB Dataset of Diverse Text for Language Modeling". The corpus itself is described as an 825 GiB English text corpus. ^[1][2]

How it was built

The Pile is built from 22 smaller, high-quality datasets. Some were already in existence and others were newly constructed for the project. Many of them come from academic or professional sources. ^[1][2]

The paper was submitted to arXiv on 31 December 2020. Leo Gao is the lead author, and there were 11 co-authors. ^[1]

Results and use

The Pile became the training data for EleutherAI's GPT-Neo, GPT-J, GPT-NeoX-20B and Pythia models. ^[3]

Criticism

The Pile has been criticised over its sources. Its Books3 subset was taken offline in August 2023 after an action by an anti-piracy group. ^[3]

In July 2024 a Proof News and Wired investigation found that the dataset contained subtitles from over 170,000 YouTube videos across more than 48,000 channels. ^[3]

Later influence

In June 2025 EleutherAI released Common Pile v0.1, an 8 TB successor that contains only public-domain and openly licensed text. ^[4] Common Pile is described as containing only public-domain and openly licensed text, which is the main point of difference from The Pile. ^[4]

For readers, The Pile is a useful case study. It shows that a dataset can be important for open research and still face serious questions about where its text came from. Anyone studying models trained on it should look closely at what the dataset contains.