EleutherAI releases The Pile training dataset

Background: the corpus decides what a model can do

A language model learns its behaviour from the text it reads during training. A corpus that is large but narrow produces a model that does well on the kind of text it saw and poorly on everything else. Text collected from the open web is easy to gather in huge quantities, but it mixes many kinds of writing unevenly and leaves some of them thin. Building a training set is therefore a question of choosing a mix of sources, not only of choosing a size.

What was released

On 31 December 2020, EleutherAI published a dataset called The Pile together with its paper on arXiv ^[1]. The dataset contains 825 GiB of English text ^[1], assembled by combining 22 smaller, high-quality datasets into a single collection ^[2]. It was released openly so that anyone could use it to train language models ^[2]. The paper's lead author was Leo Gao, who wrote it with 11 co-authors ^[1].

How the collection is put together

The 22 component datasets are not all of the same kind, and that is deliberate: many of them derive from academic or professional sources rather than from the open web ^[3]. That gives the collection a substantial share of papers, reference material and specialist writing alongside more ordinary text. The data is distributed as jsonlines files compressed with zstandard ^[4], a combination that keeps a very large corpus practical both to download and to stream during training. Alongside the data, the project defines Pile BPB — bits per byte — as a benchmark for measuring how well a model models text across domains, rather than on a single kind of text ^[4].

What the paper found

The paper tested how existing models cope with the collection as it stands. Untuned GPT-2 and GPT-3 struggled on many of its components, among them academic writing ^[3]. In other words, the parts of the corpus that look least like ordinary web text were exactly where general-purpose models did worst. Training on the Pile changed the picture: models trained on it improved significantly over both Raw CC and CC-100, the two baselines, on all components of the Pile ^[3].

Why it mattered

The result made two points at once. First, a corpus assembled from many curated sources is a different kind of object from text collected in one sweep, and models trained on it change across the board, not only on the source types that were added ^[3]. Second, a diverse corpus comes with its own way of measuring progress: bits per byte across many domains gives a summary of cross-domain modelling ability that a single benchmark on one kind of text cannot ^[4]. Because the data and the benchmark were released openly ^[2][4], other groups could train on the same text and compare results directly, which is what makes a shared training set useful beyond the team that built it.