RedPajama: open reproduction of the LLaMA training dataset

RedPajama is a project to create leading open-source models, and it begins in an unusual place: not with an architecture or a training run, but with the data. The project's first step was to reproduce the training dataset used for LLaMA, a corpus of more than 1.2 trillion tokens.^[1] In its dataset card the result is described as a clean-room, fully open-source implementation of the LLaMA dataset.^[3] "Clean-room" here means the corpus was assembled independently from openly available material rather than by copying the original artifact.

Why start from the data

A large language model learns by predicting text, and everything it learns comes from the text it is trained on. That text is measured in tokens — the chunks, often parts of words, into which a model's input is split. Before training, a corpus of trillions of tokens has to be collected, cleaned and mixed in fixed proportions. The mixture matters: a corpus made only of web pages teaches a model something different from one that also contains source code, scientific papers and encyclopedia articles.

Reproducing a known dataset gives everyone a common reference point. If two groups train on the same data, differences in results can be attributed to the model rather than to the corpus. Publishing the data openly also lets researchers outside the organisation that built the original model inspect what such a model is actually trained on.

What is in the corpus

The 1T release is not a single scrape but a blend of six sources, and the dataset card gives the token count for each: CommonCrawl accounts for 878 billion tokens, C4 for 175 billion, GitHub for 59 billion, ArXiv for 28 billion, Wikipedia for 24 billion and StackExchange for 20 billion, adding up to the 1.2 trillion total.^[3]

The shape of that mixture is worth reading carefully. Two web-derived sources, CommonCrawl (a public archive of crawled web pages) and C4 (a cleaned web corpus), supply the overwhelming majority of the tokens. The remaining four are smaller but qualitatively different: GitHub contributes source code, ArXiv scientific preprints, Wikipedia encyclopedia articles and StackExchange question-and-answer threads. Each brings a style of text that the web at large does not reliably provide at length.

What was released

The main artifact, RedPajama-Data-1T, consists of 2,084 jsonl files — a format in which each line is a separate JSON record, a convenient layout for text corpora that are read record by record.^[3] Alongside the full corpus, a smaller 1B-token sample was also released, a more manageable version of the same data.^[3]

The data was accompanied by code. The RedPajama-Data repository on GitHub, which holds code for preparing large datasets for training large language models, was created on April 14, 2023.^[2]

Where it sits

RedPajama treats the training corpus as a first-class, openly published artifact. The published token breakdown makes the composition of a pretraining set checkable rather than a matter of guesswork, and the accompanying preparation code means the pipeline can be rerun rather than only looked at. The project's stated aim goes further than the data: reproducing the corpus is the starting point for the larger effort of creating leading open-source models.^[1]