The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale

The problem: web text is not training data

Large language models learn their abilities from the text they are pretrained on, and much of the openly available pretraining text is derived from Common Crawl, an archive that has been accumulating crawled web pages for years. Raw crawl is messy: it contains duplicated pages, boilerplate navigation and advertising, spam, machine-generated text, and pages in many languages. Turning raw crawl into a usable corpus therefore means making a long chain of decisions about which pages to keep and how to clean them. Those decisions shape what a model learns, yet they are often not spelled out in detail when a dataset is published.

What FineWeb is

The FineWeb paper was submitted to arXiv on June 25, 2024 by Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra and Thomas Wolf ^[1]. FineWeb itself is a 15-trillion token dataset derived from 96 Common Crawl snapshots, and it produces better-performing language models than other open pretraining datasets ^[1]. A token is the unit a model reads text in — roughly a word or a piece of a word — so 15 trillion tokens is a corpus at a scale no one could assemble by hand.

Drawing on 96 snapshots rather than a single crawl means the corpus is assembled from many separate captures of the web taken at different times, which widens the range of pages and sites it covers.

Documenting every design choice

The paper's central methodological claim is documentation. It carefully documents and ablates all of the design choices used in FineWeb, including in-depth investigations of deduplication and filtering strategies ^[1]. An ablation means removing or altering one component, retraining, and measuring how much the result changes, so that a step's contribution is measured rather than assumed. Here the ablations cover deduplication — finding and removing near-identical pages so that the same text is not seen over and over — and filtering — deciding which pages are worth keeping at all ^[1].

Along with the datasets, the authors publicly released the data curation codebase and all of the models trained during the ablation experiments ^[1]. That release makes the work checkable: other groups can rerun the pipeline, inspect the ablation models, or reuse a single component in their own curation.

FineWeb-Edu

The second release is FineWeb-Edu, a 1.3-trillion token collection of educational text filtered from FineWeb ^[1]. Rather than only deleting bad pages, this step actively keeps pages judged educational, producing a corpus that is much smaller but far more targeted.

The payoff appears at evaluation time: LLMs pretrained on FineWeb-Edu exhibit dramatically better performance on knowledge- and reasoning-intensive benchmarks like MMLU and ARC ^[1]. Such benchmarks pose questions that require recalling facts and reasoning through several steps, so gains there suggest the educational filter is selecting text that actually conveys knowledge, not merely text that reads fluently.

Why it matters

Three things stand out. First, scale: a 15-trillion token corpus built from public web crawl is large enough to serve as a pretraining base in its own right ^[1]. Second, method: by ablating deduplication and filtering instead of asserting that they help, the paper turns curation folklore into evidence about which choices matter ^[1]. Third, reproducibility: because the curation code and the ablation models were released together with the data ^[1], claims about how the corpus was built can be inspected and rerun rather than taken on trust. The title's word, decanting, captures the aim — pouring off the clearer part of a very large and murky body of web text.