Common Crawl

What it is

Common Crawl is a nonprofit 501(c)(3) organization that crawls the web and freely provides its archives and datasets to the public^[1]. Its corpus — the accumulated body of pages it has collected — contains petabytes of data and has been collected regularly since 2008^[2]. A crawl is a single pass over the public web in which an automated program follows links from page to page and stores copies of what it finds. Common Crawl publishes the results of those passes as a series of crawls, each identified by a code such as CC-MAIN-2026-43^[2]. What the project releases is a raw archive rather than a curated collection, so anyone who wants to work with it has to filter, deduplicate and clean the pages first.

Origins

The project was founded by Gil Elbaz in 2007 in San Francisco, and it began archiving websites that year^[1]. Collection of the corpus has continued regularly since 2008^[2], but Common Crawl only began publishing its crawls in 2011^[1]. So for its first few years the archive was being built faster than it was being handed out, and the public datasets the project is known for date from that later starting point.

How the data is reached

Access to the data is free on Amazon Web Services, though users may incur storage and compute costs^[1]. In practice that means the archive itself costs nothing, while the bill falls on whoever downloads large volumes and processes them — a cost that matters because working with petabytes of pages requires substantial machinery on the user's side. The model is closer to a public library of raw material than to a finished product: the organisation maintains and publishes the collection, and users bring their own tools.

Who uses it

For most of its history the data was mostly used by researchers and some startups^[1]. That changed in the 2020s, when AI companies started training large language models using the data^[1]. A large language model learns statistical patterns from very large quantities of text, and a broad sample of the public web is one way to obtain text at that scale. A project started to keep a copy of the web therefore became, in effect, part of the supply chain for a different technology.

The November 2025 investigation

In November 2025, an investigation by The Atlantic revealed that Common Crawl misled publishers when it claimed it respected paywalls in its scraping and was not honoring removal requests^[1]. A paywall is the practice of placing content behind a subscription; a removal request is a publisher asking that its material be taken out of a dataset. The finding matters because the archive's value rests on a simple bargain with the sites it copies: material is collected and redistributed openly, and the sites are told how their wishes will be handled. Both parts of that account were called into question by the investigation^[1].

Place in the development of AI

Common Crawl sits at a point where web archiving and machine learning meet. The crawl is a general-purpose record of the public web, built for anyone to use; the same property — enormous, broad, freely available text — is what made it useful for training models. Its history also shows the tensions that come with that role: costs that land on users rather than the archive^[1], and a long-running argument about whether the sites being copied have any say. For readers trying to understand where the text behind modern AI systems comes from, Common Crawl is one of the clearest concrete answers.