Pythia

What Pythia is

Pythia is a family of open language models from EleutherAI. It contains 16 models in total: eight sizes — 70 million, 160 million, 410 million, 1 billion, 1.4 billion, 2.8 billion, 6.9 billion and 12 billion parameters — each trained twice, once on The Pile and once on a deduplicated version of The Pile^[1][2]. The largest model in the family has 11.85 billion total parameters^[2]. All of them are released under the Apache 2.0 license and hosted on the Hugging Face Hub^[2].

The released models are not the first version. They were renamed and retrained in January 2023; the model now called 12B, for example, was previously called 13B^[2].

Why it was built

The suite was built to answer a question that final model weights cannot answer on their own: how large language models develop and evolve over the course of training^[1]. Most model releases publish only the finished model, so a researcher who wants to know when a capability appears, or how a behaviour changes as training proceeds, has nothing to look at. Pythia was designed around that gap. The work is described in a paper first posted to arXiv on 3 April 2023, whose first author is Stella Biderman^[1].

The same data, in the same order

The design decision that shapes everything else is uniformity. Every model in the suite saw the same public data in exactly the same order: about 300 billion tokens — 299,892,736,000 in total — with a batch of 2 million tokens for 143,000 steps^[1][2]. Because the data and its ordering are fixed, differences between two models can be traced to their size, or to the difference between the raw Pile and the deduplicated Pile, rather than to a different mixture of training data or a different shuffle. The two Pile variants exist for the same reason: they let researchers ask what deduplication alone changes.

Checkpoints as the main instrument

For each model, 154 checkpoints are public. They include step 0, ten log-spaced checkpoints from the early part of training, and then one checkpoint every 1,000 steps^[1][2]. Log spacing means the early checkpoints cluster tightly at the start, where a model changes fastest, and thin out later. With a checkpoint every thousand steps, a researcher can take a measurement — an accuracy score, a probe, a test for memorisation — and repeat it along the whole training run instead of only at the end.

To make that possible, the authors also released tools to download and reconstruct the exact training dataloaders of each model^[3]. A checkpoint says what the model was at some step; the dataloader says what it was reading at that step. Together they turn a training run into something that can be replayed and audited rather than only inspected at its endpoint.

What the paper studies

The paper presents case studies on three questions: memorization, the effect of term frequency on few-shot performance, and reducing gender bias^[3]. Each is a question about training dynamics rather than about a single final score, which is why the suite was built the way it was. Memorisation, for instance, can be followed step by step to see when a model starts reproducing text it was trained on; the term-frequency question asks how much a model's few-shot behaviour depends on how often a term appeared in training; the bias study looks at whether an intervention during training reduces biased behaviour.

Where it sits

Pythia's contribution is less a new architecture than a piece of shared infrastructure for studying training. Fixed data order, matched runs at eight scales, dense checkpoints and open weights together make it possible to run controlled experiments on how behaviour emerges during training — and to check someone else's result by rerunning the same steps on the same data.