Synthetic data

What it is

Synthetic data is training material produced by a model rather than collected from people or the web^[1]. The defining feature is origin rather than appearance: a synthetic corpus can look exactly like ordinary prose or ordinary code, but a model wrote it, instead of it being scraped from web pages or labelled by a person. That distinction matters because the cost and the failure modes are different from those of collected data — the bottleneck moves from finding and cleaning material to steering the model that writes it.

How it works

A synthetic pipeline has two roles: a generator model that writes the material, and a student model that is trained on it. The generator can be steered towards a particular shape — a chapter that introduces an idea step by step and then sets exercises on it, say — rather than the uneven mixture of explanation, half-finished code and discussion that ordinary web pages contain. The student then learns from that shaped material, and whoever builds the training set decides how much synthetic text to mix with real text.

The best-documented case is phi-1, a Transformer-based language model for code with 1.3B parameters, published by Gunasekar et al. in June 2023 under the title "Textbooks Are All You Need"^[1]. Its training mix combined 6B tokens of "textbook quality" data selected from the web with 1B tokens of synthetically generated textbooks and exercises produced with GPT-3.5^[1]. Training ran for four days on eight A100 GPUs^[1]. Despite its small scale, phi-1 reached pass@1 accuracy of 50.6% on HumanEval and 55.5% on MBPP^[1]. A smaller 350M-parameter model trained with the same pipeline, phi-1-small, still reached 45% on HumanEval^[1]. The authors also reported surprising emergent properties relative to phi-1-base, the model before fine-tuning on the synthetic exercise dataset^[1].

Where it came from

In this record the idea is tied to the paper's claim: that data quality, and not only data volume or parameter count, drives capability at this scale^[1]. One detail of the experimental design is worth keeping. The synthetic corpus was generated by GPT-3.5, a different model from the one being trained^[1]. Because generator and student were not the same model, the authors treat the synthetic text as a distinct supervision signal rather than as self-training, the case in which a model is trained on its own output^[1].

Where it is used today

Broadly, synthetic data fits wherever real material is scarce, expensive or awkward to obtain: producing practice material for a model to learn from, generating text in a fixed format, and building test sets. The workflow is much the same each time — pick a generator, decide the shape of what it writes, then decide how the output is mixed with real data.

Limits

A student can only learn what the generator produces, so errors and stylistic habits in the generated material are inherited rather than averaged away. Diversity is bounded by the generator too: a synthetic corpus tends to be narrower and more uniform than the open web. And a result obtained with a code model at one scale does not by itself settle how much synthetic data helps elsewhere. The phi-1 authors framed their finding as a claim about data quality rather than as a general recipe^[1].