Pre-training and fine-tuning

What it is

Pre-training and fine-tuning is a two-stage recipe for building AI models. In the first stage, a large model learns from very large amounts of unlabeled text, using a self-supervised objective. That means the training signal comes from the text itself rather than from human labels. In the second stage, the model is adapted to a specific task using a much smaller labeled dataset. That second stage is called fine-tuning.^[1]

How it works

Suppose a model has read a large body of text and learned to predict what comes next. A company then wants a tool that sorts customer messages into complaints and praise. Instead of training a new model from scratch, the company fine-tunes the pretrained model on a few labeled messages. The model keeps the general language knowledge it gained earlier and only has to learn what the new task asks for.

Two stages, two kinds of data

The two stages use different data. The first stage uses a very large pile of text that nobody has labeled. That text is plentiful, but training on it is expensive. The second stage uses a small set of examples whose correct answers are known. Those examples are costly to produce, but training on them is comparatively light. The recipe combines the strengths of each.

Where it came from

In June 2018, OpenAI's GPT paper pre-trained a Transformer decoder to predict the next word on the BooksCorpus dataset, then fine-tuned it on individual tasks. It improved the state of the art on 9 of the 12 tasks studied.^[1]

Later that year, Google's BERT pre-trained a Transformer encoder using masked language modelling and next-sentence prediction. Masked language modelling hides some words in a sentence and asks the model to predict them from the context on both sides.^[2]

Why it matters

Pre-training moves most of the cost into one expensive run. The result can then be reused for many different tasks, which is the basis of the term foundation model.^[3] Instead of building a separate system for each job, a team starts from one general model and adapts it. A team with little labeled data can still get useful results, because most of the general learning has already happened in the pre-training stage.

Where it stands today

Today's chat assistants add further stages after pre-training, such as supervised instruction tuning and reinforcement learning from human feedback.^[5] Pre-training supplies the broad base, and these later stages shape how the model behaves in conversation.

Limits

Pre-training learns from whatever text it is given, so it absorbs the patterns in that text, including errors and biases. Fine-tuning still needs labeled examples for each new task, and creating those labels takes human effort. The large first stage is also costly, which keeps it out of reach for many groups.