Instruction tuning

What it is

Instruction tuning is a form of supervised fine-tuning that teaches a large language model to follow user instructions^[1]. It belongs to the fine-tuning stage of training a model, not to pre-training^[1].

A model trained only to predict text can be very good at continuing a passage, but continuing a passage is not the same as doing what a user asks. Instruction tuning is the step that narrows that gap: the model is shown instructions together with the responses that should follow them, and its parameters are adjusted so that this kind of request-and-response behaviour becomes more likely.

How it works

GPT-style models are first pre-trained on large amounts of data to predict the next token, and only then fine-tuned^[1]. Pre-training gives the model its general grasp of language; fine-tuning shapes how it responds. In the formulation used by the FLAN paper, instruction tuning means fine-tuning a language model on a collection of tasks described via instructions^[2].

A concrete illustration helps. A training set for instruction tuning might contain an item that asks the model to translate a sentence into another language, followed by the correct translation, and another item that asks it to judge whether a review is positive or negative, followed by the label. What the model is meant to learn is not one particular task but the habit of reading a description of a task and producing the requested output. Because the same instruction format is reused across many different tasks, the format itself becomes a signal: text phrased this way is something to be carried out rather than continued.

The FLAN paper shows that this works. Instruction tuning substantially improves zero-shot performance on unseen tasks^[2]. Zero-shot here means that at test time the model is given a task it was not trained on, described only by an instruction, and has to perform it without any further examples.

Where it came from

The FLAN paper was first submitted in September 2021^[2]. OpenAI introduced InstructGPT in 2022, a version of GPT-3 fine-tuned in this way to follow instructions^[1]. The FLAN preprint date of September 2021 precedes the InstructGPT paper submitted in March 2022, so instruction tuning as a named method predates the RLHF-based InstructGPT recipe^[2].

How it differs from RLHF

Instruction tuning is often discussed alongside reinforcement learning from human feedback, but the two are different procedures. Reinforcement learning from human feedback first trains a reward model to predict human preferences and then optimises the language model against that reward model with reinforcement learning^[1]. Instruction tuning, by contrast, is ordinary supervised fine-tuning on instruction-and-response examples. In the first case the model is pushed toward whatever the reward model scores highly; in the second it is trained directly to reproduce the answers it was given.

Where it is used

ChatGPT was based on a fine-tuned version of GPT-3.5^[1]. In the pipeline described above, instruction tuning is the stage that sits between a model that has only learned to predict tokens and a model that a person can actually give directions to. That is why it matters: pre-training alone produces a text continuer, and the fine-tuning step is what turns the same underlying model into something that answers a question or carries out a request.

Limits

Because the method is supervised fine-tuning, what the model learns is bounded by the examples it is shown. Biased or inaccurate training data makes the model's outputs unreliable^[1]. Getting the format right — reading an instruction and responding in kind — also does not by itself guarantee that the content of the answer is correct; the quality of the demonstrations is what the behaviour is built on.