In-context learning

What it is

In-context learning is the ability of a large language model to perform a new task from instructions and a few examples placed in the prompt, without updating its weights^[1]. Nothing is trained and no parameter changes. The model reads one prompt — a task description, sometimes a handful of worked examples, then the input it has to handle — and produces an answer. The "in-context" part is literal: whatever the model takes from those examples lives inside the current input, and the next request starts over. That is what lets a single trained model be used for tasks nobody trained it on, as long as the task can be described in words.

How it works

The GPT-3 paper, which named the phenomenon and studied it systematically, compared three settings: zero-shot, with an instruction only; one-shot, with a single example; and few-shot, with several examples^[1]. Comparing the three makes visible how much difference the number of examples alone makes^[1]. Take a task such as labelling short reviews as positive or negative. A zero-shot prompt only states what to do. A few-shot prompt also lists a few reviews, each already marked with its label, and then gives the unlabelled review; the model continues the pattern and emits a label. Because the model conditions on everything in the prompt at once, those examples work as a demonstration of the wanted output rather than as data to be fitted. They are not training data in the usual sense: they are read once, as part of the input, and are gone with the next request.

Where it comes from

The term comes from the GPT-3 paper of May 2020, "Language Models are Few-Shot Learners"^[1]. The paper trained eight model sizes, from 125 million to 175 billion parameters, all on a total of 300 billion tokens^[1]. GPT-3 itself had 175 billion parameters, and the paper found that few-shot performance improved strongly with model size^[1]. That is part of why the idea drew attention: a prompt that a small model handles badly becomes usable as models grow.

What changed

Before, adapting a model to a task normally meant collecting a dataset and fine-tuning. In-context learning offered a different route: users write prompts, and a prompt can include a few examples for the model to learn from in context^[2]. Structuring such inputs has a name of its own, prompt engineering^[2]. The unit of work shifts from a dataset to a piece of text that can be edited and tried again.

Limits

Results are highly sensitive to the prompt: wording, formatting and the order of the examples all matter^[2]. And every example has to fit inside the model's context window, the amount of text it can read at once, which was 2,048 tokens for GPT-3; that caps how much can be demonstrated in one go^[1][2]. Within those limits, no training run is needed — the whole adaptation is a text edit.