AI alignment

What alignment means

Alignment is the problem of making a model's behaviour match what its users and designers actually intend, rather than only what its training objective literally rewards.^[1][2] The two are not the same thing: an objective is a stand-in for what people want, and a system optimising it can find ways to score well that nobody asked for. In the language-model setting, the framing starts from the observation that making models bigger does not inherently make them better at following a user's intent — that gap is what the word *aligned* refers to.^[2]

Learning a goal from human comparisons

One influential approach replaces a hand-written reward function with human judgement. Christiano et al. published a foundational method in June 2017 that defines goals in terms of non-expert human preferences between pairs of trajectory segments.^[1] A person is shown two short pieces of behaviour and asked which one they prefer; the answers are used to learn a model of that preference, and the preference model supplies the reward that reinforcement learning optimises. Nobody has to write the reward down — the goal is communicated by example instead.

The method solved complex reinforcement learning tasks without access to the reward function, including Atari games and simulated robot locomotion, while collecting feedback on less than one percent of the agent's interactions with the environment.^[1] The authors reported that complex novel behaviours could be trained with about an hour of human time, reducing the cost of human oversight enough to apply it to state-of-the-art RL systems.^[1] That cost figure is the practical heart of the idea: oversight only scales if a person's time buys a large amount of training signal.

Carrying it over to language models

Ouyang et al. applied the same idea to language models in March 2022, starting from labeler-written prompts and prompts submitted through the OpenAI API.^[2] InstructGPT was trained in stages: supervised fine-tuning of GPT-3 on labeler demonstrations, then a dataset of rankings of model outputs used to fine-tune further with reinforcement learning from human feedback.^[2] The first stage shows the model the rough shape of a good answer; the second turns human rankings into the signal that pushes it further in that direction.

The headline result was that a much smaller model could be preferred by people. In human evaluations, outputs from the 1.3B parameter InstructGPT model were preferred to outputs from the 175B GPT-3, despite having 100x fewer parameters.^[2] InstructGPT also showed improvements in truthfulness and reductions in toxic output while having minimal performance regressions on public NLP datasets.^[2]

Why it matters

Alignment is easy to state and hard to measure. Preference comparisons give a target that can actually be collected and optimised, without the designer having to specify the correct answer in advance, and — as the reinforcement learning work showed — at a cost that fits inside an ordinary training run.^[1] Applying the recipe to a language model made the difference visible in a way a benchmark score cannot: people simply preferred talking to the smaller, tuned model.^[2]

Limits

The authors cautioned that InstructGPT still makes simple mistakes, and presented fine-tuning with human feedback as a promising direction rather than a solved problem.^[2] The signal is also indirect. What labelers prefer is a proxy for what users and designers intend, and the feedback covers a small sample of a system's possible behaviour rather than all of it.^[1] Alignment is therefore better understood as something to be measured and maintained than as a step that can be completed once.