Deep reinforcement learning from human preferences

The problem

Reinforcement learning trains an agent by trial and error. The agent acts, receives a reward signal, and gradually favours the actions that lead to higher reward. The method is only as good as the reward. Many goals people care about are hard to write down as a formula, such as a robot that behaves safely or a system that gives helpful answers. When a reward is written by hand, the agent often finds ways to score well without doing what its designers intended. Hand-written rewards are often gamed, and this is the long-standing weakness the paper addressed. ^[1]

The approach

The paper was submitted to arXiv on 12 June 2017, the same day as the Transformer paper, and it takes a different route. Instead of writing the reward, the authors asked people to compare short video clips of the agent's behaviour and say which one looked better. A separate reward model, itself a neural network, was trained to predict these human preferences. The agent was then trained with ordinary reinforcement learning against this learned reward. ^[1]

The idea splits the work in two. People judge outcomes they can recognise, and the machine turns many such judgements into a scoring function that guides its own practice. Comparing two options is often easier for people than assigning either one an exact number, which is why preference feedback is attractive. The authors were Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg and Dario Amodei, from OpenAI and DeepMind. ^[1]

The difference is easiest to see with an example. Suppose a designer wants a cleaning robot to keep a room tidy, and writes a reward that counts how many objects are out of sight. A robot can score well on that measure by pushing everything under the furniture, which is not what anyone wanted. Comparing two short clips of behaviour is a different kind of task: a person can notice that one attempt is clearly sloppier than another without first writing down a complete definition of tidiness.

Results

The agent learned to play Atari games and to control simulated robots performing locomotion tasks, all without access to the true reward. Human feedback accounted for less than one percent of the agent's interactions. The method also taught a new behaviour: a simulated backflip, learned from about an hour of human time. ^[1]

Why it matters

The paper showed that people can steer a learning system by comparing its behaviour, without writing down a precise score in advance. ^[1] That idea was later applied to language models, in InstructGPT (2022) and then ChatGPT. It is the 'RL' in RLHF, reinforcement learning from human feedback. ^[2]

What followed

The basic loop of collecting comparisons, training a reward model and optimising against it became a template for later work. The loop also raised a concern. A learned reward is only an approximation of what people want, and a system can learn to exploit its flaws. Much of the later work on alignment has had to deal with exactly this problem.