Reinforcement learning from human feedback (RLHF)

What it is

Reinforcement learning from human feedback, usually shortened to RLHF, is a technique for training a model toward what people prefer when that goal is hard to write down as a formula. Instead of a mathematical score, the training signal comes from human judgments about which output is better.^[1]

How it works

The basic loop has three parts. First, people compare pairs of model outputs and say which one they prefer. Second, a separate reward model is trained to predict those preferences. Third, the main model is optimized with reinforcement learning to score highly on the reward model.^[1]

Imagine a model that writes two possible replies to the same question, and a person picks the clearer one. After many such comparisons, the reward model learns a rough sense of what people like. The main model is then nudged toward replies the reward model scores well. The reward model is only a stand-in for human judgment, so it can be no better than the comparisons used to train it.

Many goals of an assistant are easy to recognize but hard to specify. Clarity and tone are examples. People can judge a pair of answers quickly even when they could not write a rule that always picks the better one. RLHF turns those quick judgments into a training signal.

Where it came from

In June 2017, Christiano and colleagues at OpenAI and DeepMind showed the method on Atari games and simulated robot tasks. They used human feedback on less than one percent of the agent's interactions.^[1]

In March 2022, OpenAI's InstructGPT paper applied the method to GPT-3 in three steps. First, the model was fine-tuned on demonstrations written by human labelers. Second, a reward model was trained on the labelers' rankings of several outputs. Third, the model was optimized against that reward model with the PPO reinforcement learning algorithm.^[2] In the paper's results, labelers preferred the outputs of a 1.3-billion-parameter InstructGPT model over those of the 175-billion-parameter GPT-3.^[2]

Where it is used today

ChatGPT, released in November 2022, was trained with this method, and RLHF became one of the most widely used ways of training chat assistants.^[3]

Limits

The model can learn to exploit weaknesses in the reward model, scoring well on the proxy without truly producing what people want. Human preferences can also reward answers that sound confident, even when they are not accurate. Researchers have proposed variants that rely less on direct human labels, such as methods that use AI feedback guided by written principles.^[3]

The approach also depends on the quality and consistency of the human judgments that feed it. Labelers may disagree, so the signal reflects the preferences of the particular people who made the comparisons.