Training language models to follow instructions with human feedback (InstructGPT)

The problem

A language model trained to predict text is not automatically good at following instructions. GPT-3 could continue a passage plausibly, but it often did not do what the user asked, and it could produce untruthful or toxic output.^[1] The InstructGPT work aimed to make the model's behaviour match what users actually want, a goal often called alignment.

Predicting text and following instructions are different goals. A model that is good at continuing a document does not necessarily treat a request as something to carry out.

Publication

OpenAI described InstructGPT in a blog post on 27 January 2022.^[2] The paper, by Long Ouyang and colleagues, was submitted to arXiv on 4 March 2022 and published at NeurIPS 2022.^[1]

How it worked

The approach has three steps. First, GPT-3 was fine-tuned with supervised learning on demonstrations written by human labelers, which show examples of good responses to prompts.^[1]

Second, labelers ranked several model outputs for the same prompt. A separate reward model was trained on these rankings to predict which outputs people prefer, so that it could score any output.^[1]

Third, the model was optimised against the reward model using PPO, a reinforcement learning algorithm. PPO nudges the model toward outputs that score highly, and it makes only limited changes at each update.^[1]

Together, learning from human preference rankings and optimising with reinforcement learning is now often called reinforcement learning from human feedback (RLHF).

Labelers are the people who write demonstrations and compare outputs. Their judgements stand in for what users want, so the quality of this feedback shapes what the model learns.

Results

In the paper's comparisons, labelers preferred outputs from the 1.3-billion-parameter InstructGPT model over outputs from the 175-billion-parameter GPT-3.^[1] The InstructGPT models were also more truthful and less toxic in the paper's evaluations.^[1] The comparison matters because the much smaller model was the one labelers preferred. Because the reward model learns from preferences rather than from fixed rules, the same method can in principle capture qualities that are hard to write down as explicit rules. The comparisons here rest on what labelers judged to be better, so they describe human preference rather than a fixed score.

Why it matters

This is the training recipe behind ChatGPT, which was released ten months after InstructGPT.^[3]

The paper also shows that human examples and rankings, applied after the initial training, can change how a model behaves. That is a general lesson about adapting a model once its basic training is finished.

What followed

OpenAI made InstructGPT models the default in its API.^[2] Once models trained this way were served to users, the same approach shaped the chat assistants that followed.^[1]

For readers, the key change is in what the model is optimised for. Instead of only predicting the next word, it is pushed toward answers that people judged to be better.