InstructGPT

What it is

InstructGPT is a set of GPT-3 models trained to follow instructions. OpenAI announced them on 27 January 2022 in a post titled "Aligning language models to follow instructions" ^[1]. The method used reinforcement learning from human feedback (RLHF), in which people's judgements of model outputs are used to steer the model toward better answers.

OpenAI said the InstructGPT models were much better at following user intentions than GPT-3. They were also more truthful and less toxic, and OpenAI deployed them as the default language models in its API ^[1].

Origins

The problem the work addressed was simple to state. GPT-3 continued text plausibly, but it often did not do what the user asked, and it could produce untruthful or toxic output ^[2]. A model that predicts likely text is not the same as a model that helps with a request, and InstructGPT was built to close that gap.

The paper "Training language models to follow instructions with human feedback" was posted to arXiv on 4 March 2022. Its first author was Long Ouyang, with co-authors including John Schulman, Paul Christiano, Jan Leike and Ryan Lowe. It was published at NeurIPS 2022 ^[2].

Main features

The training had three steps. First, GPT-3 was fine-tuned with supervised learning on demonstrations written by human labellers. Second, a reward model was trained on labellers' rankings of model outputs. Third, the model was optimised against that reward model with the PPO reinforcement-learning algorithm ^[2].

The reward model is the part that turns human preferences into a signal the training can use. Proximal Policy Optimization (PPO) is the reinforcement-learning method that adjusts the model in small steps so that it earns higher reward without drifting far from its starting behaviour.

The prompts came both from labellers and from submissions to the OpenAI API ^[2]. Using real API prompts meant the training reflected the kinds of requests that users were actually making.

Results

In human evaluations, outputs from the 1.3-billion-parameter InstructGPT model were preferred to those of the 175-billion-parameter GPT-3, despite the InstructGPT model having about 100 times fewer parameters ^[2]. This result is a central point of the paper: better training on human judgements can matter more than raw size.

The paper reported better truthfulness and less toxic output, with minimal regressions on public NLP datasets. It also noted that the models still make simple mistakes ^[2].

Later history

When OpenAI introduced ChatGPT in November 2022, it described the chatbot as a sibling model to InstructGPT, trained with the same RLHF methods ^[4].

The original InstructGPT API models were text-ada-001, text-babbage-001, text-curie-001, text-davinci-001, text-davinci-002 and text-davinci-003. OpenAI shut them down on 4 January 2024 ^[3].

Why it matters

InstructGPT is an early and influential example of aligning a language model with human preferences. The idea that a model could be trained to follow instructions through human feedback, rather than only by predicting text, is the background to the ChatGPT entry.