OpenAI introduces InstructGPT and makes it the default in its API

Background

By 2022 large language models (LLM) such as GPT-3 could continue a passage of text fluently, but that did not mean they reliably did what a user asked. A model trained only to predict the next word in text from the web might answer off-topic, state falsehoods or produce toxic content. Alignment is the work of adjusting a model so that its behaviour matches what people actually want. OpenAI's alignment research was addressing this problem, and InstructGPT applied techniques from that research.^[1]

What happened

On 27 January 2022 OpenAI announced InstructGPT in a post titled “Aligning language models to follow instructions.”^[1] The company said the models were much better than GPT-3 at following what users intended, while also being more truthful and less toxic.^[1] The InstructGPT models, trained with humans in the loop, became the default language models on OpenAI's API.^[1]

The research was described in a paper, “Training language models to follow instructions with human feedback,” posted to arXiv on 4 March 2022. Its first author was Long Ouyang, and co-authors included John Schulman, Paul Christiano, Jan Leike and Ryan Lowe.^[2]

Key details

The method had three stages. First, starting from prompts written by labelers and prompts submitted through the OpenAI API, the team collected demonstrations written by labelers and used them to fine-tune GPT-3 with supervised learning. Fine-tuning means continuing training on a smaller, curated dataset so that a model's behaviour shifts. Second, the team collected rankings of several model outputs for the same prompt. Third, it further tuned the model with reinforcement learning from human feedback (RLHF), a method that uses human judgements as the training signal.^[2]

A notable result came from human evaluations. Outputs from the 1.3-billion-parameter InstructGPT model were preferred over those of the 175-billion-parameter GPT-3, even though the InstructGPT model had about 100 times fewer parameters. Parameters are the numerical values a model learns during training, and a larger parameter count generally means a larger model. The comparison suggests that the training method can matter as much as size, at least for the tasks that were measured.^[2]

The paper also reported improvements in truthfulness and reductions in toxic output on public NLP datasets, with minimal performance regressions. It noted that the models still make simple mistakes.^[2]

Impact

Making InstructGPT the default meant that applications built on OpenAI's API would, by default, use a model tuned this way rather than the original GPT-3 models.^[1]

In November 2022 OpenAI introduced ChatGPT and described it as a sibling model to InstructGPT, trained with RLHF using the same methods but with slight differences in data collection.^[3] The approach documented in the InstructGPT paper therefore carried into a conversational product that the public could use directly. The work also illustrates that how a model is trained, not only how large it is, shapes how useful it is in practice.