Red teaming

What red teaming is

Red teaming is the practice of actively probing a model for harmful or unwanted behaviour before it is deployed.^[1] Instead of waiting for users to run into problems, the people doing the probing take the attacker's side: they deliberately try to make the model produce something it should not. What surfaces can be offensive language, private data that should not have been repeated, or harms that only build up over the course of a conversation.

Why the test cases need to be automated

Doing this by hand has an obvious bottleneck. Human annotation is expensive, and that cost limits both the number and the diversity of hand-written test cases.^[1] A small set of prompts written by a handful of people tends to reflect those people's own idea of what could go wrong, and nothing else.

In February 2022, Perez and colleagues published a method that turns the problem around: use one language model to generate the test cases, and aim them at another language model that is the target of the test.^[1] The generating model is asked for inputs that are likely to provoke the target into behaving harmfully; the target's replies are then examined. This replaces a slow, expensive human writing process with a machine that can produce candidate tests in bulk.

How the replies are judged

Test cases are only useful if someone reads the answers. Here that job is done by a classifier — a model trained to sort text into categories, in this case to detect offensive content. Run over the target's replies, the classifier uncovered tens of thousands of offensive replies in a 280B parameter chatbot.^[1] A classifier is cheaper and faster than a person reading each reply, but it can only flag what it was trained to recognise.

The same caveat applies to the test cases themselves, and the authors address it by varying how the cases are produced.

Generation methods and what they turned up

The paper explores several generation methods, from zero-shot generation to reinforcement learning, in order to produce test cases with varying levels of diversity and difficulty.^[1] Zero-shot generation means simply asking the generating model for a test case without first showing it examples; reinforcement learning means rewarding the generating model when the test cases it produces succeed in drawing out bad behaviour. The point of having several methods is that a narrow generator produces narrow tests.

Prompt engineering — writing the instructions to the generating model carefully — gave the authors control over what kind of test case came out. With that control they automatically found several distinct problems: groups of people that the chatbot discussed in offensive ways; personal and hospital phone numbers that it produced as its own contact information; leakage of private training data; and harms that emerge over the course of a conversation rather than in a single reply.^[1] These are different kinds of failure, and finding them with one automated pipeline is the practical argument for the approach.

Where it sits, and where it stops

The authors describe model-based red teaming as one promising tool among many that are needed for finding and fixing undesirable behaviours before they affect users.^[1] It is not a substitute for other forms of testing, and it does not close the problem on its own.

A limitation is built into the design. Because the approach relies on classifiers and prompts that were designed in advance, it finds harms that someone has already thought to specify; it does not guarantee coverage of unknown failure modes.^[1] In other words, automated red teaming can search a defined space of worries very thoroughly, but it cannot tell you that the space is complete.