Constitutional AI: Harmlessness from AI Feedback

The problem

The paper asks how to train a harmless AI assistant with little direct human labelling of harmful outputs. Its opening line states the goal: “As AI systems become more capable, we would like to enlist their help to supervise other AIs.”^[1]

The method

The proposed training is a process of self-improvement. The only human oversight is a list of rules or principles, rather than human labels of harmful outputs.^[1]

The method has two phases. In the supervised phase, the model critiques and revises its own responses against the principles. It is then fine-tuned on those revisions.^[1]

In the reinforcement learning phase, AI-generated preference labels replace human ones. The authors call this “RL from AI Feedback”, or RLAIF.^[1]

Results

The authors report an assistant that engages with harmful queries by explaining its objections, rather than evasively refusing. They also argue that chain-of-thought reasoning improves transparency.^[1]

Why it matters

The paper shows a way to shape a model's behaviour with a written set of principles, which is the basis of Anthropic's later approach. Anthropic's first Claude models, released in March 2023, were trained with this method.^[2][3]

Later influence

In May 2023 Anthropic published the constitution it used. Its principles drew on sources including the UN Universal Declaration of Human Rights.^[2][3]

The paper was submitted to arXiv on 15 December 2022 by 51 authors. Yuntao Bai, Saurav Kadavath, Sandipan Kundu and Amanda Askell led the work, with Jared Kaplan as last author. Dario Amodei, Chris Olah, Tom Brown, Sam McCandlish and Ben Mann are among the co-authors.^[1]