OpenAI announces its Superalignment team

Background

Alignment is the problem of making an AI system pursue what its designers and users actually intend. The most common current method is reinforcement learning from human feedback (RLHF): people rank or score several of a model's answers, and the model is then adjusted toward the answers people preferred. The premise is that a human can tell a good answer from a bad one. In its announcement, OpenAI said superintelligence may arrive within this decade, and that existing techniques relying on human supervision, such as RLHF, would not scale to superintelligence ^[1]. The reasoning is not complicated: if a system is stronger than its supervisors in some area, its answers can go beyond what those supervisors are able to judge, and their approval stops being a dependable training signal.

What was announced

On July 5, 2023, OpenAI announced the creation of a Superalignment team ^[1]. The team was co-led by OpenAI co-founder and chief scientist Ilya Sutskever and alignment lead Jan Leike ^[1]. OpenAI said it would dedicate 20% of the compute it had secured to date, over the next four years, to the team's work ^[1]. The stated goal was to solve the core technical challenges of superintelligence alignment in four years ^[1].

Putting a share of compute, named leaders and a deadline into one public statement is what made the announcement notable: it treated alignment as an engineering problem with dedicated resources and a schedule, rather than as an accompanying safety concern.

The research plan

As described at the time, the team aimed to build a roughly human-level automated alignment researcher and then use vast amounts of compute to iteratively align superintelligence ^[1]. In other words, AI systems would take part in solving the alignment problem for AI.

The plan covered three areas ^[1]. The first is scalable oversight: finding ways to supervise a stronger system effectively even when humans cannot directly check whether its answers are right. The second is robustness and automated interpretability. Robustness asks whether a system stays reliable when inputs differ from its training distribution or are deliberately chosen to break it; automated interpretability uses tools to look inside a model and understand why it produced an output, rather than only studying inputs and outputs. The third is adversarial testing, and it included a particular practice: deliberately training misaligned models in order to verify that detection techniques actually work ^[1]. That is close to running a control experiment for safety checks — if a method cannot catch a bad model that was built on purpose, there is little reason to trust it in the wild.

Why it matters

The announcement mattered in two ways. One is the timescale it put on the record: the statement's judgment was that superintelligence may arrive within this decade, while the team gave itself four years ^[1]. The other is the methodological direction. If human supervision cannot keep pace with model capability, part of the supervision has to be handed to AI systems themselves, and that is exactly where the idea of an automated alignment researcher sits ^[1].

What such an announcement sets out is a goal and a plan, not a result. The four-year deadline, the compute commitment and the research directions were arrangements made at the time of the announcement; whether they hold depends on the research itself.