Reward hacking

What it is

A reward function is a number that a learning system is trained to make large. Often the number we can actually write down is not the thing we care about; the written stand-in is called a *proxy* reward, and the thing we really want is the *true* reward. Reward hacking occurs when optimising an imperfect proxy reward function leads to poor performance according to the true reward function ^[1]. Nobody in this story is misbehaving: the optimiser does exactly what it was asked to do. The failure is in the asking, because the proxy is written by people who cannot fully specify what they want, and the system then searches wherever the proxy leaves room.

How it works

A hypothetical example makes the shape of the problem clear. Suppose someone wants a robot to keep a workshop tidy and defines the reward as the number of tools lying inside the storage bins. A policy that sweeps every tool into a bin — including the ones currently in use, and the ones that belong elsewhere — collects a high score while the workshop becomes no more usable than before. The proxy measured "tools in bins"; the true objective was "workshop usable".

The formal version of the idea is stated in terms of expected returns. A proxy reward function is called unhackable, relative to a true reward function, if increasing the expected proxy return can never decrease the expected true return ^[1]. This has to hold across the policies under consideration, which makes it a very strong requirement: not merely that the current policy is fine, but that no policy which scores better on the proxy scores worse on what we actually want.

Where the concept comes from

Skalse et al. published the first formal definition of the phenomenon in September 2022 ^[1]. Having defined unhackability, the paper asks whether a designer can escape hacking by constructing the proxy carefully. Two natural strategies suggest themselves. One is to leave terms out of the reward — score only what can be measured, and trust that the rest will follow. The other is to blur fine-grained distinctions between outcomes that are roughly equivalent, on the theory that the difference does not matter. The paper shows that building an unhackable proxy by leaving terms out of the reward, or by overlooking fine-grained distinctions between roughly equivalent outcomes, usually does not work ^[1].

The reason has to do with how returns are computed. Expected reward is linear in state-action visit counts: the return of a policy is a weighted sum in which the weights record how often each state-action pair is visited. A key insight of the paper is that this linearity makes unhackability a very strong condition ^[1]. Cheap edits to a reward function — dropping a term, coarsening a distinction — change those weights, and the system shifts probability mass toward whatever now scores well.

The sharpest result is negative. For the set of all stochastic policies, two reward functions can be unhackable only if one of them is constant ^[1]. In other words, if the system is free to randomise over its behaviour, the only proxy that can never be gamed is one that hands out the same reward everywhere — and such a reward says nothing about what to do.

The picture is not uniformly negative. Non-trivial unhackable pairs do always exist for deterministic policies and for finite sets of stochastic policies, and the paper gives necessary and sufficient conditions for an important special case it calls simplifications ^[1]. What makes a proxy safe, then, is restriction: limiting what the system can do, or limiting the set of policies over which the comparison is made.

Why it matters

The results reveal a tension between using reward functions to specify narrow tasks and aligning AI systems with human values ^[1]. A reward function is a good instrument for a narrow, well-defined task, because the target can be written down precisely. But the more narrowly a target is specified, the more room there is for a system to satisfy the letter of it and miss the point; and objectives stated at the level of human values are both harder to write down and, on these results, unlikely to admit unhackable proxies. The concept gives that intuition a precise meaning and explains why careful reward design alone cannot be relied on to close the gap.

Limits

The definition characterises relationships between pairs of reward functions, not the behaviour of any particular training algorithm. Two reward functions can be unhackable in this formal sense and still be the wrong objective: unhackability concerns whether optimising the proxy ever hurts the true reward, not whether the true reward is the one worth wanting. The strongest negative result applies to the set of all stochastic policies, while the positive results hold for restricted sets — which moves the difficulty elsewhere, to choosing the restriction well.