Andrew Barto and Richard Sutton receive the 2024 ACM Turing Award

Background

Reinforcement learning is the branch of machine learning in which a program learns what to do by acting and observing the reward that follows, instead of being shown the right answer for every input. The learner has to trade off trying actions whose effects it can already predict against trying new ones that might turn out better, and it often has to credit a reward received now to a decision made much earlier. Turning that loose description into mathematics and into algorithms that actually converge is the work this award recognises.

In a series of papers beginning in the 1980s, Andrew Barto and Richard Sutton introduced the main ideas, constructed the mathematical foundations, and developed important algorithms for reinforcement learning.^[1]

The award

In 2024 the Association for Computing Machinery named Barto and Sutton as the recipients of the ACM A.M. Turing Award, for developing the conceptual and algorithmic foundations of reinforcement learning.^[1] The award carries a $1 million prize, with financial support provided by Google, Inc.^[1]

The ideas

ACM singled out temporal difference learning as their foremost contribution, alongside policy-gradient methods and the use of neural networks to represent learned functions.^[1]

Temporal difference learning addresses a practical problem: waiting until the end of a task to find out how much reward was actually collected makes learning slow, and sometimes impossible. The method updates an estimate of future reward using the difference between two successive estimates — the one made before an action and the one made after it — so the learner improves step by step, while it is still acting, rather than only at the end. The estimate is in effect taught by a later version of itself.

Policy-gradient methods take a different route. Instead of learning values for states or actions and then choosing greedily among them, they adjust the parameters of the policy directly, nudging the probabilities of the actions that were taken in the direction that made measured returns higher. This makes it natural to represent a policy with a smooth, parameterised function and to handle problems where the best action is not obvious from a value table.

Using neural networks to represent learned functions closes the loop with modern machine learning. The value estimates or the policy are stored in a network with adjustable weights rather than in a lookup table, so the same algorithms can be applied to inputs — images, board positions, sensor readings — that are far too numerous to enumerate one by one.

The textbook

Their 1998 textbook, *Reinforcement Learning: An Introduction*, is still the standard reference in the field and has been cited over 75,000 times.^[1] Generations of students learned the vocabulary of states, actions, rewards, value functions, and temporal difference updates from it, which is part of why the field shares a common notation and a common set of example problems.

The laureates

Barto is Professor Emeritus of Information and Computer Sciences at the University of Massachusetts, Amherst.^[1] Sutton is a Professor of Computer Science at the University of Alberta and a Research Scientist at Keen Technologies.^[1]

Why it matters

For a long time reinforcement learning was a specialised subject with few large-scale successes. Major practical advances came from merging it with deep learning algorithms, producing deep reinforcement learning; the most prominent example was the victory by AlphaGo over the best human Go players in 2016 and 2017.^[1] That result depended on both halves of the combination: the search over moves and the learning of values that reinforcement learning supplies, and the neural networks that let a program judge positions it had never seen.

The same toolkit now sits behind work on game playing, robot control, and the training of large models, and the vocabulary Barto and Sutton fixed — rewards, value functions, the exploration-exploitation trade-off — is how those systems are described. The Turing Award citation credits them with the foundations on which that work rests.^[1]