Andrew Barto

Overview

Andrew G. Barto is a computer scientist known for helping to shape reinforcement learning (RL) as a general framework for machine learning. He is Professor Emeritus in the Department of Information and Computer Sciences at the University of Massachusetts, Amherst.^[1] In 2024 he was named a recipient of the ACM A.M. Turing Award together with Richard Sutton.^[1]

Education and early career

Barto earned a BS in Mathematics, with distinction, from the University of Michigan, and also received his MS and PhD in Computer and Communication Sciences there.^[1] He began his career at UMass Amherst in 1977 as a postdoctoral Research Associate. He subsequently held positions as Associate Professor, Professor, and Department Chair.^[1]

Reinforcement learning as a framework

In the early 1980s, motivated by observations from psychology, Barto and his PhD student Richard Sutton began to formulate reinforcement learning as a general problem framework.^[1] In this framework, an agent learns by acting in an environment and receiving reward signals, instead of being given a labelled correct answer for every input. The question is how an agent can improve its behavior when feedback is delayed and incomplete. For example, an agent chooses an action in some state, the environment returns a reward or penalty, and the agent uses that signal to adjust later choices. This connects machine learning with psychology, because it resembles how animals and people learn from the consequences of their actions.

Basic algorithms

Barto and Sutton, jointly and with others, developed basic algorithmic approaches for reinforcement learning.^[1] These include temporal difference learning, policy-gradient methods, and the use of neural networks to represent learned functions.^[1] Temporal difference learning updates an estimate using other estimates, so learning can proceed before the final outcome is known. Policy-gradient methods adjust the parameters of a policy in a direction expected to improve reward. Neural networks provide flexible function approximators for representing what has been learned. These are basic approaches: building blocks rather than a single complete system.

Link to neuroscience

Recent research, including work by Barto, has shown that specific RL algorithms developed in AI provide the best explanations for a wide range of findings concerning the dopamine system in the human brain.^[1] Dopamine is a neurotransmitter involved in reward-related signalling. The point is about explanation: algorithms built for artificial agents can account for patterns observed in biological reward systems. This has made reinforcement learning a shared language between AI and neuroscience.

Honors

His honors include the UMass Neurosciences Lifetime Achievement Award, the IJCAI Award for Research Excellence, and the IEEE Neural Network Society Pioneer Award.^[1] He is a Fellow of IEEE and of the AAAS.^[1]

Place in the field

Barto's work gave reinforcement learning both a problem framework and a set of basic algorithms, and later connected those algorithms to findings about the brain's reward system. The 2024 Turing Award was given to Barto and Sutton jointly.^[1]