Human-level control through deep reinforcement learning

The problem

Reinforcement learning is the branch of machine learning in which an agent is not handed the right answer but learns from reward: it acts, sees what score or feedback comes back, and adjusts. The authors of this paper noted that reinforcement learning agents had until then been limited to domains where useful features could be handcrafted by a person, or where the space of possible states was fully observed and low dimensional ^[1]. Hand-built features and small state spaces keep a learning problem tractable. Raw sensory input, such as the pixels of a screen, satisfies neither condition: it is high dimensional, and the structure that matters has to be discovered by the learner rather than supplied to it.

The approach

Volodymyr Mnih, Koray Kavukcuoglu, David Silver and colleagues built what they called a deep Q-network agent and tested it on classic Atari 2600 games ^[1]. The agent received only the pixels of the game screen and the game score as inputs ^[1]. It was given no description of the objects on screen, their positions, or the rules of the game.

The "Q" in the name refers to the value of taking a particular action in a particular state, that is, an estimate of the reward that can be expected from that point onward. Q-learning is a standard way of learning such estimates from experience. Computing them with a deep neural network means the estimate can be produced directly from raw input, so the same machinery can be pointed at any task that supplies a screen and a score.

A single algorithm, network architecture and set of hyperparameters was used across the games tested ^[1]. That detail carries much of the argument: the games differ in appearance, rules and scoring, so results obtained with one fixed configuration say something about generality rather than about tuning a system separately for each task.

The results

The paper appeared in Nature, volume 518, pages 529 to 533, on 25 February 2015 ^[1]. Across a set of 49 games, the authors reported that the agent surpassed all previous algorithms and reached a level comparable to a professional human games tester ^[1]. The comparison was made using the same algorithm, network architecture and hyperparameters throughout ^[1].

Why it mattered

The authors described the work as bridging the divide between high-dimensional sensory inputs and actions ^[1]. Going directly from what a camera or a screen delivers to what an agent should do is the hard step for anything that has to operate in a messy world, and hand-designed features had been a way of avoiding it. They described the result as the first artificial agent capable of learning to excel at a diverse array of challenging tasks ^[1].

Scope

The claim was human-level play, measured across a set of games, not on any one of them alone, and it was a comparison with a professional human tester rather than a claim of superhuman performance ^[1]. The setting is also a favourable one in a specific sense: in these games the score gives an unambiguous numerical reward, and a game can be played many times over at little cost. Most real-world tasks do not hand the learner so clean a signal, which is why the demonstration was read as a statement about learning from raw input rather than as a finished recipe for control in general.