Playing Atari with Deep Reinforcement Learning

The problem the paper addressed

Reinforcement learning is a way of training an agent by trial and error: the agent acts in an environment, receives a numerical reward, and adjusts its behaviour so that the reward it collects over time increases. Deep learning is a family of methods that take raw, high-dimensional data — images, sound — and learn useful representations from it, instead of relying on descriptions designed by hand. Combining the two is awkward. Reward signals are often delayed and sparse, so the agent gets little feedback about which of its many actions actually mattered; consecutive frames of a game are strongly correlated, while neural networks are usually trained on data treated as independent; and as the agent changes its behaviour, the data it sees changes with it.

Against that background the authors made a specific claim: they presented what they described as the first deep learning model to learn control policies directly from high-dimensional sensory input using reinforcement learning ^[1]. The aim was to drop hand-built summaries of the game state and let the system act on what is on the screen. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra and Martin Riedmiller posted the paper to arXiv on 19 December 2013 ^[1].

How the model worked

The model was a convolutional neural network trained with a variant of Q-learning ^[1]. Convolutional networks are built for grid-like inputs such as images: small filters slide across the picture and respond to local patterns, and stacking such layers builds up progressively more abstract features. Here the network took raw pixels as input and output a value function that estimates future rewards ^[1]. Nothing about the game — the position of a paddle, the score, the identity of an enemy — was handed to it in coded form; the same network that looked at the screen was the one that decided what to do.

Q-learning is a classical reinforcement learning algorithm that learns how good it is to take a particular action in a particular situation. It does this by estimating the total reward expected from that point onward and repeatedly correcting the estimate toward the reward actually received plus the estimate for the situation that follows. The variant used in this paper couples that learning rule to a deep convolutional network, so the estimate is computed from pixels rather than from a table of states.

The testbed was the Arcade Learning Environment, a collection of Atari 2600 games used as a benchmark for agents that learn from the screen. The authors applied the method to seven of those games with no adjustment of the architecture or the learning algorithm ^[1]: the same network design and the same learning procedure were used throughout, and only the game changed.

Results

The authors reported that the method outperformed all previous approaches on six of the seven games ^[1]. They further reported that it surpassed a human expert on three of the games ^[1]. Read literally, the two statements together also mark the limits of the result: beating earlier methods on six games leaves one where it did not, and clearing human level on three leaves four where it did not. The reported comparison is against previous approaches and against human players, and the paper was distributed as a preprint on arXiv ^[1].

Why it matters

The significance lies in what the demonstration rules out. A great deal of earlier work on learning control assumed that someone had already decided which aspects of the situation mattered, and handed the learner a compact description; here the input was raw pixels, and the representation was learned along with the behaviour. Because the same architecture and algorithm were applied to seven different games without change ^[1], the result was not a system tuned to one title but a single procedure that had to cope with whatever appeared on the screen.

The paper also connects two strands of machine learning that had largely been pursued separately. Deep learning supplies the ability to turn high-dimensional sensory data into useful internal representations; reinforcement learning supplies the objective — future reward — that tells the representation what it is for. The claim the authors made for their model, learning control policies directly from high-dimensional sensory input ^[1], is precisely the join between the two.

What the paper does not settle is how far the approach extends. The evidence is seven games, and the headline comparison against human experts holds on three of them ^[1]; the authors' description of the model as the first of its kind is their own characterisation of the work rather than a measured quantity. Those qualifications matter for reading the result: the contribution is a demonstration that a single network can go from pixels to actions under a reward signal, and a clear statement of where that demonstration did and did not reach.