Deep Q-Network (DQN)

What it is

The Deep Q-Network, usually shortened to DQN, is a reinforcement-learning agent that pairs a classical algorithm, Q-learning, with a deep convolutional neural network. Its input is not a description of the game but the picture on the screen; its output is a joystick direction. In DeepMind's later work, separate DQN agents were trained on 50 Atari 2600 games, each one receiving only raw screen pixels and a reward signal tied to the game score, and each one choosing a joystick direction as its action^[2]. The agents were trained without prior knowledge of the game rules^[2].

Where it came from

The method was first described in a paper titled "Playing Atari with Deep Reinforcement Learning", posted to arXiv on 19 December 2013 by Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra and Martin Riedmiller^[1]. The games in that first study came from the Arcade Learning Environment, a testbed built around Atari 2600 titles^[1].

How it works

Q-learning is a standard reinforcement-learning method: the agent tries to learn a value function that estimates how much reward it can expect after taking a given action in a given state, and it improves that estimate from experience by comparing what it predicted with what it actually received. Deep Q-learning replaces the table of state-action values with a neural network, so the estimate can be computed from a high-dimensional input instead of a small, hand-enumerated state space.

Because the input is an image, the network is convolutional: it builds up features from local patterns of pixels and produces a value for each possible action. Which shapes matter and which movements predict reward are learned from the pixel stream; nothing about the game is written in by hand.

Training a network this way is hard to stabilise. Consecutive frames of a game are very similar, so the network sees nearly the same example again and again, and the target it is fitting shifts as its own estimates change. A key technique in DQN is experience replay: the agent stores its experiences and then randomly samples and replays them, providing decorrelated training data^[2].

Results

The 2013 paper evaluated the method on seven Atari 2600 games from the Arcade Learning Environment; it outperformed previous approaches on six of them and surpassed a human expert on three^[1]. Using deep Q-learning with a convolutional network that learned directly from raw pixels, it surpassed human abilities in Pong, Breakout and Enduro, and beat prior state-of-the-art results on Seaquest, Beamrider and Q*bert^[3].

In the larger follow-up study, DeepMind reported that DQN reached human-level performance in almost half of the 50 games^[2].

Place in the field

DQN's importance lies in how little it was given. The same method and the same input format were applied across dozens of different games, with separate agents trained per game and no rules supplied in advance^[2]. Learning a control policy directly from pixels, rather than from features designed by a researcher for one particular game, made the Atari suite a demanding and reusable test of whether a general learning procedure could work.

Work along this line continued. In 2020 DeepMind's Agent57 reportedly exceeded human-level performance on all 57 games of the Atari 2600 suite^[3], extending the reach of the approach beyond the 50 titles used in the earlier experiments.