A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play

The question

The paper's title states its hypothesis: that one general reinforcement learning algorithm, learning through self-play, can master chess, shogi and Go. Self-play means the system learns by playing against itself — earlier versions of the program against later ones — rather than by studying games played by people. Reinforcement learning means the teaching signal is the outcome of a game, a win or a loss, rather than a human labelling each position as good or bad. Games have long served as test beds for machine intelligence because their rules are exact and the result of a game is unambiguous, which makes strength easy to measure. Putting three different games under one algorithm turns the work into a test of generality: if the same method works in all of them, it is not a bundle of game-specific tricks.

What was done

The authors presented a single algorithm, driven by self-play, and applied it to chess, shogi and Go ^[1]. The three games differ in board size, in how pieces move and in how long a game typically lasts, so a method that carries over from one to the next has to acquire its knowledge from play rather than from encoded expertise. In Go, the opponent used for the reported comparison was AlphaGo Zero ^[2].

Results

The arXiv abstract claims that within 24 hours the system reached a superhuman level of play in chess and shogi, and convincingly defeated a world-champion program in each case ^[1]. DeepMind's later account puts timings on the learning curve: in chess the system first outperformed Stockfish after just 4 hours of training, and full training took approximately 9 hours for chess, 12 hours for shogi and 13 days for Go ^[2].

The score lines reported by DeepMind are as follows. In chess, against Stockfish, the program won 155 games and lost six out of 1,000 ^[2]. In shogi, against the 2017 CSA world champion version of Elmo, it won 91.2% of games ^[2]. In Go, against AlphaGo Zero, it won 61% of games ^[2]. In each case the opponent was a strong program built for that single game.

The search figures sharpen the comparison. In chess the program searches about 60 thousand positions per second, against roughly 60 million per second for Stockfish ^[2] — about a thousandth as many, so its strength cannot be explained simply by looking at more moves.

Publication

The work was first posted to arXiv on 5 December 2017, under the v1 title "Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm"; the listing gives 13 authors, with Demis Hassabis listed last ^[1]. The Science version appeared on 7 December 2018 in volume 362, issue 6419, pages 1140–1144 ^[3], under a title that adds Go to the two games named in the preprint.

Why it matters

The weight of the result comes from several claims taken together: one algorithm rather than three, learning from self-play rather than from human games, and high strength reached in hours for chess and shogi and in days for Go ^[1][2]. The search-rate contrast gives the finding its edge. A program that examines far fewer positions per second still beat a specialised engine, which moves attention from raw computation towards how a system learns and where it spends its search ^[2]. The paper also left a set of comparable numbers — training time, game scores and positions searched per second — that later work could be measured against on the same terms ^[2].