AlphaZero

What it is

AlphaZero is a general game-playing system from DeepMind, first described in a preprint posted to arXiv on 5 December 2017 ^[1]. Its subject is not one game but three: chess, shogi and Go. The same algorithm, with the same settings, was applied to all three, and the only thing it was given about each game was its rules ^[1]. Starting from random play, it reached a superhuman level in all three within 24 hours ^[1].

Where it came from

The work generalised the self-play reinforcement-learning approach of AlphaGo Zero — an earlier system built for Go — to games beyond Go ^[1]. David Silver led its development ^[3]. The preprint lists 13 authors, among them Demis Hassabis, Thore Graepel and Karen Simonyan ^[1].

Chess was a pointed choice of target. The preprint notes that chess is the most widely studied domain in the history of AI ^[1], and that the strongest chess programs rely on search techniques, domain-specific adaptations and hand-crafted evaluation functions ^[1]. In other words, the best chess software carried a great deal of knowledge that people had put in by hand, above all about what a good position looks like. AlphaZero was presented as a test of whether one general method could do without that.

How it learns

At the centre of the system is self-play: the program plays games against itself, beginning with moves chosen at random. When a game ends, its result — a win, a loss or a draw — is used as a training signal, making the choices that led to that result somewhat more or less likely to be repeated. Each round of games produces a slightly better player, which then generates the games for the next round. This is reinforcement learning in the plain sense of the term: no one tells the system what a good move is, only whether the game was won.

Because nothing beyond the rules was supplied, any understanding of openings, piece activity or positional soundness had to be produced by the system itself in the course of this loop. The same recipe was run on three games with quite different rule sets, rather than being rebuilt separately for each one.

Results

Within 24 hours of starting from random play, the system reached a superhuman level in chess, shogi and Go ^[1]. It convincingly defeated a world-champion program in each of the three games ^[1]. The point of running all three was the comparison: one algorithm, one set of settings, the same starting condition, three different games.

What came after

AlphaZero was followed by MuZero, described in a preprint in November 2019, which matched AlphaZero's performance without being given the game rules ^[2]. The MuZero preprint states that MuZero matched AlphaZero on Go, chess and shogi, even though AlphaZero had been given the rules of those games ^[2]. The direction of travel is visible in the pair: the same self-play recipe, with less and less supplied from outside.

Why it matters

AlphaZero's contribution is easiest to state as a removal. Chess, the most studied domain in the history of AI, had for a long time been approached with hand-built evaluation and hand-tuned search ^[1]; AlphaZero reached superhuman strength in it, and in two other games, from the rules alone and within a day ^[1]. That made the case that a single learning method, given enough self-play, can substitute for the domain knowledge that had been the main currency of game-playing AI. It also set the pattern for the line of work that came after it, in which each system is asked to do the same with less input ^[2].