Mastering the game of Go without human knowledge

The question the paper asked

Earlier versions of AlphaGo were trained on games played by humans, DeepMind says ^[1]. That leaves an obvious question: how much of the playing strength comes from those human examples, and how much could be recovered without them? The title of the Nature paper puts the question directly — mastering the game of Go *without human knowledge* ^[2]. Go is a board game in which the number of possible positions is enormous compared with chess, and strength is judged over the whole board rather than by material captured, which is why it resisted strong programs for a long time.

How AlphaGo Zero learned

AlphaGo Zero starts from random play, with no human game records to learn from, and improves by playing games against itself ^[1]. The mechanism is a loop: the system plays a game, sees who won, and adjusts so that the moves that appeared in winning games become somewhat more likely and the moves that appeared in losing games become less likely. Nothing needs to be labelled by a person; the only signal is the result at the end of the game. Repeated over a very large number of games, this is enough to build up strength from nothing.

The scale DeepMind describes is large. Over a few days the system played millions of AlphaGo-versus-AlphaGo games, which the company characterises as accumulating thousands of years of human knowledge in that time — the point being that the knowledge is generated rather than supplied ^[1].

Results

According to DeepMind, AlphaGo Zero beat the previously published, champion-defeating version of AlphaGo by 100 games to 0, and that result came after only three days of self-play training ^[1]. Training continued: after 40 days of self-training, DeepMind says, AlphaGo Zero was stronger than the version known as AlphaGo Master, which had defeated world number one Ke Jie ^[1].

The comparison was not only about playing strength but also about computing hardware. A chart in the DeepMind post lists AlphaGo Zero and AlphaGo Master as each using 4 TPUs, against 176 GPUs for AlphaGo Fan and 48 TPUs for AlphaGo Lee ^[1].

Authors and publication

The paper was published in Nature on 19 October 2017, in volume 550, pages 354–359 ^[2]. Its author list includes David Silver, Julian Schrittwieser and Karen Simonyan, with Demis Hassabis listed last ^[2]. DeepMind's blog post about the work, written by David Silver and Demis Hassabis, is dated 18 October 2017 ^[1].

Why it matters

The result turned a claim about method into something testable. If an earlier AlphaGo needed human games and AlphaGo Zero did not, then the human records were not where the strength had to come from: a system that starts from random play and trains against itself can reach and pass the level of a program that had beaten a champion ^[1]. That is the sense in which the paper is about more than Go — it is a demonstration that the human examples, rather than the search and the learning, can be removed from the recipe. The hardware figures tell the other half of the story: the later systems did this with fewer accelerators than the earlier ones ^[1], so the gain came from the training procedure rather than simply from more computing.