AlphaStar

What it is

AlphaStar is an artificial intelligence system built by DeepMind to play StarCraft II, a real-time strategy game in which two players gather resources, build bases, produce units and fight for control of the map. Unlike a turn-based board game, both sides act at the same time and neither can see everything the opponent is doing, so a player has to decide with incomplete information and with a very large set of possible actions. AlphaStar played all three of the game's races — Protoss, Terran and Zerg — using one neural network per race^[2].

Where it came from

DeepMind's collaboration with the game's maker, Blizzard, was announced at BlizzCon in November 2016^[1]. In August 2017 the two organisations released tools for bot development together with data from 65,000 historical games^[1]. AlphaStar itself was introduced in January 2019, when it won 10 consecutive matches against two professional players; at that point, however, it could see the whole map, which a human player cannot, and a later version without that advantage lost a match^[3]. The research group was led by Oriol Vinyals^[4].

How it was trained

Training began with imitation learning from human players, which was used to build an initial policy. That initial policy on its own already played better than 84% of active players^[2]. The second stage was league-based multi-agent reinforcement learning, in which many agents are trained against one another. Within the league, main agents aim to win against everyone, while exploiter agents exist to expose the main agent's weaknesses, so the system keeps running into opponents that punish its habits^[2].

The interface was also narrowed to something closer to a human's. AlphaStar played through a camera interface and under action-rate limits that a professional player approved^[2], with agents capped at a maximum of 22 actions per 5 seconds^[2].

Results

From July 2019 a constrained version played anonymously on the public European 1v1 ladder^[1]. On 30 October 2019 DeepMind reported, with a paper in Nature, that AlphaStar had reached Grandmaster level for all three races, ranking above 99.8% of active players on Battle.net^[2]. The 2019 version played anonymously on Battle.net under the same maps and conditions as human players^[2]. DeepMind said AlphaStar was the first AI to reach the top league of a widely popular esport without any game restrictions^[2].

Why it matters

Real-time strategy games are hard to test on because the state of the game is only partly visible, the number of possible actions is huge, and a single decision can take many seconds to pay off. AlphaStar's results were built on two different mechanisms working in sequence: copying what humans already do gives a starting point that is already competent, and league training then pushes past that starting point by repeatedly creating opponents that exploit the current agent's weaknesses^[2]. Playing anonymously on a public ladder, against people who did not know they were facing a machine and under the same maps and conditions as everyone else, is a different kind of evidence from a staged exhibition match^[1][2]. The version that appeared on the ladder was deliberately restricted in what it could see and how fast it could act, so its rating was earned under conditions a professional player had approved^[2].

AlphaStar therefore stands as a demonstration that a single learned system can play a complex, imperfect-information game at a level near the top of the human ladder, across three different ways of playing the same game^[2].