OpenAI Five

What it is

OpenAI Five is a team of five neural-network bots that play the video game Dota 2, developed by OpenAI and trained with reinforcement learning ^[1]. Dota 2 is a five-a-side game in which each player controls a single hero, the two sides compete over objectives across a map they cannot fully see, and a match runs for tens of minutes. Long time horizons, hidden information and a very large space of possible actions make it a hard game for a machine to play well.

Each of the five bots was driven by a single-layer LSTM network with 4,096 units ^[1]. That is a comparatively small network; what set the system apart was not the size of the model but the amount of experience it was given.

Origins

The project's first public demonstration was at The International 2017 in August, where the professional player Dendi lost a live one-on-one match to a bot ^[1]. Beating a single professional player in a simplified one-on-one setting was a narrow test, and full five-bot team play against a human side was reported in June 2018 ^[1].

How it was trained

Reinforcement learning means the system is given a goal and a reward signal and improves through trial and error, instead of being taught from labelled examples. For OpenAI Five the main source of practice was self-play: the bots played against copies of themselves, and the resulting stronger versions became the next opponents.

That approach is expensive. By 2018 training ran on 256 Nvidia P100 GPUs on Google Cloud Platform and 128,000 pre-emptible CPU cores ^[1]. The 2018 system played around 180 years' worth of games against itself per day ^[1]. OpenAI said the version current at The International in August 2018 had experienced about 10,000 years of Dota self-play over 1.5 realtime months, and that by April 2019 the total had reached about 45,000 years over 10 realtime months, an average of 250 years of simulated experience per day ^[5].

Jakub Pachocki, later OpenAI's chief scientist, led the development as research director, and Greg Brockman also led the project ^[2][3].

Results

At The International 2018 the bots lost both of their games, against paiN Gaming and an all-star team of former Chinese players ^[1]. The following spring the result reversed: on 13 April 2019 in San Francisco they won a best-of-three series against OG, the reigning champions of The International 2018 ^[1].

A wider public test came during a four-day online event in April 2019, when they played 42,729 public games and won 99.4% of them ^[1]. The arXiv paper gives the same win rate for that showcase, describing it as 3,193 teams across 7,257 games ^[4].

Why it matters

The work was documented in the paper "Dota 2 with Large Scale Deep Reinforcement Learning", first submitted on 13 December 2019, which lists Greg Brockman and Jakub Pachocki among its OpenAI authors ^[4]. The paper states that OpenAI Five became the first AI system to defeat the world champions at an esports game ^[4].

Beyond the headline result, the project was a large-scale test of a simple idea: that one learning system, given enough self-play and enough compute, could reach a high level in a team game whose strategy is normally written by hand. The published account of how the training was scaled, and of what it cost in hardware and simulated play time, is part of what later work in the field could build on.