OpenAI Universe

What it was

On 5 December 2016, OpenAI released Universe, a software platform for measuring and training an AI's general intelligence across games, websites and other applications^[1]. The aim OpenAI stated for it was concrete: an agent should use a computer the way a person does, seeing the screen as pixels and operating a virtual keyboard and mouse^[1]. The target was not a tidy table of scores but the same unruly interfaces people already use.

What shipped with it

The release included 1,000 environments^[1]. Much of that bulk came from the web: 1,000 Flash games were selected from more than 30,000, and 100 of them shipped with reward functions^[1]. The gap between those two numbers says something about the work involved — a game built for people carries no machine-readable score unless someone adds one, so only part of the selection came with rewards attached.

The collection also included Atari 2600 games from the Arcade Learning Environment and browser tasks such as the Mini World of Bits benchmark, which contributed 80 environments^[1]. OpenAI said hundreds of the environments were ready for reinforcement learning^[1], a smaller number than the thousand shipped, which indicates that the rest were there to be worked on rather than simply trained against.

Partners providing games or applications included EA, Microsoft Studios, Valve, Wolfram, NVIDIA and Kongregate^[1].

How it worked

Universe builds on Gym, released in April 2016, and lets any program be turned into a Gym environment^[1]. The two connect at import time: importing the universe library registers its environments with Gym automatically, so they can be created with a single call^[1]. From the point of view of a training script, a Flash game and an Atari game are reached the same way.

Underneath, Universe runs each program behind a VNC remote desktop, so it needs no access to the program's internals, source code or bot API^[1]. VNC is a protocol for viewing and controlling a remote screen; by putting every application behind it, Universe could take software that was never designed to be learned against and expose it as though a person were sitting at it. The agent receives pixels and sends keyboard and mouse events, and nothing in the middle requires cooperation from the application.

Each environment is packaged as a Docker image hosting a VNC server for pixels and input and a WebSocket server for rewards and control messages^[1]. Docker gives every environment a sealed container with its dependencies inside, so many of them can coexist without conflicting over libraries. The split between the two servers matters as well: one channel carries what the agent sees and does, the other carries the reward and the control messages a training loop needs.

Where it fit

Universe did not replace Gym; it fed it. Gym gave reinforcement learning a common interface for environments, and Universe extended the reach of that interface to arbitrary software^[1]. Anyone with a program — a game, a website, a simulation — could wrap it and get back an environment that behaved like the rest.

The design also set out the terms of the problem. Because rewards had to be added by hand for most applications, and because only 100 of the Flash games shipped with reward functions^[1], Universe made plain how much learning on real software depends on defining what counts as success. And because it asked for general intelligence measured across games, websites and other applications^[1], it framed progress as breadth rather than a single high score.