OpenAI Gym

Gym is a toolkit for developing and comparing reinforcement learning algorithms ^[1]. Reinforcement learning is the branch of machine learning in which a program is not shown the right answer but learns by acting: it tries actions in an environment, receives a reward signal, and gradually shifts its behaviour so that the reward it collects over time grows. Judging such a program is awkward, because how well it does depends as much on the environment it was trained and tested in as on the algorithm itself. Gym's answer was to standardise both sides of that experiment.

Where it came from

OpenAI announced Gym as a beta on 27 April 2016, in a post by co-founder Greg Brockman ^[1]. OpenAI said it had built the toolkit to speed up its own reinforcement learning research, and hoped it would serve the wider community as well ^[1].

Two parts

Gym has two halves. The first is a growing suite of environments, ranging from simulated robots to Atari games. The second is a website for comparing and reproducing results ^[1]. The environments are written in Python ^[1], and the toolkit is designed to work with algorithms written in any framework, with TensorFlow and Theano named as examples ^[1]. That last point matters in practice: a researcher who has already built a learning system in one framework should not have to rewrite it in order to test it on a standard task.

The first release

The first release grouped environments into five collections: classic control and toy text, algorithmic, Atari, board games, and 2D and 3D robots ^[1]. The collections differ in what they demand of a learner. The Atari environments integrate the Arcade Learning Environment ^[1], so the same set of pixel-based video games used in earlier benchmark work could be reached through Gym's interface. The robot environments rely on the MuJoCo physics engine ^[1], a simulator that models the contact forces and joint dynamics of a moving body. The board-game environments offer 9x9 and 19x19 Go against the Pachi engine ^[1], giving a game with a very large space of possible positions alongside the smaller control tasks.

Versions and comparability

Every environment carries a version number, which is incremented whenever the environment changes, so that results stay comparable ^[1]. This is a small design decision with large consequences. If a task's dynamics, starting conditions or reward scale are quietly edited, numbers reported before and after the edit are no longer measuring the same thing. Tying a version number to each environment lets a reported score be matched to the exact version it was produced on.

A curated list rather than a leaderboard

OpenAI said it would not run a leaderboard. Instead it would keep a curated list of contributions, and it said it valued generality of technique over raw scores ^[1]. The reasoning behind that choice is visible in the two parts of the toolkit: an algorithm that posts a high number on one game while needing hand-tuning for every new task is less useful than one that works across a range of them, and a curated list can record what a method actually needed in order to work, which a bare score cannot.

Where it sits

Gym was released early, as a beta, and was presented as infrastructure rather than as a result: not a new learning algorithm, but the shared ground on which algorithms could be tried and compared. Its components map onto the practical problems of the field at the time — environments scattered across separate codebases, results that were hard to reproduce, algorithms locked into particular frameworks, and benchmarks that rewarded narrow tuning. The suite of environments, the framework-independent interface, the per-environment version numbers and the preference for a curated list over a leaderboard are all answers to one of those problems ^[1].