Genie

What it is

Genie is a generative world model from Google DeepMind, described in the paper "Genie: Generative Interactive Environments", which was posted to arXiv on 23 February 2024.^[1] A world model, broadly speaking, is a model that learns to predict how an environment will change: given the current view and an action, it predicts what comes next. What makes Genie distinctive is that it turns this into something playable — the frames it produces are not a fixed video but a virtual world that changes from frame to frame in response to what the user does.^[1]

The authors described it as the first generative interactive environment trained unsupervised on unlabelled internet videos, without ground-truth action labels.^[1] The difficult part is the absence of action labels. Training a model to respond to controls normally requires paired data: a frame, plus a record of which key was pressed at that moment. Gameplay videos on the internet contain only pixels; nobody recorded what the player was doing.

How it works

Genie has 11 billion parameters and consists of three parts: a spatiotemporal video tokenizer, an autoregressive dynamics model and a latent action model.^[1] The tokenizer compresses video into discrete tokens that the rest of the system can work with, and the dynamics model predicts the tokens for what comes next, one step at a time; it is a decoder-only MaskGIT.^[3] The third part does the work that the missing labels would otherwise do: with no recorded button presses, the model has to derive an implicit representation of actions on its own, associate the change between frames with one of these abstract operations, and let the dynamics model use that as input.^[1]

Genie was trained on a filtered set of 30,000 hours of internet gameplay videos.^[3]

What it is used for

Users can prompt Genie with text, synthetic images, photographs or sketches to generate controllable virtual worlds and act in them frame by frame.^[1] Because the prompt can be a sketch or a photograph, the model is not limited to reproducing the games in its training data; a still image is enough to define a new environment to move around in.

The paper also reports experiments outside games. It describes training a 2.5-billion-parameter model on a robotics dataset, and a separate model trained on action-free robot videos from the RT1 dataset.^[3] These are tests of whether the same approach works for a different kind of visual sequence.

Limits

Speed is the obvious one. Genie ran at around 1 FPS, and the authors said future advances are needed to reach an efficient frame rate for interaction.^[3] A world model that updates about once a second is a research demonstration rather than something one can play in the ordinary sense.

Where it sits

Genie was followed by Genie 2 in December 2024, which generates playable 3D environments.^[2] The line of work it belongs to treats video as a source of training signal for models of environments that can be interacted with, rather than only watched.