Runway Gen-1

What Gen-1 does

Runway Gen-1 is a video-to-video model: rather than generating a clip from nothing, it takes an existing video and produces a new one guided by words or images^[1]. Runway describes the mechanism as applying the composition and style of an image or a text prompt to the structure of a source video^[5]. The model was introduced in February 2023^[1], and the company's own pitch for it was that it let someone film something new without filming anything at all^[5].

Origins

The model was described in the paper "Structure and Content-Guided Video Synthesis with Diffusion Models" by Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog and Anastasis Germanidis, posted to arXiv on 6 February 2023^[2]. Runway's standing in the field at the time was such that MIT Technology Review described it as the original startup behind Stable Diffusion^[3].

How it works

Gen-1 is a video diffusion model. A diffusion model learns to turn noise back into data — an image, or here a video — by reversing a gradual noising process, so that what starts as random static is progressively cleaned up into something coherent.

To edit footage while leaving its shape intact, Gen-1 conditions on monocular depth estimates at varying levels of detail^[2]. A depth map records how far each part of a scene sits from the camera; estimated from a single, "monocular" viewpoint, it captures the layout of a shot — where objects are and how they move — without carrying over colour or texture. Because that guidance can be supplied at different levels of detail, the model can hold on to the coarse layout of the source while replacing finer appearance. In effect the source video contributes structure and the prompt contributes content and look.

The paper also reports that the model was trained jointly on images and videos, and that it introduces a new guidance method for controlling temporal consistency^[2]. Training on still images as well as video gives the model far more material from which to learn appearance; temporal consistency is what keeps successive frames from flickering or drifting apart, which is the central difficulty in making generated video look like video rather than like a flip book of unrelated pictures.

Modes

Runway shipped the model with several modes: Stylization, Storyboard, Mask, Render and Customization^[1]. Customization is fine-tuning of the model for higher fidelity^[1]. The modes correspond to different ways of bringing a source video and a prompt together, from restyling whole shots to working within a defined region of the frame.

Evaluation

Runway reported user studies in which Gen-1's outputs were preferred 73.53% of the time over Stable Diffusion 1.5 and 88.24% of the time over Text2Live^[1]. These figures come from the company's own comparisons rather than from independent benchmarks, so they measure how the system was received by testers rather than a general ranking of video models.

Place in the line

Gen-1 belongs to the first wave of video-generation systems that anyone could actually rent and use, being counted among the first commercially available video-generation models^[4]. Its successor, Gen-2, added text-to-video, so that a clip could be prompted from a written description rather than only transformed from existing footage^[4]. That step marks the difference between Gen-1's approach — editing what was already shot — and the later pattern of generating a shot from nothing but a sentence.