Runway introduces Gen-1 for video-to-video generation

Background

Image generation models had become good at turning a text prompt into a single picture. Video is harder. A clip is a long sequence of frames, and a model that treats each frame independently has no way of keeping a surface the same colour from one frame to the next; the eye reads the result as flicker rather than as footage. Video-to-video generation addresses this by taking existing footage as input and producing new footage with a different appearance while keeping the motion and layout of the original. That places the task closer to editing and post-production than to generation from nothing: restyling a shot, or previewing how a scene might look under a different visual treatment.

What Runway released

In February 2023, Runway introduced Gen-1, a model that generates new videos from existing ones using text or image prompts^[1]. A text prompt describes the look the user wants; an image prompt supplies a reference image instead. Either way the source clip remains the input, and the output is a new clip derived from it. Alongside the product, Runway published a research paper describing the method, posted to arXiv on 6 February 2023^[2].

How it works

The model conditions on depth estimates taken from the source video^[2]. A depth estimate is a per-pixel map of how far away each surface appears to be — near surfaces one value, distant surfaces another — computed from the footage itself. Depth carries the geometry of a scene but not its colours or textures, so using it as a conditioning signal tells the model what has to stay put: the outline of a person, the layout of a room, the direction the camera travels. Colour, material and style can then be regenerated to match the prompt without the composition falling apart.

Modes

Gen-1 shipped with several modes: Stylization, Storyboard, Mask, Render and Customization^[1]. They correspond to different ways of steering the output — applying a look, working from a storyboard, restricting changes to a masked region, rendering, or adapting the model to particular footage. In Customization mode, users could fine-tune the model on their own footage to get higher fidelity^[4]. Fine-tuning means continuing to train a model briefly on a small, specific set of material so that it reproduces that material more faithfully.

Evaluation

Runway reported results from user studies comparing Gen-1 with earlier tools. Users preferred its output over Stable Diffusion 1.5 in 73.53% of comparisons^[1], and over Text2Live in 88.24%^[4]. These figures come from Runway's own user studies and measure human preference between paired outputs.

What it changed

Gen-1 sits between generating still images and generating and editing video. The emphasis on holding structure with depth is what separates it from applying a style filter frame by frame: a per-frame filter can change colours but cannot guarantee that anything stays consistent across the cut, whereas treating structure as a separate conditioning signal lets the rewritten footage still read as the same shot. Framed that way, video-to-video puts a generative model in the position of a post-production tool — you bring footage and ask for a different look, rather than building a shot from nothing.