Patrick Esser

Overview

Patrick Esser is a researcher working on generative models for images and video. His published work follows a single line: the latent diffusion paper that became the basis of Stable Diffusion, a video synthesis paper behind Runway's Gen-1, and the Stable Diffusion 3 paper on rectified flow transformers. In 2024 he co-founded the image-model company Black Forest Labs.^[1][4]

The latent diffusion paper

"High-Resolution Image Synthesis with Latent Diffusion Models" was submitted to arXiv on 20 December 2021 and accepted at CVPR 2022.^[5] Esser is listed as its fourth author, after Robin Rombach, Andreas Blattmann and Dominik Lorenz.^[5] The paper is the basis of Stable Diffusion.^[1]

The method it introduced is usually called latent diffusion. A diffusion model generates an image by starting from random noise and denoising it step by step; doing that directly on high-resolution pixels is expensive, because every denoising step has to touch every pixel. Latent diffusion instead compresses the image into a much smaller representation — a latent — using an autoencoder, runs the denoising process inside that compressed space, and decodes the result back into pixels. Because the latent is far smaller than the image, each denoising step becomes much cheaper to run.

Video synthesis

On 6 February 2023 a paper titled "Structure and Content-Guided Video Synthesis with Diffusion Models" appeared on arXiv with Esser as lead author.^[2] It is the paper behind Runway's Gen-1, and Runway co-founder Anastasis Germanidis is among the co-authors.^[2] The title names the central design choice: generation is steered by two separate kinds of guidance, one describing the structure of a video and one describing its content, so that the model follows an intended arrangement rather than inventing everything from the text prompt alone.

Stable Diffusion 3

On 5 March 2024 Esser was lead author of "Scaling Rectified Flow Transformers for High-Resolution Image Synthesis", the paper behind Stable Diffusion 3.^[3] Rectified flow is a way of formulating the generative process: instead of the curved, noisy trajectory of a classic diffusion model, the model learns to carry samples along paths between noise and data, and "rectifying" the flow means making those paths straighter, which in principle makes it easier to sample in fewer steps. The paper pairs this with a transformer backbone — hence "rectified flow transformers" — and examines how the approach scales for high-resolution image synthesis.^[3]

Black Forest Labs

In 2024 Esser co-founded Black Forest Labs with Robin Rombach and Andreas Blattmann.^[4] On 1 August 2024 the company announced its launch together with a $31 million seed round led by Andreessen Horowitz.^[6]

Place in the diffusion line

The papers and the company form a fairly continuous line. The same small group of researchers appears on the latent diffusion paper — Rombach, Blattmann and Esser — and two of them went on to co-found Black Forest Labs with Esser.^[5][4] Esser's position within that group also shifted over time: fourth author on the latent diffusion paper, then lead author on the video synthesis paper and on the Stable Diffusion 3 paper.^[5][2][3] Read together, the three papers trace the movement of one research programme from image synthesis in a compressed latent space, to video, to rectified flow transformers.