High-Resolution Image Synthesis with Latent Diffusion Models

The problem

Diffusion models decompose the image formation process into a sequential application of denoising autoencoders, and with that recipe they had already achieved state-of-the-art synthesis results on image data and beyond^[1]. The same formulation allows a guiding mechanism that controls the image generation process without retraining^[1]. The obstacle was cost. Because these models typically operate directly in pixel space, optimising a powerful diffusion model often consumes hundreds of GPU days, and inference is expensive due to sequential evaluations^[1]. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser and Bjoern Ommer, in a paper submitted to arXiv on 20 December 2021, set out to keep the quality and flexibility of diffusion models while making them trainable on limited computational resources^[1].

Moving diffusion into a latent space

Their move is to apply diffusion models in the latent space of powerful pretrained autoencoders instead of on pixels^[1]. An autoencoder is a pair of networks: an encoder that compresses an input into a compact code, and a decoder that rebuilds the input from that code. Once such a pair has been trained, the denoising network only ever handles the compact codes, so each of its many sequential steps works on far less data than a full-resolution image would demand; the decoder converts the final code back into a picture.

The authors describe training diffusion models on such a representation as allowing, for the first time, a near-optimal point between complexity reduction and detail preservation, greatly boosting visual fidelity^[1]. The design divides the labour: the autoencoder decides how faithfully an image can be represented, and the diffusion model decides what the image contains. Each half can be considered and traded off on its own, which is where much of the flexibility of the approach comes from.

Conditioning through cross-attention

Introducing cross-attention layers into the architecture turns diffusion models into flexible generators for general conditioning inputs such as text or bounding boxes, and makes high-resolution synthesis possible in a convolutional manner^[1]. Cross-attention is the part of a network that lets one stream of information query another; here the generating network can consult the conditioning signal at each denoising step and steer accordingly. Because the condition enters through these layers rather than through a bespoke architecture, one model can be driven by different kinds of input.

Results

The latent diffusion models achieve a new state of the art for image inpainting and competitive performance on unconditional image generation, semantic scene synthesis and super-resolution, while significantly reducing computational requirements compared with pixel-based diffusion models^[1]. Read together, the results say two things: on the tasks the authors measured, moving into latent space did not cost quality, and it did cut the computation needed for training and inference.

Code and why it matters

The code is released in the CompVis latent-diffusion repository, and the paper is the CVPR 2022 version^[1]. Publishing the implementation alongside the paper means others can run the method rather than rebuild it from the description.

The contribution is a way of splitting image synthesis into two stages that can be improved separately: a representation stage that fixes how much detail survives compression, and a generative stage that runs cheaply inside that representation. On top of that sits a conditioning mechanism general enough to accept text or bounding boxes without retraining^[1]. Together these changes lower the barrier to training and running diffusion models, which is the practical claim the paper makes: quality and flexibility need not be bought with hundreds of GPU days^[1].