Denoising Diffusion Probabilistic Models

What the paper set out to do

Denoising Diffusion Probabilistic Models was submitted to arXiv on 19 June 2020 by Jonathan Ho, Ajay Jain and Pieter Abbeel of UC Berkeley, and was published at NeurIPS 2020^[1]. Its subject is image generation: whether a family of models called diffusion probabilistic models could produce pictures that stand comparison with the strongest image generators of the day.

The framework itself is older. Diffusion probabilistic models were proposed by Sohl-Dickstein and colleagues in 2015, and this paper builds on that work while connecting it to denoising score matching^[1]. What the paper added was a training procedure that worked at the scale of standard image benchmarks^[1]. That is the step that made diffusion a practical rival to GANs^[2][3].

How it works

The central idea is to destroy an image slowly and learn to undo the destruction. In the forward direction, Gaussian noise is added to an image over many small steps until essentially only noise remains. In the reverse direction a neural network looks at a noisy image and predicts the noise that was added, so that the noise can be subtracted; repeat this many times and a picture is produced from pure noise^[1].

The choice of prediction target matters. The network is not trained to output the clean image directly but to output the noise itself, and the loss measures how close the predicted noise is to the noise that was actually added^[1]. Generation runs the same machinery in reverse: sampling starts from pure noise and applies the network repeatedly, removing a little noise at each step^[1].

The network used for this is a U-Net, a convolutional architecture that shrinks the spatial resolution of its input and then builds it back up, passing information between the two halves. In the experiments the number of diffusion steps was set to T = 1,000, chosen so that the number of neural network evaluations needed to draw a sample matched that of earlier work, which keeps comparisons of image quality like for like^[1].

Results

On CIFAR-10 the model reached an Inception score of 9.46 and an FID of 3.17, the state of the art at the time for unconditional generation, meaning generation without class labels^[1]. Both are standard ways of scoring generated images: the Inception score rewards images that a classifier recognises as a clear instance of one class and rewards variety across classes, while FID compares the distribution of generated images with that of real ones. The paper also reported high-quality 256×256 images on LSUN^[1].

Why it matters

Diffusion models had been described in the literature for several years; the significance of this paper is that it made them work as image generators, at a quality that put them next to GANs rather than behind them^[2][3]. The working parts it set out — a U-Net predicting the added noise, sampling by running the process backwards over a thousand steps — are the ones later image generators inherited.

What followed

A follow-up paper in May 2021 reported diffusion models with image quality superior to the state of the art, including GANs^[2]. Diffusion-based image generation is now the approach used by DALL-E 2, Imagen and Stable Diffusion^[3]. Seen from that later vantage point, the paper marks the point at which diffusion moved from a proposal in the literature to a working method for image synthesis.