Diffusion models

What it is

Diffusion models are generative models, meaning they learn to produce new data such as images. They work by learning to reverse a gradual noising process. During training, noise is added to images step by step, and a neural network learns to predict and remove that noise.^[1]

How it works

Think of a photograph slowly covered with static until it becomes pure random noise. Training teaches a network to undo one small step of that corruption at a time. Generation runs the process in the other direction. The model starts from pure random noise and denoises it over many steps into a new image.^[1]

The gradual approach is the key. Adding noise destroys the image's details in small, controlled steps, so each step is a manageable change for the network to learn to reverse. A single jump from a clear image to pure noise would be far harder to invert.

Where it came from

In June 2020, the DDPM paper by Ho, Jain and Abbeel showed that diffusion could produce image quality competitive with GANs, a rival family of generative models. On CIFAR-10, a common image dataset, the paper reported an FID score of 3.17. Lower FID scores mean that generated images look closer to real ones.^[1]

In December 2021, Rombach and colleagues proposed latent diffusion, which runs the process in a compressed latent space instead of on raw pixels. This cut the computation needed. The approach became the basis of Stable Diffusion, which was publicly released in August 2022.^[2]

Where it is used today

Text-to-image systems condition the denoising on a text embedding, a numerical representation of the prompt, so that the picture follows the description. The same general approach has been extended to video and audio.^[3] In robotics it is used to generate sequences of actions.^[4] What these uses share is the core loop: start from noise, condition on whatever should guide the result, and remove noise step by step.

Where it stands

Diffusion is one family of generative models among several. GANs, mentioned above, are another, and the two are often compared on image quality. Diffusion's distinctive feature is that it builds an output gradually, and each step can be guided by other inputs, such as a text prompt.

Limits

Generation takes many denoising steps, which costs time and computing power. Latent diffusion addresses part of this by working in a smaller space, which cuts compute.^[2] The process also starts from random noise, so the same prompt does not always produce the same image. That variety is useful for creative work, but it also means results cannot be guaranteed to match a description exactly. The number of steps is also a trade-off: fewer steps are faster but may lose detail, while more steps take longer to run.