Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (Imagen)

What the paper set out to do

A text-to-image model has to do two jobs at once. It must turn a written description into a representation the image generator can be conditioned on, and it must produce pixels at photographic quality. The Imagen paper, submitted to arXiv on May 23, 2022 by Chitwan Saharia, William Chan, Saurabh Saxena and 11 other Google authors including Jonathan Ho and Mohammad Norouzi, describes a text-to-image diffusion model that the authors present as reaching a high degree of photorealism and a deep level of language understanding ^[1].

How it works

Imagen pairs two components. The text prompt is encoded by a large transformer language model pretrained on text-only corpora — T5 is the example the paper names — and a diffusion model then generates the image conditioned on that encoding ^[1]. Diffusion here means the usual procedure: the model learns to reverse a gradual noising process, so that generation starts from random noise and removes noise step by step until an image remains.

The paper's central finding is about where the gains come from. Increasing the size of the language model improved sample fidelity and image-text alignment more than increasing the size of the image diffusion model did ^[1]. A generic language model that had never been trained on image-text pairs was therefore doing substantive work for image synthesis.

Results

The headline number is a zero-shot FID-30K of 7.27 on COCO, achieved without any training on COCO ^[1]. FID compares the distribution of generated images with that of real ones, and a lower value is better. For comparison, the paper reports 12.4 for GLIDE and 10.4 for DALL-E 2 ^[1]. On image-text alignment, human raters judged Imagen's samples to be on par with the COCO data itself ^[1].

DrawBench and side-by-side comparisons

Alongside the model, the authors introduced DrawBench, described as a comprehensive and challenging benchmark for text-to-image models ^[1]. Using human evaluation, they put Imagen up against VQ-GAN+CLIP, Latent Diffusion Models and DALL-E 2, and raters preferred Imagen in side-by-side comparisons ^[1].

Why it matters

Two contributions stand out. The first is the scaling result: it locates a source of improvement in the text encoder rather than in the image generator, which indicates that the two halves of such a system can be studied and scaled somewhat separately, and that pretraining on text alone transfers to image synthesis ^[1]. The second is the benchmark. DrawBench gives a fixed and deliberately hard set of evaluation cases for text-to-image models, and the paper reports its comparisons against named contemporary models rather than presenting numbers on their own ^[1]. Together they made the text side of text-to-image generation an explicit object of study, and gave later comparisons a common yardstick.