Imagen

What it is

Imagen is a text-to-image diffusion model built by the Brain team at Google Research. The team described it in a paper titled "Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding", posted to arXiv on 23 May 2022^[2]. The system takes a sentence of ordinary text and returns a picture meant to match it. The two things the title points at — photographic realism and a deep understanding of language — are what the work tries to deliver at the same time.

How it works

Imagen splits the job between two kinds of network.

The text side is handled by a large frozen T5-XXL encoder, which turns the input prompt into embeddings — numerical vectors that stand in for the meaning of the words^[1]. "Frozen" means these weights are not updated during image training: the language model stays as it was pretrained, and all the learning happens in the image models.

The image side is a cascade. A conditional diffusion model first generates a small 64×64 image, and then text-conditional super-resolution models upscale that image to 256×256 and finally to 1024×1024^[1]. The "condition" here is the text embedding produced on the other side — every denoising step refers back to it. Diffusion models work by learning to reverse a noising process: training adds noise to images step by step, and the network learns to take a noisy sample and remove a little of the noise, conditioned on the text, until a clean picture remains. Generating small first and then refining keeps the computation manageable while still ending with a large image.

The language model is responsible for understanding the prompt, and the image models for turning that understanding into pixels; the clearer the condition the text side supplies, the more the image side has to work with. That division of labour is the reason the paper goes on to examine the size of the language model.

What the experiments showed

The paper's most quoted finding concerns scale. The researchers found that increasing the size of the language model improved sample fidelity and image-text alignment more than increasing the size of the image diffusion model^[2]. The paper describes the language component as a generic large language model pretrained on text-only corpora, such as T5^[2] — it was not built for images at all.

Two quantitative results stand out. Imagen set a new state-of-the-art FID score of 7.27 on COCO, without ever training on COCO^[2]. FID compares the distribution of generated images with that of real ones; a lower score means the two are harder to tell apart statistically.

The team also introduced DrawBench, a benchmark for evaluating text-to-image models, which Google Research describes as a comprehensive and challenging benchmark for such systems^[1]. On DrawBench, human raters preferred Imagen over VQ-GAN+CLIP, Latent Diffusion Models and DALL-E 2 for both sample quality and image-text alignment^[2]. Because the comparison was made by people rather than by a single automatic score, it could capture things an automatic metric tends to miss — for instance whether an image really shows the objects named in the prompt and the relations between them.

Release and aftermath

Google chose not to release code or a public demo for Imagen, citing misuse risks and social biases in web-scraped training data^[1]. Outside knowledge of the model therefore comes mainly from the published paper.

Within Google's own line of models, Imagen was succeeded by Imagen 2 in December 2023^[3].