OpenAI introduces DALL·E

Background

A caption is a short written description of an image. Image models were typically trained to recognise a fixed list of labelled categories. The 2021 work aimed at more flexible links between language and images, so that a model could respond to descriptions in ordinary words.

What happened

On 5 January 2021 OpenAI announced DALL·E, a neural network that creates images from text captions for a wide range of concepts expressible in natural language.^[1] On the same day it introduced CLIP, a network that learns visual concepts from natural-language supervision. CLIP can classify images zero-shot, using only the names of the categories.^[2]

The DALL·E paper, *Zero-Shot Text-to-Image Generation*, was posted to arXiv on 24 February 2021 by Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen and Ilya Sutskever.^[3]

Text-to-image generation means producing a picture from a written description. A model that connects words and images needs a shared way to represent both, so that a caption and a picture can be related to one another.

Key details

The method treats text and images as one stream. A transformer models the text tokens and image tokens autoregressively, predicting them one after another as a single sequence, much as a language model predicts the next word. The paper relied on data and scale rather than complex architectures or extra labels.^[3]

The model is a 12-billion-parameter autoregressive transformer, trained on 250 million text-image pairs collected from the internet.^[3]

At 16-bit precision the 12-billion-parameter model takes about 24 GB of memory, more than a 16 GB NVIDIA V100 GPU provides, so the team used parameter sharding, which splits a model's parameters across multiple devices.^[3] The paper reported that, evaluated zero-shot, the model was competitive with previous domain-specific models.^[3]

The successor, DALL·E 2, entered beta on 20 July 2022. OpenAI invited 1 million people from its waitlist, offered free monthly credits, and sold additional credits in increments of 115 generations for $15.^[4]

Representing images as discrete tokens lets the same kind of model used for text be applied to pictures, with the caption serving as the starting context for the image tokens.^[3]

Impact

The 2021 work showed that a single autoregressive model could generate images from text, treating the caption and the image in much the same way. Approaches that connect text and images in a shared way became a major direction in later image generation. The move to paid credits in the successor shows image generation shifting from research demonstrations toward products that people pay to use.

The claim of zero-shot performance means the model was judged on tasks it had not been specifically trained for. That is a useful test, but like other benchmark results it describes the tests chosen rather than every possible use.