DALL·E

What DALL·E is

DALL·E is a text-to-image model revealed by OpenAI on 5 January 2021. It is a neural network that creates images from text captions for a wide range of concepts. Its name is a mash-up of Pixar's WALL-E and the painter Salvador Dalí.^[2][3] OpenAI introduced it on the same day as CLIP, a separate system for connecting text and images.^[4]

Where it came from

The system was documented in the paper “Zero-Shot Text-to-Image Generation”, by Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen and Ilya Sutskever, which appeared on arXiv on 24 February 2021.^[1] Under the hood it used a version of GPT-3 modified to generate images: a 12-billion-parameter autoregressive transformer trained on 250 million text-image pairs collected from the internet.^[1][3]

“Autoregressive” here means the model produces its output one unit at a time, with each unit conditioned on everything produced before it — the way a language model writes a sentence word by word.

How it works

The method has two stages. First, a discrete VAE compresses each 256×256 image into 32×32 = 1,024 image tokens. A VAE is an encoder–decoder pair that learns a compact representation of its input; making that representation discrete lets an image be treated as a sequence of symbols, much like words. Then up to 256 text tokens and the image tokens are modelled as a single stream, and the model generates the image token by token.^[1] Because text and image share one stream, DALL·E received text and image as a single stream of data containing up to 1,280 tokens and was trained with maximum likelihood to generate the tokens one after another.^[5]

Scale brought engineering problems. The 12-billion-parameter model needed about 24 GB of memory in 16-bit precision, more than a 16 GB NVIDIA V100 GPU could hold, so the team used parameter sharding — spreading the model's parameters across several devices instead of keeping them all in one place.^[1]

What it could do

OpenAI said DALL·E could create anthropomorphized versions of animals and objects, combine unrelated concepts in plausible ways, render text and apply transformations to existing images.^[5] The paper's evaluation was deliberately zero-shot: the model was tested on MS-COCO captions without having been trained on that dataset, and human evaluators preferred its samples over prior work trained on the dataset 90% of the time.^[1]

Why it mattered

The design puts text and image tokens into one autoregressive sequence, so a single model trained to predict the next token handles both modalities rather than a caption encoder handing off to a separate image generator. The zero-shot result is the part that stands out: a model that never saw MS-COCO could still produce images that people preferred to the output of systems built on that dataset.^[1]

Where it sits

DALL·E was succeeded by DALL·E 2 in April 2022 and DALL·E 3 in September 2023.^[3]