Zero-Shot Text-to-Image Generation (DALL·E)

The problem the paper takes on

Text-to-image generation means building a system that produces a picture from a written description. The paper starts from a complaint about how the field had been approaching that job: the traditional focus, its authors write, was on finding better modeling assumptions for training on a fixed dataset^[1]. Work in that style tended to build elaborate architectures, add auxiliary losses to the training objective, or feed the model side information such as object part labels or segmentation masks during training^[1]. Each of those choices encodes human knowledge about what images contain and how they should be described; each also ties the model to a particular dataset and annotation budget.

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh and five other OpenAI authors, including Ilya Sutskever, submitted the paper to arXiv on February 24, 2021^[1].

The approach: one stream, one objective

The method is deliberately plain. A transformer models the text and the image tokens as a single stream of data and does so autoregressively^[1]. Two ideas are packed into that sentence.

A token is a small discrete unit — roughly, a chunk of text or a chunk of an image — that the model treats as one symbol in a vocabulary. Putting text and image tokens into one stream means the model sees a caption and its picture as a single sequence, with no special machinery separating the language part from the visual part.

A transformer is a neural network built around attention, a mechanism that lets every position in a sequence weigh information from other positions before making its prediction. Autoregressive means the model predicts the next token from all the tokens that came before it, then feeds that prediction back in and predicts again. This is the same operation used in language modeling, applied to a vocabulary that happens to contain both words and image tokens. Because the whole thing is one sequence and one prediction target, the training signal is a single objective rather than a stack of separately tuned losses.

What it was trained on

The model was a 12-billion-parameter autoregressive transformer trained on 250 million image-text pairs collected from the internet^[1]. Parameters are the learned numbers inside the network; twelve billion of them puts the model in the range where a simple objective and a large dataset can substitute for architectural cleverness.

The evaluation was zero-shot, meaning the model was tested without being specially trained or fine-tuned for the individual tasks it was measured on. A zero-shot result is a test of generality: it asks whether one model, trained once, can handle prompts and settings it was never tuned for.

The result

The paper's claim is measured rather than triumphant. With sufficient data and scale, the authors report, the approach is competitive with previous domain-specific models when evaluated in a zero-shot fashion^[1]. Competitive, not superior: models built specifically for a domain, with their hand-designed components and extra supervision, remained a fair comparison, and the simple model matched them without any of that.

Why it matters

What is notable here is the direction of the trade. Instead of adding structure to compensate for limited data, the paper removes structure and adds data. If that works, several things follow. Side information such as object part labels and segmentation masks is no longer needed, which matters because producing those annotations is expensive and ties a model to the categories someone thought to label^[1]. Complex architectures and auxiliary losses are no longer needed either, which makes the recipe easier to describe and to scale^[1].

The result also reframes zero-shot evaluation as the interesting question. A model that is competitive on tasks it was not tuned for is one whose competence is not confined to a single benchmark. The paper does not claim the problem is solved; it shows that a single autoregressive transformer over a combined stream of text and image tokens, given enough data and parameters, is a workable answer where the field had expected a complicated one^[1].