Learning Transferable Visual Models From Natural Language Supervision (CLIP)

CLIP is a model from OpenAI that learns to connect images and text. Instead of being trained to sort pictures into a fixed set of categories, it learns from ordinary sentences found alongside images on the internet, and can then be pointed at a new classification task just by writing the categories out in words.

What it set out to do

For a long time, image recognition meant training on a labelled dataset: a fixed list of categories, with every image tagged by a person. Such models work well on the categories they were trained on, but they are tied to that list, and moving to a new task usually means collecting and labelling more images. CLIP's authors started from a different source of supervision. Images on the internet arrive with text around them — captions, descriptions, alt text — and that text is far cheaper than hand-labelling. The question was whether learning to match images with natural language would produce a visual model that transfers to tasks it was never explicitly trained on.

How it works

CLIP trains an image encoder and a text encoder together on 400 million image-text pairs collected from the internet ^[2]. The training signal is contrastive: for a batch of images and their captions, the model is rewarded for pulling each matching image-caption pair together in a shared embedding space and pushing mismatched pairs apart ^[2]. After training, an image and a sentence can be compared directly, because both are mapped into the same space.

That shared space is what makes zero-shot classification possible. To sort an image into one of a set of categories, you write a text prompt for each category — for example "a photo of a dog" — encode the prompts as text, encode the image, and pick the category whose text embedding is closest to the image embedding ^[2]. Nothing is retrained, and the categories only need to be describable in words.

Results

The headline result was on ImageNet. Working zero-shot — with no labelled examples from the benchmark — CLIP reached 76.2% top-1 accuracy, against 11.5% for the earlier Visual N-Grams model ^[2]. More striking still, its zero-shot accuracy matched the original ResNet-50 without using any of the 1.28 million labelled training images that ResNet-50 was trained on ^[2]. A model that had never been trained to classify was matching a supervised classifier on the benchmark that had long defined the field.

Why it matters

CLIP's real contribution was the joint image-text space itself. It later showed up most often as a component inside text-to-image systems rather than as a classifier. Stable Diffusion uses a frozen CLIP ViT-L/14 text encoder to turn a written prompt into a conditioning signal ^[3], and OpenAI's DALL-E 2, released in April 2022, generates images starting from CLIP image embeddings ^[4]. A model built to sort pictures ended up best known for helping make them.

The design also changed how a task is specified. Because categories live in the text encoder, a new task can be handed to the model as a sentence rather than as a newly trained output layer, which is what let the same frozen encoder be reused in later systems ^[3][4].

Publication and after

OpenAI announced CLIP on 5 January 2021, the same day as DALL-E. The paper went up on arXiv on 26 February 2021 with Alec Radford as first author, and was published at ICML 2021 ^[1]. Its reported numbers — 76.2% zero-shot on ImageNet against 11.5% for Visual N-Grams, and parity with a supervised ResNet-50 without any labelled training images — became a common reference point for what learning from natural language could do for vision ^[2].

Limits

The method is only as good as the text that describes an image. A category that is hard to put into words, or a task that needs fine distinctions no caption would mention, is not automatically within reach; what CLIP provides is a way to specify a task in language, not a guarantee that language is enough.