CLIP

What CLIP is

CLIP is an image-text model released by OpenAI on 5 January 2021. The project's repository describes it plainly as a neural network trained on a variety of (image, text) pairs^[2]. The paper behind it, with Alec Radford as first author and 11 co-authors, was posted to arXiv on 26 February 2021^[1]. The full name, Contrastive Language-Image Pre-training, describes the training method rather than a task: instead of learning to output one fixed set of labels, CLIP learns to place images and sentences in a shared space where matching pairs sit close together.

How it works

CLIP is built from two encoders trained together^[1]. The image encoder turns a picture into a vector, and comes in ResNet and Vision Transformer variants. The text encoder, a Transformer using masked self-attention, turns a caption into a vector in the same space. Training uses a contrastive loss whose job is to match each image with its own caption: a true pair should score higher than a pair formed by setting the wrong caption next to an image.

What makes this possible is the scale of the data. CLIP was trained on about 400 million image-caption pairs collected from the internet, a collection called WebImageText. The dataset itself was not released^[1].

Zero-shot classification

The consequence is that CLIP can classify images without any training on the task at hand. You write the candidate categories out as text prompts and compare the image with them; the prompt that scores highest is the answer. On ImageNet this procedure matched the original ResNet-50 while using none of the 1.28 million labelled training examples^[1]. The paper benchmarks the model on more than 30 datasets, covering tasks such as optical character recognition, action recognition and geo-localisation^[1].

Using and releasing the model

The published code exposes the two halves of the model separately: an example loads a model with `clip.load`, then gets image and text features from `encode_image` and `encode_text`^[2]. The repository requires PyTorch 1.7.1 or later together with torchvision, and the code is released under the MIT license^[2].

Weights arrived in stages. ViT-B/32 and RN50 came first, followed by RN101, RN50x4, RN50x16, ViT-B/16, RN50x64 and ViT-L/14, with ViT-L/14@336px arriving last in April 2022^[2].

Limits and intended use

The model card restricts intended use to research: deployed uses are out of scope, and surveillance and facial recognition are always out of scope^[3]. It also lists known weaknesses, among them fine-grained classification and counting objects^[3].

Where it sits

CLIP's influence came largely through parts of it that other systems borrowed. A frozen CLIP ViT-L/14 text encoder conditions Stable Diffusion v1, and DALL-E 2 generates images from CLIP image embeddings^[4]. LAION's open-source OpenCLIP reproduced and scaled the approach on public datasets^[4].