LAION-5B: An open large-scale dataset for training next generation image-text models

The problem

Large vision-language models learn by matching an image with the text that describes it, and they need training data on the order of billions of image-text pairs to do this well. The teams behind models such as ALIGN, BASIC, GLIDE, Flamingo and Imagen could study models at that scale because they had data at that scale. Those datasets were not openly available, which left researchers outside those teams without comparable data for studying large-scale language-vision models. LAION-5B was assembled to fill that gap: an openly available collection of billions of image-text pairs.^[1]

What the dataset contains

LAION-5B consists of 5.85 billion image-text pairs, all of them filtered with CLIP.^[1] CLIP is a model that scores how well an image and a caption fit together; using it as a filter means keeping the pairs whose image and text appear to match and discarding the rest. Of the 5.85 billion pairs, 2.32 billion contain English text.^[1]

The collection is divided into three parts: 2.32 billion English image-text pairs, 2.26 billion multilingual pairs, and 1.27 billion pairs that are not specific to any particular language.^[2] Filtering was done with OpenAI's ViT-B/32 CLIP model for the English portion and a multilingual CLIP ViT-B/32 model for the other languages.^[2]

What is released alongside the data

The release is more than a pile of pairs. It includes several nearest neighbour indices, which let a user find pairs similar to a given image or text without scanning the whole collection; an improved web interface for exploring the dataset and generating subsets; and detection scores for watermark, NSFW and toxic content, so that users can screen what they take.^[1]

Results

The authors report that foundational models such as CLIP, GLIDE and Stable Diffusion could be replicated and fine-tuned on the dataset.^[1] They also state that their OpenCLIP ViT-L/14 models were the first open-source reproductions of OpenAI's largest CLIP models.^[2]

Why it matters

A model's capability depends heavily on the data it is trained on, and for years the largest image-text collections sat inside a small number of organisations. By publishing a dataset at the scale of billions of pairs, together with the tools to search it and cut subsets from it, the authors made it possible for outside groups to train, fine-tune and compare vision-language models on the same footing.^[1] The three-way split matters for the same reason: a researcher can work with English data, multilingual data, or a mixture that is not tied to a single language, depending on what the experiment needs.^[2] The OpenCLIP results, meanwhile, show that the openly released data and training recipes were enough to reproduce one of the strongest closed CLIP families.^[2]

The paper is by Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti and ten other authors, and was submitted to arXiv on 16 October 2022.^[1]