ImageNet Spring 2010 release

Background

ImageNet is an image database built on WordNet, a lexical resource that groups English words into sets of related terms and arranges those sets in a hierarchy running from broad to narrow. The China Computer Federation credits the dataset to Fei-Fei Li, Kai Li and others, who created and released it in 2009 at Princeton; in that account it holds 21,841 categories and more than 14 million images^[1]. Because the categories come from a language resource rather than from whatever happened to be photographed, the collection is organised as a tree of noun senses, and each image sits somewhere in that tree.

The spring 2010 release

On 11 May 2010 Fei-Fei Li announced the 2010 Spring Release of ImageNet. The announcement put the total at more than 11 million images spread over 15,500 visual categories, and described the range of those categories as running from activities and scenes to objects^[2]. Those figures differ from the 21,841 categories and more than 14 million images that the China Computer Federation attributes to the dataset as a whole^[1]; the announcement describes the contents of one release, the other describes the finished collection.

Annotations, features and tools

The release was not only a pile of pictures. The announcement says bounding box annotations are provided for nearly 1,000 popular visual categories, with 200 images per category^[2]. A bounding box is the rectangle drawn around the object of interest in a picture, so that a training program is told not just that a dog is somewhere in the image but which part of the image contains it. That kind of localisation is what an object detector has to produce, and hand-drawn boxes are expensive to obtain.

Precomputed image features were shipped as well. The announcement lists densely sampled SIFT features for 1.2 million images and 1,000 categories, including raw descriptors, quantized codewords and spatial coordinates^[2]. SIFT describes a small patch of an image by the pattern of local gradient directions inside it, which makes the description fairly stable under changes of scale and small shifts. Dense sampling means the descriptor is computed at regular positions across the image rather than only at a handful of detected points, so every region gets covered. The three files serve different purposes: the raw descriptors are the vectors themselves, the quantized codewords map each vector onto a finite vocabulary so that an image becomes a countable histogram, and the spatial coordinates record where each patch was taken from, so that arrangement as well as content can be used.

A MATLAB Toolbox accompanied the release. According to the announcement, it helps users work with the WordNet hierarchical structure behind the visual categories and download full resolution images of a subtree^[2]. In practice this let a researcher pick one branch of the hierarchy — say a particular kind of animal or vehicle — and pull exactly those images at original resolution, instead of taking the whole archive.

The challenge

Announcing the release, the team also called for participation in ILSVRC2010, a recognition challenge run in partnership with PASCAL VOC 2010, in which systems were to recognise 1,000 visual categories^[2]. A challenge of this shape fixes the category list and the data, and everyone's method is scored on the same set, so numbers from different groups become comparable instead of each paper inventing its own split. Pairing the new dataset with an established evaluation campaign was a way of putting the images in front of researchers who already had systems to test.

Why it matters

Read together, the parts of the announcement describe a package rather than a database: images at scale, human-drawn boxes on a popular subset, features precomputed for groups without the machines to compute them, tooling for navigating a category hierarchy, and a public challenge with a fixed set of categories^[2]. Each piece lowers the cost of entry for someone who wants to train or measure a recogniser but cannot crawl and label a million photographs themselves. The shift marked by this release is that ImageNet stopped being only something its authors built and became something others were invited to run their methods against — the role a benchmark plays in a research field.