ImageNet: A Large-Scale Hierarchical Image Database

The problem

By 2009 the Internet held a vast number of images, and the authors argued that this could support more sophisticated recognition algorithms. Such algorithms, however, needed larger and cleanly labelled datasets to be trained and benchmarked. The paper set out to build one.^[1]

How the database was built

ImageNet organises images according to the WordNet hierarchy. In WordNet each concept is a synonym set, or synset. WordNet has roughly 80,000 noun synsets, and the goal was to populate most of them with an average of 500 to 1,000 clean, full-resolution images each. That would mean tens of millions of annotated images.^[1]

A hierarchy matters for a labelled dataset because a concept such as dog can sit under broader categories, so each image can be placed at a precise concept and still belong to the more general ones. A small collection can make it hard to tell whether a recognition method works in general, which is why the paper stressed scale and clean labels.

Candidate images were collected from the Internet and then verified by people through Amazon Mechanical Turk. The authors checked the labels on 80 randomly sampled synsets at different depths of the tree, and the labelling precision averaged 99.7%.^[1]

At the time of the paper, ImageNet had 12 subtrees, including mammal, bird, fish, vehicle, furniture, tool, flower and fruit. Together they contained 5,247 synsets and 3.2 million images.^[1]

Results and demonstrations

The paper demonstrated three simple uses of the data: object recognition, image classification and automatic object clustering.^[1] Those tasks are basic building blocks of computer vision: naming the object in a picture, assigning a whole image to a category, and grouping similar objects without labels. Its main contribution was the dataset itself, which made it possible to train and compare recognition methods at a scale that earlier collections did not offer.

Why it mattered

ImageNet became the basis of the ImageNet Large Scale Visual Recognition Challenge (ILSVRC), which has been run annually since 2010. The challenge used 1,000 object classes and roughly 1.2 million training images. The ILSVRC 2010 dataset contained 1,461,406 images, compared with 19,737 images in the PASCAL VOC 2010 dataset.^[3]

A shared benchmark gives researchers a common test. Every method is trained on the same data and scored the same way, so results from different groups can be compared directly.

Later influence

The organisers' 2015 overview of the challenge describes the benchmark and the advances in object recognition it made possible. The competition attracted participants from more than fifty institutions, and the overview compares state-of-the-art computer-vision accuracy with human accuracy.^[3] The paper, by Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li and Li Fei-Fei of Princeton University, appeared at the 2009 IEEE Conference on Computer Vision and Pattern Recognition, held in Miami from 20 to 25 June 2009.^[1]