First ImageNet Large Scale Visual Recognition Challenge announced

Background

Comparing recognition algorithms requires a shared task and a shared set of images to run them on. Before 2010, one widely used benchmark in the field was PASCAL VOC, which concentrated on classification and detection across 20 object categories ^[1]. ImageNet was being assembled as a far larger image collection. The question the organizers posed was how well recognition algorithms would perform once the number of categories, and the scale of the data, grew sharply.

The announcement

On 13 April 2010, Fei-Fei Li announced the ImageNet Large Scale Visual Recognition Challenge 2010 in a message to the ImageNet community ^[1]. The stated goal was to better understand how algorithms perform on hierarchical recognition at large scale, with an eye toward image search and auto-annotation ^[1]. Both of those applications depend on recognition working across many categories rather than a handful: searching a photo collection by what is in the pictures, and attaching labels to images without a person doing it by hand.

In its first year the challenge focused on one task only: image categorization for images of 1,000 object categories ^[1]. Set against the 20 categories of PASCAL VOC, that was a large jump in difficulty, and the announcement presented it as complementing the older benchmark rather than replacing it ^[1].

How the first edition was set up

The announcement spelled out where the images would come from. Training and validation data came from ImageNet together with a newly collected dataset, while test data came only from the new set ^[1]. Organizers also released a baseline algorithm with precomputed descriptors, so that entrants had a reference point to measure their own methods against ^[1]. Results were to be presented at the PASCAL VOC workshop held at ECCV 2010 ^[1].

The call did not stay inside the ImageNet community. The same announcement was circulated on the Connectionists mailing list at Carnegie Mellon University in May 2010 ^[2].

Eight editions

According to the China Computer Federation, ILSVRC was first held in 2010, ended in 2017, and ran for eight editions ^[3]. The single-task, 1,000-category setup announced in April 2010 was therefore the starting point of a series rather than a one-off contest.

Why a shared challenge matters

A benchmark is a piece of shared infrastructure. Fixing the task, the data splits and a published baseline gives every group the same yardstick, so that a number reported by one laboratory can be compared with a number reported by another, and so that a new method can be checked against something simple. The first edition's design reflects that idea: training and validation images came from a mix of sources while test images came only from the newly collected set, and a baseline with precomputed descriptors was published alongside the data ^[1]. Entrants could therefore see whether their own approach was adding anything over a straightforward reference.

The choice of task carried a similar logic. Categorization over 1,000 categories is not a scaled-up version of a 20-category problem in any simple sense; more classes means more room to confuse one object with another, and the announcement framed the increase explicitly against PASCAL VOC's 20 categories ^[1]. The motivation given, image search and auto-annotation, kept the practical use in view throughout ^[1].

What the announcement set in motion was a recurring public comparison: a yearly event, announced openly and circulated beyond its own community, in which groups could submit results on a common task with a common test set. Whether a given year's methods were good or bad was left to the numbers; the challenge's job was to make those numbers comparable.