Clarifai wins ILSVRC 2013 classification

Background

ILSVRC is an evaluation campaign built around the ImageNet image dataset. In the classification task a system is shown an image and must assign it to a category. The score is the top-5 error: a prediction counts as correct if the right category is among the system's five highest-scoring guesses, so a lower number is better. The second task, object localisation, asks a system to say where in the image an object sits, not only what it is. Because entries are scored on the same data with the same metric, their numbers can be put side by side in one table.

What happened

In 2013 the classification task was won by Clarifai, with an ensemble of multiple convolutional neural networks (CNNs) ^[2]. The results table in the GoogLeNet paper lists Clarifai as the first-place entry for that year, at 11.7% top-5 error without external data and 11.2% when ImageNet 22k data was used ^[1]. The object localisation task was won by OverFeat ^[2]. The two top places therefore went to different teams, and the classification winner was not a single network but a combination of several.

Key details

Away from the leaderboard, a second piece of work from the same months is closely tied to the benchmark. Zeiler and Fergus submitted "Visualizing and Understanding Convolutional Networks" to arXiv on 12 November 2013; the paper reported architectures that outperformed Krizhevsky et al. on the ImageNet classification benchmark ^[3]. Their numbers were picked up in the comparison table of the GoogLeNet paper, which lists Zeiler & Fergus at 14.8% top-5 test error with 6 nets and 16.1% with a single net ^[1]. arXiv records a third revision of the Zeiler and Fergus paper on 28 November 2013, in the same month as the original submission ^[3].

Read together, the two records describe a field moving on two fronts at once: pushing the classification error down through larger ensembles, and trying to explain what the trained networks had learned inside.

Why it matters

The 2013 entries became the reference points that later systems measured themselves against. When the GoogLeNet authors described their own model, they placed Clarifai, Zeiler & Fergus and other entries in the same comparison table, quoting each one's top-5 error ^[1]. A result only becomes useful as a baseline when its exact number and its exact data conditions are written down, and that is what these tables did: Clarifai's figure was recorded twice, once without external data and once with ImageNet 22k data, so the two settings could not be confused ^[1].

The year also left a methodological mark. The winning classification entry was an ensemble rather than a single model ^[2], and the visualisation paper that circulated in the same weeks framed its contribution as understanding the networks as well as beating a previous benchmark ^[3].