ResNet wins ILSVRC 2015 with 3.57% error

Background

Image classification — deciding which category a picture belongs to — is one of the standard tasks used to test neural networks, and the ImageNet competition, ILSVRC, is where large-scale systems are compared against each other. The usual way to make such a system more capable is to stack more layers, because a deeper network can in principle represent more complicated functions. In practice depth is hard to use: the more layers there are, the harder the network is to optimize, and simply adding layers does not reliably produce a better classifier. The 2015 competition posed that question directly — whether the ceiling on useful depth could be raised.

What happened

The paper Deep Residual Learning for Image Recognition was first submitted to arXiv on 10 December 2015 ^[1]. Its abstract states that an ensemble of residual nets achieved 3.57% error on the ImageNet test set and won first place in the ILSVRC 2015 classification task ^[1]. Wikipedia's account of ImageNet gives the same number from the other direction: Microsoft's very deep convolutional network, with over 100 layers, won the 2015 contest with 3.57% error on the test set ^[2]. The same account adds that the winning entry was ResNet and that it exceeded human performance ^[2]. The winner, in other words, was not a slightly deeper version of familiar designs but a network built on a different principle.

Depth and the residual idea

The residual nets evaluated in the paper reach a depth of 152 layers, which the abstract describes as 8x deeper than VGG nets while having lower complexity ^[1]. Depth on that scale is the visible marker of the result; what makes such depth trainable is a change in what a layer is asked to learn.

Normally a stack of layers is trained to learn a mapping from its input to its output directly, which the paper calls learning unreferenced functions. It reformulates the layers instead as learning residual functions with reference to the layer inputs ^[1]. In plain terms, a block of layers no longer has to produce the wanted output from scratch; it has to produce the difference between the wanted output and what arrived, while the input itself is carried forward and added back. If the best thing for the block to do is to leave its input alone, the target for the residual is simply zero, which is an easy target to hit; forcing a stack of nonlinear transformations to reproduce its own input exactly is not. Learning the residual therefore makes "do nothing" and "change little" cheap to represent, so a network can be made very deep without the added layers undoing what earlier layers have already learned.

Why it matters

Two numbers carry the significance. The first is depth: 152 layers, eight times deeper than VGG nets and at lower complexity ^[1]. The second is accuracy: 3.57% error on the test set and first place in the classification task ^[1], a result that Wikipedia describes as beyond human performance ^[2]. Together they weaken the practical assumption that very deep networks are not worth training, because here the depth was not bought with a proportional increase in complexity, and it did not come at the cost of accuracy.

Where it stands

Nothing in the residual formulation is tied to one dataset or one contest: it is a statement about how to parametrize a stack of layers, and it applies wherever such stacks are used. What the 2015 result established publicly is that the limit on useful depth was not where it had been assumed to be, and the paper's way of putting the fix — learn the residual rather than the whole mapping — gave a compact way of describing why. The public record for this entry is the arXiv submission of 10 December 2015 ^[1] and the reported outcome: first place in the ILSVRC 2015 classification task with 3.57% error on the ImageNet test set, using residual nets of up to 152 layers ^[1], a winning entry that Wikipedia also records as exceeding human performance ^[2].