Deep Residual Learning for Image Recognition

Overview

The paper Deep Residual Learning for Image Recognition was submitted to arXiv on 10 December 2015 by Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun of Microsoft Research. It was published at CVPR 2016, where it received the CVPR Best Paper Award.^[1][2]

The problem

Once normalised initialisation and normalisation layers allowed networks of tens of layers to start converging, a new problem appeared. In principle, a deeper network should do at least as well as a shallower one, since the extra layers could learn to pass their input through unchanged. In practice, training did not find such solutions easily. Adding more layers to a suitably deep model made accuracy saturate and then degrade. Training error went up, and the authors stressed that this was not caused by overfitting. They called it the degradation problem.^[2]

The idea

Instead of asking a stack of layers to learn a desired mapping H(x) directly, the authors let the layers learn a residual F(x) = H(x) − x. The input is then added back through a shortcut connection, so the block's output is F(x) + x. If an identity mapping is close to optimal, the layers only need to push the residual towards zero, which is easier than learning the identity from scratch.^[1]

The identity shortcuts add neither extra parameters nor extra computation. The whole network is still trained end to end by stochastic gradient descent with backpropagation.^[1]

Results

On ImageNet the authors trained residual networks up to 152 layers deep, eight times deeper than VGG networks but with lower complexity. Even their 34-layer baseline needed 3.6 billion FLOPs, 18% of the 19.6 billion FLOPs required by VGG-19. An ensemble of residual networks reached 3.57% top-5 error on the ImageNet test set and won first place in the ILSVRC 2015 classification task.^[2]

Residual networks also took first place in ImageNet detection and localisation, and in COCO detection and segmentation, in 2015. On COCO object detection they gave a 28% relative improvement.^[2] The paper also analysed residual networks with 100 and 1,000 layers on the CIFAR-10 dataset.^[2]

Why it matters

The residual connection is a simple change to a network's wiring, but it removed the barrier that had stopped very deep models from improving. Because the shortcut adds no parameters, a residual block can be inserted into existing designs without changing their size.^[1] In plain terms, the shortcut gives the signal a direct route through the network, so information and training error can pass across many layers without being weakened at each one.

Later influence

According to Kaiming He's homepage, residual connections are now used throughout modern deep learning, including Transformers, AlphaGo Zero and AlphaFold. In 2026 the paper received the CVPR Test of Time Award (Longuet-Higgins Prize).^[3]