Residual connection

What it is

A residual connection, also called a skip connection, takes the input x of a block, sends it around the block, and adds it to whatever the block computes. The output is x + f(x), where f can be any neural module — a stack of convolution layers, an attention layer, a small network.^[1] The added path carries no parameters of its own; it simply passes the input forward.

How it works

The important part is what the block has to learn. If the mapping a layer should ideally compute is H, a block with a residual connection does not have to learn H directly: it learns the residual H − x, the amount by which the output should differ from the input.^[1]

In the forward pass, the output of a block equals its input plus the sum of the residual functions computed in between. The signal can therefore travel directly from a shallow layer to a deep one instead of being transformed at every step.^[1] In the backward pass, the gradient contains a direct term coming from the identity path, so it does not vanish easily even when the gradients through f are small.^[1]

A concrete illustration: if the input to a block already carries the information that is needed, the block can push f(x) close to zero and the output is simply x, unchanged. If the input needs adjusting, the block only has to learn that adjustment rather than the whole mapping from scratch.^[1]

Where it came from

The idea is associated with Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun. The architecture was developed in 2015 and published as "Deep Residual Learning for Image Recognition" at CVPR in 2016; the paper was first submitted in December 2015.^[1][2] It reformulates layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions.^[2]

The problem it addressed was degradation: plain networks that are simply stacked deeper performed worse than shallower ones.^[1] The authors provide evidence that residual networks are easier to optimize and can gain accuracy from considerably increased depth.^[2] They evaluated residual nets with a depth of up to 152 layers on ImageNet — 8x deeper than VGG nets but still with lower complexity — and an ensemble of these nets reached 3.57% error on the ImageNet test set, winning first place in the ILSVRC 2015 classification task.^[2] The original network, trained to 152 layers as ResNet-152, won the 2015 ILSVRC.^[1]

Where it is used today

Residual connections are used in transformer models, including BERT, GPT and ChatGPT.^[1] From GPT-2 onward, models adopted pre-normalisation, which places the normalisation inside the residual block, before the sub-layer, rather than after it.^[1] Very deep transformers cannot be trained without residual connections.^[1]

Limits

A residual connection makes depth trainable; it does not make it cheap. Pre-activation versions could train 200 to more than 1,000 layers, and training a 200-layer model took three weeks on eight GPUs in 2016.^[1] Depth still runs into the cost of computation, and the details of the design around the connection matter: the shift from post-normalisation to pre-normalisation in GPT-2 onward shows that where normalisation sits relative to the residual path changes how well very deep models train.^[1] What the connection removes is one specific obstacle, the difficulty of optimizing very deep plain stacks; everything else about building a deep network still has to be solved separately.