Identity Mappings in Deep Residual Networks

Background

Deep residual networks (ResNets) are built out of residual blocks. A block takes its input, pushes it through a few layers to compute a transformation, and adds that result back to the input; the input itself also travels around the block along a shortcut, usually called a skip connection. Stacking many such blocks lets a network become very deep, because the shortcut offers the signal a route that does not pass through every layer. The paper was first submitted to arXiv on 16 March 2016 as 1603.05027, by Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun.^[1] What it examines is the fine print of the arrangement: what the shortcut should compute, and where the nonlinear activation should sit. The authors analyse the propagation formulations behind residual building blocks to see how signals actually move through a stack of them.^[1]

The analysis

Their conclusion is that when identity mappings are used as skip connections and the activation is placed after the addition, forward and backward signals can be propagated directly from one block to any other block.^[1] An identity mapping means the shortcut changes nothing: whatever enters along that path leaves the block unchanged. If instead the shortcut scales its input, or runs it through a layer of its own, every block puts something in the way of the signal, and information has to pass through that operation again and again as the stack grows.

With the shortcut left as an identity, the direct route is unobstructed in both directions of travel. Forward, the representation computed at one block is available, essentially unchanged, at any later block. Backward, the gradient used to train the earlier layers can reach them along the same path rather than only through the stacked transformations. What each block still learns is its residual: the correction added on top of the signal that arrives through the shortcut.

The authors do not stop at the argument. A series of ablation experiments — versions of the block with individual design choices changed one at a time — supports the importance of these identity mappings.^[1]

A new unit and the results

On that basis the authors propose a new residual unit, which they say makes training easier and improves generalisation.^[1] They report 4.62% error on CIFAR-10 with a 1001-layer ResNet, plus improved results on CIFAR-100,^[1] and they also report a 200-layer ResNet on ImageNet.^[1]

The striking part is the depth. A network of a thousand layers goes well beyond what plainly stacked layers can be trained to do; that this model both trained and generalised is the extreme case the residual idea was aiming at. The ImageNet result shows the same unit working at a more usual scale on a larger and harder benchmark.

Why it matters

The paper matters in two ways. First, it offers a reason for the residual design rather than just a recipe: the identity shortcut creates a direct route for forward and backward signals, and that route is what makes very deep stacks trainable. Second, it turns the reason into a design rule that anyone can apply or test — keep the shortcut an identity, put the activation after the addition. An ablation is a cheap way to check such a rule, and here the experiments point the same way as the analysis.

There is also a practical lesson about depth. If the shortcut is clean, adding blocks does not by itself make optimisation harder, which is how the reported networks could be so deep. The unit proposed here is a small change in wiring, but it is the kind of change that can decide whether a deep model trains at all.