Going Deeper with Convolutions

A deeper network at the same cost

Convolutional neural networks (CNNs) recognise images with stacks of learnable filters: early layers respond to simple patterns such as edges and colour patches, later layers combine those responses into parts and then into objects. The obvious way to make such a network more accurate is to make it bigger — more layers, more filters per layer. The catch is that both cost computation, and the cost grows quickly.

"Going Deeper with Convolutions", first submitted to arXiv on 17 September 2014, takes that trade-off as its subject ^[1]. The authors wanted to raise both the depth and the width of a deep convolutional network while holding the computational budget fixed ^[1]. The paper also entered the resulting system in the ImageNet Large-Scale Visual Recognition Challenge 2014, a public benchmark in which participants submit systems for image classification and object detection and are ranked against one another; the entry set a new state of the art in both tasks ^[1].

Inception and GoogLeNet

The architecture proposed in the paper carries the codename Inception. Its stated hallmark is not a particular layer type but better utilisation of the computing resources inside the network: the aim is to spend the available arithmetic where it buys the most, so that greater depth and width become affordable at a constant budget ^[1].

The paper says its architectural decisions were guided by the Hebbian principle and by an intuition about multi-scale processing ^[1]. The Hebbian principle comes from neuroscience — roughly, the idea that units which are active together tend to become more strongly connected — and functions in the paper as a design guide rather than a training rule. Multi-scale processing reflects a basic property of photographs: the same kind of object may fill most of the frame or occupy only a few dozen pixels, so a system fixed to a single scale does poorly on objects of other sizes.

The particular incarnation submitted to the 2014 challenge was named GoogLeNet, and it was 22 layers deep ^[1]. The two names refer to different things: Inception is the architecture, GoogLeNet the network built from it for the competition.

The result

According to the CVPR version of the paper, the final submission obtained a top-5 error of 6.67% on both the validation and the testing data and ranked first among the participants ^[2]. Top-5 error is the share of images for which the correct label is not among the system's five best guesses; lower is better.

The paper sets that number against earlier entries in a results table ^[2]. SuperVision, the 2012 approach, is listed at 16.4% and 15.3%, and Clarifai, the previous year's best, at 11.7% and 11.2% ^[2]. In the 2014 edition, MSRA is listed third at 7.35%, VGG second at 7.32% and GoogLeNet first at 6.67% ^[2]. The authors describe 6.67% as a 56.5% relative reduction compared with the SuperVision approach of 2012 and about a 40% relative reduction compared with Clarifai ^[2].

Why it matters

Two things stand out in that table. The first is the pace: within the same comparison, the best top-5 error falls from 16.4% to 6.67%, a change the authors themselves frame as a large relative reduction over two years ^[2]. The second is how tight the 2014 field had become. GoogLeNet's lead over the second-place entry is well under a percentage point, and the second and third entries are separated by hundredths of a point ^[2]. Once results compress like that, further gains have to come from somewhere other than simply adding capacity.

That is where the paper's argument lies. Its contribution is not only a winning network but a way of posing the design question: instead of asking how large a network can be made, ask how the fixed cost of a network should be distributed inside it, so that depth and width can both grow without the budget growing with them ^[1]. The paper also left the field with two durable names — Inception for the architecture and GoogLeNet for the 22-layer network that competed — which are easy to confuse and are not the same thing ^[1].

Read today, the paper is a compact record of a moment when accuracy gains were being extracted less from raw scale than from the internal economics of a network: which parts of it deserve more computation, and which can be computed cheaply without losing what the model needs to see. The multi-scale intuition behind the architecture — that objects arrive at many sizes and a network should be built to expect that — is stated plainly and needs no result table to be convincing ^[1].