Deep learning on GPUs

What it is

A graphics processing unit (GPU) is a chip originally built to draw images on screen. It can carry out a very large number of simple arithmetic operations at the same time. Training a neural network consists largely of big matrix multiplications and convolutions, which can be split into many independent pieces. That is why GPUs became central hardware for deep learning.

How it works

Consider a layer of a neural network that takes many inputs and produces many outputs. Each output is a weighted sum of the inputs, a dot product, and each one can be computed without waiting for the others. A GPU assigns these small calculations to many cores that run together, so the whole layer is finished far sooner than if the calculations ran one after another. Training repeats this forward computation, and then a backward pass that calculates how to adjust the weights, many times over.

Where it came from

In March 2010 Dan Ciresan, Ueli Meier, Luca Maria Gambardella and Jürgen Schmidhuber reported that plain multi-layer networks with many hidden layers, trained with ordinary online backpropagation on many deformed training images, reached a 0.35% error rate on the MNIST handwritten-digit benchmark. They noted that graphics cards were needed to speed up learning enough to make this possible.^[1]

In 2012 the AlexNet paper by Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton stated that convolutional networks had been prohibitively expensive to apply at large scale to high-resolution images. The authors argued that current GPUs, paired with a highly optimised implementation of 2D convolution, made training such large networks feasible.^[2]

In October 2014 NVIDIA researchers described cuDNN, a library of optimised GPU routines for deep learning, modelled on the BLAS linear-algebra libraries. Integrating it into the Caffe framework improved performance on a standard model by 36% while reducing memory use.^[3]

In January 2016 the Nature paper describing AlphaGo reported that the program evaluated its policy and value networks on GPUs. The single-machine version used 48 CPUs and 8 GPUs, and the distributed version used 1,202 CPUs and 176 GPUs.^[4]

Where it is used today

The 2017 Transformer paper reported that its large model trained in 3.5 days on eight GPUs. The Transformer is built from attention operations that can be computed in parallel across the positions of a sequence, and that kind of architecture fits GPU hardware well. Its arrival is part of the road to today's large models.^[5]

Limitations

GPU memory is limited, and the constraint can shape the design of a network. AlexNet, with 60 million parameters, took five to six days to train on two NVIDIA GTX 580 GPUs with 3 GB of memory each. Because one GPU's memory could not hold the whole network, the authors split it across the two GPUs, letting them communicate only in certain layers.^[2]

Progress also depended on hardware. The AlexNet authors wrote that their experiments suggested results could be improved simply by waiting for faster GPUs and bigger datasets. Training large models remains costly in both time and computing equipment, which is one reason the speed of GPUs matters so much.^[2]