Deep learning

What the review set out to cover

By the time this review appeared, deep learning methods had dramatically improved the state of the art in speech recognition, visual object recognition, object detection and many other domains such as drug discovery and genomics^[1]. The article that set out the ideas behind those results was written by Yann LeCun, Yoshua Bengio and Geoffrey Hinton and published in Nature, volume 521, pages 436 to 444, in 2015^[1]. It is not the announcement of a single new model: it lays out a definition, the training method, and the families of networks that had produced the results.

Layers, and levels of abstraction

The definition the authors give is compact: deep learning allows computational models that are composed of multiple processing layers to learn representations of data with multiple levels of abstraction^[1]. A representation is simply the way data is described inside the model — which numbers stand for an image, a stretch of audio or a sentence. In older pipelines those descriptions were often designed by hand; here the layers produce them. Each layer rewrites the output of the layer below it, so the description at the top is built out of the descriptions underneath. If the input is an image, the lower layers may respond only to small local variations in brightness, while the upper layers combine those local cues into larger structures.

Training by backpropagation

The review also states how the parameters get set. Deep learning discovers intricate structure in large data sets by using the backpropagation algorithm to indicate how a machine should change its internal parameters^[1]. In plain terms, the model makes a prediction, the error is measured, and the error signal is passed back through the layers so that each parameter receives a direction in which to move. Repeated over a large data set, this gradually shapes the internal representation. Backpropagation is therefore the part that connects the layered architecture to the data.

Different nets for different data

The authors distinguish two families of networks. Deep convolutional nets, they write, had brought breakthroughs in processing images, video, speech and audio^[1]. A convolution slides small filters across a signal, so the same local pattern can be picked up wherever it occurs, which suits data laid out on a grid or along time. Recurrent nets, by contrast, had shone light on sequential data such as text and speech^[1]: a recurrence carries state from one element of a sequence to the next, so the order of the elements matters. Which architecture fits depends on the shape of the data.

What the review claimed

The case for importance in the article is empirical rather than programmatic: these methods had improved the state of the art across the domains the authors list, from speech and object recognition to drug discovery and genomics^[1]. By placing the definition, the training algorithm and the two network families in one text, the review gave readers a shared vocabulary — multiple processing layers, levels of abstraction, backpropagation — for talking about a fast-moving area. Nature lists the article as volume 521, pages 436 to 444, published in 2015^[1].