Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups

In October 2012, IEEE Signal Processing Magazine published an article by Geoffrey Hinton, Li Deng, Dong Yu, George E. Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N. Sainath and Brian Kingsbury: "Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups". It ran in volume 29, issue 6, pages 82–97.^[1]

The standard recipe

Most speech recognition systems at the time handled the problem in two stages. Hidden Markov models dealt with the temporal variability of speech — an utterance arrives as a sequence of sound units, and the model tracks which unit is being spoken as the signal unfolds — while Gaussian mixture models determined how well each state of that sequence fitted a frame, or a short window of frames, of the acoustic signal.^[1] A frame here is a short slice of the signal from which a vector of coefficients is computed. The mixture model supplies a score for each candidate state, and the recogniser searches for the sequence of states, and hence of words, that scores best overall.

The alternative the authors described

The authors described an alternative in which a feed-forward neural network takes several frames of coefficients as input and produces posterior probabilities over hidden Markov model states as output.^[1] Rather than asking a mixture of Gaussians how plausible a stretch of speech is under each state, the network receives a handful of consecutive frames stacked into one input vector and returns a probability for every state. Because the network is feed-forward, information moves in one direction only, from the input through the hidden layers to the output; because several frames are presented at once, each decision is made with several frames of context rather than a single instant of sound.

What the evidence showed

The article reported that deep neural networks — networks with many hidden layers, trained using new methods — had been shown to outperform Gaussian mixture models on a variety of speech recognition benchmarks, sometimes by a large margin.^[1] Depth is the point of the comparison: each hidden layer computes a new representation from the outputs of the layer below, so a deep stack can build progressively more abstract descriptions of the input instead of relying on one relatively shallow mapping. The article ties the gains to both ingredients — many hidden layers, and the new training methods that make such stacks trainable.

Four groups, one argument

The article was not the report of a single system. It presented the shared views of four research groups that had recently succeeded in using deep neural networks for acoustic modelling in speech recognition.^[1] Bringing the four accounts together let the authors describe a common recipe — frames in, state posteriors out, deep stacks trained with the new methods — rather than four separate experiments, and to set out where their experience agreed.

Why it mattered

The comparison the article drew sits at the level of a single component: the acoustic model that scores frames against states. Since the network still produces probabilities over hidden Markov model states, the rest of a recogniser — the state sequence, the search over words — does not have to be replaced.^[1] That is what made the argument portable. The claim was not that speech recognition needed to be rebuilt, but that the scoring model inside it could be swapped for a deep network, and that this network had been measured against the model it would displace on a variety of benchmarks, sometimes by a large margin.^[1]