Microsoft Research's deep-neural-network speech recognition breakthrough

Background

Conventional large-vocabulary speech recognisers combined two statistical components. A hidden Markov model (HMM) tracked how sound unfolded over time, while a Gaussian mixture model (GMM) scored how well each short slice of audio matched a particular state. Because neighbouring sounds blur into one another, systems used context-dependent states, and to keep the numbers manageable those states were tied together into clusters called senones. Such CD-GMM-HMM systems, trained discriminatively, were a strong baseline.

What was presented at Interspeech 2011

At Interspeech 2011, held in Florence from 28 to 31 August, Microsoft Research presented work it described as a research breakthrough in using artificial neural networks for large-vocabulary speech recognition ^[1]. Microsoft Research says the work was spearheaded by Dong Yu of Microsoft Research Redmond and Frank Seide of Microsoft Research Asia, and that University of Toronto intern George Dahl joined the team in June 2010 ^[1].

The team did not treat the neural network as a small add-on stage. It proposed modelling thousands of senones directly with deep neural networks, producing the first hybrid context-dependent DNN-HMM model applied successfully to large-vocabulary speech recognition ^[1]. In a hybrid system of this kind the network takes over the acoustic scoring that the Gaussian mixture used to do, while the HMM still handles the sequence of states and a language model still constrains which word sequences are plausible, so the recogniser's decoding machinery can be kept. To make the experiments feasible, the team used general-purpose graphics processing units to train and decode speech ^[1].

Results

On a voice-search task, Microsoft Research reports a relative error reduction of more than 16 percent over state-of-the-art conventional CD-GMM-HMM systems ^[1]. The Interspeech 2011 paper by Frank Seide, Gang Li and Dong Yu applied CD-DNN-HMMs to the Switchboard phone-call transcription benchmark ^[2]. There the word error rate fell from 27.4% for discriminatively trained Gaussian-mixture HMMs to 18.5%, a 33% relative improvement, using over 300 hours of training data, over 9000 tied states and up to 9 hidden layers ^[2]. The same paper reports relative error reductions of 22–28% on four less well-matched transcription tasks ^[2].

The two headline numbers come from different settings: the 16 percent figure is the gain on the voice-search task, the 33 percent figure the gain on Switchboard ^[1][2].

Why it mattered

Two aspects of the result stand out. The first is size: a drop in word error rate from 27.4% to 18.5% on a public conversational benchmark is large, and the fact that four less well-matched tasks still showed 22–28% relative reductions indicates the gain was not confined to one carefully tuned condition ^[2].

The second is engineering fit. Because the DNN replaced only the acoustic scorer, the hybrid design could sit inside existing HMM-based decoders rather than requiring a new kind of recogniser, and GPUs made training and repeated experiments affordable ^[1]. For a field that had long built its systems around Gaussian mixtures, this showed that a deep network could carry the acoustic modelling at large vocabulary, with the rest of the pipeline left in place.