Speech Recognition with Deep Recurrent Neural Networks

The problem

Speech recognition turns a recording of speech into a sequence of words. That is awkward for an ordinary classifier. The output is much shorter than the input, and the training data usually says nothing about which stretch of audio corresponds to which sound — the alignment is unknown. Systems therefore needed extra machinery to guess that alignment before anything could be classified.

The paper puts the alternative in one sentence: end-to-end training methods such as connectionist temporal classification make it possible to train recurrent neural networks for sequence labelling problems where the input-output alignment is unknown ^[1]. Connectionist temporal classification, usually shortened to CTC, does this by letting the network emit a label or a blank at every step and then summing over all the ways an output sequence could be aligned with the input. Nobody has to mark the boundaries by hand; the alignment is marginalised out during training.

What they tried

The paper investigates deep recurrent neural networks. A recurrent network reads its input one step at a time and carries a hidden state forward, so information from earlier frames can still influence a decision made much later; the authors describe this as the flexible use of long-range context that empowers recurrent networks ^[1]. Making such a network deep means stacking layers, so that higher layers build more abstract representations of the signal, the same idea that made deep feedforward networks effective. The paper frames its subject as the combination of those two things: the multiple levels of representation that have proved effective in deep networks, joined to the long-range context that recurrence provides ^[1].

Long short-term memory, the recurrent architecture the authors lean on, adds gates that decide what to keep, what to write and what to expose at each step. Plain recurrent networks tend to lose information over long spans; gating gives them a way to hold onto it.

The starting point

The paper is candid about where recurrence stood. The authors note that recurrent network performance in speech recognition had so far been disappointing, with better results returned by deep feedforward networks ^[1]. In handwriting, by contrast, the picture was already different: combining end-to-end methods with the long short-term memory architecture had delivered state-of-the-art results in cursive handwriting recognition ^[1]. That contrast is the background against which the speech experiments are set — a recipe that had worked on one sequence problem, tried on a harder one.

Why it matters

The paper's contribution is as much a matter of framing as of results. It puts three threads — deep architectures, recurrent architectures, and end-to-end sequence training — into a single frame, and asks what happens when they are combined rather than used separately. If the recipe works, the hand-built stages that dominated speech recognition, in particular the forced alignment of audio to sub-word units, become optional: a network can be trained on pairs of audio and transcript and learn the correspondence itself.

The trade-off is real. End-to-end training shifts work from engineering to data and computation, because the model must discover the alignment that a pipeline would have supplied. Making a recurrent network deep enough to build good representations makes that computation heavier still.

The paper, titled “Speech Recognition with Deep Recurrent Neural Networks”, was written by Alex Graves, Abdel-rahman Mohamed and Geoffrey Hinton and posted to arXiv on 22 March 2013 ^[1].