Deep Speech: Scaling up end-to-end speech recognition

The problem

Traditional speech recognition systems are built from laboriously engineered processing pipelines. The authors of Deep Speech describe that style of system as complex to construct and note that it tends to perform poorly in noisy environments.^[1] In a conventional pipeline, different stages handle different parts of the problem, and engineers must design components for background noise, reverberation and variation between speakers. The paper starts from a different premise: instead of chaining separately designed parts, it presents a single system trained end to end to recognise speech.^[1]

An end-to-end system

Deep Speech presents a speech recognition system built with end-to-end deep learning. The authors describe the architecture as significantly simpler than traditional systems.^[1] The system does not need hand-designed components to model background noise, reverberation or speaker variation. It instead learns a function that is robust to those effects directly.^[1] It also requires neither a phoneme dictionary nor even the concept of a phoneme.^[1] This is the main break from conventional pipelines. The model is not given a hand-built inventory of speech sounds; the mapping from audio to recognition output is learned as a whole. That makes the architecture simpler at the level of system design, because many of the pieces that would otherwise be specified by an engineer become part of what the network learns.^[1]

Training at scale

The approach depends on a well-optimised recurrent network training system that uses multiple GPUs.^[1] A recurrent network processes a sequence step by step while maintaining an internal state, which makes it a natural fit for audio that unfolds over time. The paper's system relies on this kind of sequence model and on training infrastructure designed to use several GPUs together.^[1] The authors also describe data synthesis techniques used to obtain large amounts of varied training data.^[1] Rather than depending only on a fixed, hand-collected set, training can draw on generated variation. That matters because the model is meant to absorb noise, reverberation and speaker differences from data rather than from manually written rules.^[1]

Why it matters

Deep Speech was first submitted to arXiv on 17 December 2014 as 1412.5567.^[1] Its author list runs to eleven names, from Awni Hannun to Andrew Y. Ng.^[1] The paper's importance lies in the approach it sets out. It treats speech recognition as an end-to-end learning problem, removes the need for a phoneme dictionary and the phoneme concept, and avoids hand-designed modules for acoustic variability.^[1] It also connects that approach to a practical training stack: optimised recurrent networks, multiple GPUs and synthesized data.^[1] In other words, the paper argues for replacing part of the manual engineering in traditional pipelines with scale in computation and data. It does not assign noise, reverberation or speaker variation to separate hand-built components; it asks the model to learn a robust function from examples. Nor does it require the system to start from a phoneme-level representation.^[1] These are the claims the paper makes for its system design.