DeepMind introduces WaveNet for raw audio generation

Background

Turning written text into speech is usually done in stages, and the last stage — producing an actual waveform from some abstract representation of sound — has traditionally been handled by two families of methods. Concatenative synthesis assembles short recorded fragments of a human voice into a longer utterance; parametric synthesis generates a compact set of acoustic parameters and drives a vocoder with them. Google had both kinds of system, and the comparison DeepMind later reported was made against what it called Google's best concatenative and parametric text-to-speech systems^[1].

Raw audio poses a different kind of problem. A waveform is a long sequence of numbers: raw audio typically runs at 16,000 samples per second or more^[1]. A model that generates the waveform itself has to settle every one of those numbers, rather than choosing among units that were recorded in advance.

The announcement

On 8 September 2016, DeepMind announced WaveNet, a deep neural network for generating raw audio waveforms^[1]. The accompanying blog post described the model as a fully convolutional network^[1] — that is, a network built entirely out of convolution layers, the kind of layer that slides a small learned filter across its input. The model predicts audio one sample at a time from previous samples^[1].

How it generates sound

WaveNet works as an autoregressive model: it is given the samples that came before, and it produces a prediction for the next sample; that sample is then fed back in, and the process repeats. Because the prediction happens directly in the sample domain, the same machinery applies to any waveform rather than to a hand-designed intermediate representation. The cost is sequence length. At 16,000 samples per second, one second of audio requires 16,000 successive predictions, each depending on everything that came before it, so making the computation tractable is the central engineering problem the architecture has to answer^[1].

Results

DeepMind reported blind listening tests in US English and Mandarin Chinese in which WaveNet beat Google's best concatenative and parametric text-to-speech systems^[1]. The company also said it had narrowed the gap with human speech quality by over 50% in US English and Mandarin^[1]. The quantity being measured is the remaining distance from human speech, and according to DeepMind that distance fell by more than half in both languages.

Paper and later use

The technical paper, "WaveNet: A Generative Model for Raw Audio", is dated 12 September 2016 on arXiv^[2]. Work on making the model faster continued, and in October 2017 a much faster version began producing voices for Google Assistant^[2] — the model moving from a research result to a component of a product that people would actually hear.

Why it matters

Two things stand out. First, WaveNet handed waveform generation to the network itself: instead of picking among pre-recorded fragments, the model produces sound one sample at a time^[1]. Second, the approach did not stay in the paper. A year later a faster version was generating speech inside a widely used assistant^[2]. For text-to-speech, that established a different route: rather than reassembling recordings, let a model generate the audio directly.