WaveNet: A Generative Model for Raw Audio

A network for raw waveforms

Sound stored as a waveform is a long sequence of numbers: a single second of audio can contain tens of thousands of samples ^[1]. WaveNet is a deep neural network built to generate exactly that — raw audio waveforms, rather than some intermediate representation of the sound ^[1]. The paper "WaveNet: A Generative Model for Raw Audio" was first submitted to arXiv on 12 September 2016 ^[1]. Its author list runs from Aaron van den Oord and Sander Dieleman to Andrew Senior and Koray Kavukcuoglu, nine names in all ^[1].

Modelling one sample at a time

The model is fully probabilistic and autoregressive: it gives a probability distribution for each audio sample, and that distribution is conditioned on all the samples that came before it ^[1]. Generation follows directly from that description. You give the network some audio, ask it for the distribution of the next sample, draw a value from that distribution, append the value to the input, and repeat. Because the model describes a distribution rather than one fixed output, different draws produce different sounds.

The awkward part is scale. At tens of thousands of samples per second of audio, an autoregressive model has to make tens of thousands of predictions for every second of sound it produces. The authors show that the model can nevertheless be trained efficiently on data at that rate ^[1].

Speech

Applied to text-to-speech, the model reaches state-of-the-art performance: human listeners rated it significantly more natural than the best parametric and concatenative systems, for both English and Mandarin ^[1]. The two older families work differently. Parametric systems synthesise speech from a compact set of acoustic parameters, and concatenative systems assemble recordings of short speech units; both build a waveform in stages, passing through hand-designed intermediate representations. WaveNet instead produces the waveform itself.

A single WaveNet can also capture the characteristics of many speakers and switch between them by conditioning on speaker identity ^[1]. One model therefore holds many voices, and which voice comes out is an input to the network rather than a separate system per speaker.

Music and recognition

Trained on music, the model generates novel and often highly realistic musical fragments ^[1]. The authors also show that it can serve as a discriminative model for phoneme recognition ^[1] — that is, a network of the same kind that generates audio can also be trained to say what is in it.

Why it matters

The paper's contribution is to show that a single generative model can work directly on the waveform, down at the level of individual samples, and still produce speech that listeners judge better than the output of established systems ^[1]. Several things are folded into one framework: text-to-speech, music generation and phoneme recognition all use the same kind of model; one model can cover many speakers; and probabilistic autoregressive modelling remains trainable at the density of audio data ^[1]. The result is a demonstration that the waveform itself, rather than a hand-built intermediate stage, can be the object a generative model learns to predict.