WaveNet

What WaveNet is

WaveNet is a deep generative model of raw audio waveforms, developed at DeepMind. It does not work from a compressed, hand-designed representation of sound; it works directly on the waveform, which typically runs at 16,000 samples per second or more.^[1] The model was announced on 8 September 2016, and the paper, with Aäron van den Oord as first author, was dated 12 September 2016 on arXiv.^[1][2]

How it works

WaveNet is a fully convolutional network in which each audio sample is predicted from the samples before it.^[1] Generating sound this way means making tens of thousands of predictions for every second of audio, and the network still has to hold on to enough of the past to keep pitch and timbre consistent. The convolutional layers use varying dilation factors, so the receptive field grows exponentially with depth and covers thousands of timesteps.^[1] A stack of relatively few layers therefore sees a wide stretch of the waveform, and each new sample is conditioned on a long run of earlier ones.

Because the model works on the raw waveform one sample at a time, nothing in it is specific to speech: the same model can learn any kind of audio, including music.^[1] Producing samples one after another is also inherently sequential, which makes speed a practical concern for any deployment.

Results

The first widely reported results were in text-to-speech. In listening tests, WaveNet reduced the gap between the previous state of the art and human-level performance by over 50% for both US English and Mandarin Chinese.^[1] The numbers came from blind listening tests in which listeners rated naturalness: WaveNet scored 4.21 for US English, against 4.55 for natural speech, and 4.08 for Mandarin Chinese, against 4.21 for natural speech.^[1][4] The margin was small — close to human quality, but not equal to it.

The architecture was not limited to voices. Trained on classical piano music, it produced novel piano samples.^[1] A single WaveNet model could also learn many voices, and training on many speakers improved its modelling of a single speaker.^[1]

From research model to product

In October 2017 Google announced a 1,000-fold performance improvement, and WaveNet began generating Google Assistant voices for US English and Japanese.^[2] Google then launched Cloud Text-to-Speech based on WaveNet in 2018, putting the model in front of developers as a service.^[3]

Why it matters

What made WaveNet notable was that it generated the waveform itself, sample by sample, rather than depending on a separate representation of sound sitting between the model and the audio. The same trained model carried over to different voices and to music, so one architecture covered cases that had previously needed separate machinery.^[1] Its dilated convolutional design is also what made a long receptive field affordable: depth buys range, and the model still sees thousands of timesteps of context.^[1]

The path from paper to product was short. Within about a year of publication the model was producing voices inside a widely used assistant, and within about two years it was available as a cloud service.^[2][3] For a research model, that is an unusually direct route into everyday use — and it is why WaveNet is usually treated as the point where generating raw audio with neural networks became a practical engineering option rather than a demonstration.