NETtalk

What NETtalk is

NETtalk is an artificial neural network that learns to read English text aloud. It takes written English as input and produces a phonetic transcription as output, acquiring the mapping from spelling to sound through supervised learning rather than from hand-written pronunciation rules^[1]. The transcription is a stream of phonemes, which the researchers fed into DECtalk, a speech synthesizer, so that the network's output could actually be heard^[1].

Where it came from

NETtalk is the result of research carried out in the mid-1980s by Terrence Sejnowski and Charles Rosenberg, with the original paper published in 1987^[1]. The paper, titled Parallel networks that learn to pronounce English text, appeared in Complex Systems volume 1, pages 145 to 168^[2]. Its authors are listed as Terrence J. Sejnowski of the Department of Biophysics at The Johns Hopkins University and Charles R. Rosenberg of the Cognitive Science Laboratory at Princeton University^[3].

How the network was built

The network had three layers and 18,629 adjustable weights, which was described as large by the standards of 1986^[1]. Its input had 203 units divided into 7 groups of 29 one-hot encoded characters: the network looked at a target character together with the three characters before it and the three after it^[1]. The hidden layer had 80 units. The 26 output units were split between two jobs, with 21 of them encoding articulatory features and 5 encoding stress and syllable boundaries^[1].

The training set was a 20,008-word subset of the Brown Corpus, with phonemes and stress annotated by hand. Creating that dataset took three months, or roughly 250 person-hours, while training the network itself took only a few days^[1]. Training used backpropagation on English words paired with their pronunciations, and the trained network could generate pronunciations for unseen words with a high level of accuracy^[1].

Two implementations are recorded. The original implementation on a Ridge 32 took 0.275 seconds per learning step, and a Connection Machine-1 implementation with 16,384 processors ran at a 52x speedup^[1].

What the experiments showed

Sejnowski found that phonemes that sound similar clustered together in the network's representation space, and that when some hidden neurons were removed, the output degraded but remained understandable^[1]. The published abstract describes NETtalk as a class of massively parallel network systems that learn to convert English text to speech, in which memory representations for pronunciations are learned by practice and shared among many processing units^[2]. It also reports that the more words the network learns, the better it is at generalizing and correctly pronouncing new words, and that performance degrades very slowly as connections in the network are damaged^[2].

Reception and place in the field

NETtalk achieved popular success and appeared on the Today show^[1]. Its interest for later work lies in what the design demonstrates: a single set of weights, shared across the processing units that hold pronunciation knowledge^[2], is enough for a general learning procedure to pick up an irregular mapping between spelling and sound from examples alone, and to carry that mapping over to words it never saw during training^[1]. The graceful degradation under damage, described in the abstract, points in the same direction: knowledge stored in a distributed way spreads the loss rather than destroying it all at once^[2].