A Neural Probabilistic Language Model

The paper

"A Neural Probabilistic Language Model" was published in the Journal of Machine Learning Research, volume 3, pages 1137–1155. It was submitted in April 2002 and published in February 2003. The authors were Yoshua Bengio, Réjean Ducharme, Pascal Vincent and Christian Jauvin of the Université de Montréal.^[1]

The problem

A language model estimates how probable a word sequence is, which matters for speech recognition, translation and text generation. The main point of comparison was the n-gram model, which estimates the probability of the next word from the few words just before it, by counting short sequences seen in training. The authors point to what they call the curse of dimensionality. Modelling 10 consecutive words from a 100,000-word vocabulary would need up to 10^50 − 1 free parameters, and n-gram models generalise only by stitching together very short word sequences seen in training.^[1]

The proposal

The proposal is to give every word a learned distributed feature vector, to express the probability of a word sequence in terms of those vectors, and to learn the vectors and the probability function at the same time.^[1] Instead of treating words as unrelated symbols, the model lets words that behave alike end up near each other in a learned space.

How the network works

A shared lookup table maps each preceding word to its feature vector. The vectors pass through a tanh hidden layer, a standard kind of nonlinear layer, and then a softmax output over the vocabulary. The softmax turns scores into probabilities, giving the chance of each possible next word.^[2]

This is why the model can generalise. A sentence never seen in training can still get a high probability if its words have vectors close to those of words in a sentence that was seen. "Dog" and "cat", or "bedroom" and "room", are examples of words that can stand in for one another.^[2]

Results

The authors tested the model on two datasets: the Brown corpus, with 1,181,041 words and a vocabulary reduced to 16,383, and Associated Press news from 1995–1996, with about 14 million training words and a vocabulary of 17,964.^[2] Perplexity is a standard score for language models, where lower means the model finds the text less surprising. Test perplexity was about 24% lower than the best n-gram model on Brown, and about 8% lower on AP News. The network also benefited from a longer context, and mixing it with a trigram model lowered perplexity further.^[2]

Cost and scale

Training models with millions of parameters was itself a challenge. The authors built parallel implementations, and the AP News run took 5 epochs, meaning full passes over the training data, over about three weeks on 40 CPUs.^[2]

Why it mattered

The ACM's 2019 Turing Award announcement credits this work with introducing high-dimensional word embeddings, and says the embeddings influenced machine translation, question answering and visual question answering. ACM dates the work to 2000, while the journal version is from 2003.^[3]

Later, simpler methods made word vectors cheap enough to learn from very large text collections. The core idea of this paper is what made that line of work possible.^[1]