Word embeddings

What it is

A word embedding represents a word as a dense vector: a fixed-length list of real numbers learned from data. Words that are used in similar ways end up with vectors that lie close together.^[1] Before embeddings, many language systems treated each word as an arbitrary index in a vocabulary list, so the system had no built-in sense that "dog" and "cat" are related.^[2]

How it works

Here is a toy example with three numbers per word. Real systems use many more numbers, and the values are learned rather than typed in. Suppose "cat" is [0.9, 0.1, 0.4], "dog" is [0.8, 0.2, 0.5] and "car" is [0.1, 0.9, 0.7]. Comparing the lists, "cat" and "dog" are close to each other, and "car" is further away. A system can therefore treat "cat" and "dog" as related.

The vectors are learned by training a model on text. One approach is to predict a word from the words around it; the numbers that make those predictions work well become the embeddings. Learned vectors also support a kind of arithmetic that reflects relationships. In the 2013 word2vec paper, vector('King') − vector('Man') + vector('Woman') is closest to vector('Queen'), and vector('Germany') + vector('capital') is close to vector('Berlin').^[2]

Where it came from

The idea was introduced in a 2003 paper in the Journal of Machine Learning Research by Yoshua Bengio, Réjean Ducharme, Pascal Vincent and Christian Jauvin. Their neural language model learned a feature vector for every word, together with a network that predicts the next word. Its vectors had 30, 60 or 100 features for vocabularies of about 17,000 words.^[1]

The key benefit was generalisation: a sentence never seen in training can still receive a high probability if its words have vectors close to those of words in familiar sentences.^[1]

In January 2013 Tomas Mikolov and colleagues at Google proposed two simpler architectures. CBOW predicts a word from its context, and Skip-gram predicts the context from a word. Both learned high-quality vectors from a 1.6-billion-word dataset in less than a day.^[2] Google released the training code as the open-source word2vec project. Stanford's GloVe, presented at EMNLP 2014 by Jeffrey Pennington, Richard Socher and Christopher Manning, took a different route: it learned vectors from global counts of how often pairs of words occur together, and scored 75% on a word analogy task.^[4]

Where it is used

Embedding layers became the standard input layer of neural language systems. The 2014 sequence-to-sequence translation model, for example, used 1,000-dimensional word embeddings.^[5] Modern large language models follow the same basic pattern: text is first split into tokens, and each token is mapped to a learned vector before any other computation happens.

Limits

A classic word embedding gives each word a single vector no matter the context, and it ignores word order. The word2vec authors noted that such vectors cannot represent idiomatic phrases: "Air" plus "Canada" does not give "Air Canada".^[3] Later models addressed this by giving the same word different vectors in different sentences, but the basic idea, that meaning can be captured by position in a learned space, remains the foundation.