GloVe: Global Vectors for Word Representation

The problem

A word embedding represents each word as a fixed-length list of numbers, positioned so that words used in similar ways end up close together in that space. Models cannot read text directly, so such representations are the usual entry point for turning language into numbers. Two broad families of methods were available. Count-based methods build a large table recording how often pairs of words appear near each other in a corpus, then compress that table into short, dense vectors. Prediction-based methods train a model to guess neighbouring words from a local context window — the handful of words on either side of a target word. Each captures something the other misses: the count table summarises the whole corpus at once, while window training concentrates on local usage. GloVe is described as a global log-bilinear regression model for word representation that combines global matrix factorisation of word co-occurrence counts with local context-window training^[2].

How it works

The input is a co-occurrence matrix: for every pair of words, how many times one of them appeared inside a window around the other. The model gives each word a vector and fits those vectors so that the dot product of two words' vectors, plus a bias term for each word, comes close to the logarithm of their co-occurrence count. Taking the logarithm matters because raw counts span an enormous range; the log turns a few extremely frequent pairs and a vast number of rare ones onto a comparable scale. Each non-zero entry of the matrix counts as one training sample, and the fit is a weighted least-squares problem in which very frequent word pairs are given less weight, so that a small set of common words does not dominate the objective.

What comes out is a set of vectors defined by how often words occur together across the whole corpus, rather than by any single local prediction task. That is what the title means by "global": the statistics being factorised are collected over the entire corpus, while the contexts being counted are still local windows.

Publication

The paper is titled "GloVe: Global Vectors for Word Representation" and is listed in the ACL Anthology as D14-1162^[1]. It was written by Jeffrey Pennington, Richard Socher and Christopher Manning^[1]. It appeared in the Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), held in Doha, Qatar, in October 2014, on pages 1532 to 1543, with DOI 10.3115/v1/D14-1162^[1]. The publisher is the Association for Computational Linguistics, under its SIGDAT special interest group^[1].

What was released

Alongside the paper, the authors released pre-trained vectors trained on large web corpora^[2]. This mattered for practical use: instead of gathering a corpus and running training themselves, people could download the vectors and start using them. A word could be turned into numbers in minutes.

Why it matters

GloVe was released in 2014, the same year as the word2vec papers, and became one of the two standard ways of producing static word embeddings^[1]. Its importance is twofold. On the method side, it put the two earlier routes into a single objective: instead of choosing between counting and predicting, it factorises global co-occurrence counts directly with a training procedure that still works over local windows, which made clear that the two views of word meaning were not in conflict. On the practical side, the published vectors lowered the barrier to use, turning word embeddings from something a team had to train into a resource anyone could pick up^[2].

The limitation of the whole family is easy to state: a static embedding gives every word one vector no matter which sentence it appears in, so the different senses of a polysemous word are squeezed into the same point. That limitation belongs to the class of methods rather than to one model. Even so, mapping words to dense vectors has remained a routine starting point for work on language, and the GloVe vectors have been one of the standard sets people reach for.