Efficient Estimation of Word Representations in Vector Space

The paper

"Efficient Estimation of Word Representations in Vector Space" was submitted to arXiv on 16 January 2013 by Tomas Mikolov, Kai Chen, Greg Corrado and Jeffrey Dean, all at Google in Mountain View. This paper and its follow-up are the papers behind word2vec.^[1]

The problem

Earlier neural network methods for learning word vectors had not been trained on more than a few hundred million words, and most of their cost came from the non-linear hidden layer. Learning vectors from much more text required reducing that cost.^[1]

The method

The paper proposes two simple log-linear architectures without that hidden layer. Continuous Bag-of-Words (CBOW) predicts a word from its surrounding words. Continuous Skip-gram predicts the surrounding words from the current word.^[1] For example, in "the cat sat on the mat", CBOW would use the words around the gap, "the cat ___ on the mat", to guess "sat", while Skip-gram would start from "sat" and try to predict the nearby "cat" and "on". The "bag" in CBOW means the order of the surrounding words is ignored.

The test

To measure quality, the authors built a new Semantic-Syntactic Word Relationship test set with 8,869 semantic and 10,675 syntactic analogy questions, such as "France is to Paris as Germany is to Berlin". A model is scored on how often it fills in the missing word correctly.^[1]

The training data was a Google News corpus of about 6 billion tokens, with the vocabulary restricted to the 1 million most frequent words. Some models were trained with Google's DistBelief distributed framework.^[1]

Results

Training took less than a day to produce high-quality vectors from 1.6 billion words. A 300-dimensional Skip-gram model reached 53.8% total accuracy on the test set after one epoch on 1.6 billion words, in about 2 days of training.^[1] The paper restates an earlier finding: vector('King') − vector('Man') + vector('Woman') is closest to the vector for "Queen".^[1]

Code and follow-up

After the first version, the team published single-machine, multi-threaded C++ code that trained at billions of words per hour, plus more than 1.4 million vectors for named entities trained on over 100 billion words. That code is the open-source word2vec project.^[1]

The NIPS 2013 follow-up, by Mikolov, Ilya Sutskever, Chen, Corrado and Dean, added negative sampling, subsampling of frequent words and vectors for phrases. It received the NeurIPS 2023 Test of Time Award.^[3][4]

Why it mattered

The paper's contribution was practical. Its models were simple and fast enough to train on corpora far larger than earlier neural methods could handle, which made word vectors a routine input for language systems.^[1]

The evaluation mattered as much as the training method. Because the analogy test set covers both meaning and grammar, it shows which kinds of relationships the vectors capture, rather than giving only one overall score.^[1]

Another lesson is that a simple model trained on a large amount of text can capture more than a complex model trained on little. The paper's claims are limited to the analogy tasks it tested, but within that scope the results point in that direction.^[1]