Sequence to Sequence Learning with Neural Networks

The paper

"Sequence to Sequence Learning with Neural Networks" was submitted to arXiv on 10 September 2014 by Ilya Sutskever, Oriol Vinyals and Quoc V. Le of Google. It was published at NIPS 2014, the Advances in Neural Information Processing Systems conference.^[1][2]

The problem

Deep neural networks worked well when large labelled datasets were available, but they could not be used directly to map an input sequence to an output sequence of a different length. Translation is exactly this case: an English sentence and its French version rarely have the same number of words, and word order differs between the two languages.^[1]

The method

The approach is simple in outline. One multilayer LSTM reads the input sentence into a single fixed-size vector. A second deep LSTM then generates the output sentence from that vector. The design makes minimal assumptions about the structure of the sequences.^[2] For instance, an English sentence such as "I love you" would be read into one vector, from which the decoder writes the French words one at a time.

The model

The network used LSTMs with 4 layers of 1,000 cells each and 1,000-dimensional word embeddings. Its input vocabulary had 160,000 words and its output vocabulary 80,000, for 384 million parameters in total.^[1]

Training

The authors trained on a 12-million-sentence subset of the WMT'14 English-to-French dataset, which contains 348 million French and 304 million English words. Training took about ten days on an 8-GPU machine. Each LSTM layer ran on its own GPU, while the other 4 GPUs computed the softmax.^[2]

Results

An ensemble of five deep LSTMs scored 34.81 BLEU, a standard score that compares machine translations with reference translations, where higher is better. A phrase-based statistical baseline scored 33.30. This was the first time a pure neural translation system beat a phrase-based baseline on a large-scale task. Using the LSTM to rerank the baseline's 1,000 best candidate translations gave 36.5, close to the best published result of 37.0.^[2]

A simple trick also helped. Reversing the word order of the source sentences, but not the targets, cut test perplexity from 5.8 to 4.7 and raised a single model's BLEU from 25.9 to 30.6. The authors attribute this to the many short-range dependencies that reversal creates between source and target words.^[2] The model also handled long sentences well, and it learned sentence representations sensitive to word order.^[2]

Why it mattered

The paper showed that a single neural network trained end to end could compete on a large translation benchmark. The encoder–decoder architecture it set out became a basis for many later translation models. The NeurIPS 2024 Test of Time Award, shared with the GAN paper, noted more than 27,000 citations and called the paper a cornerstone that set the encoder–decoder architecture. The announcement said this inspired the attention-based work that led to today's foundation models.^[3]