Neural Machine Translation by Jointly Learning to Align and Translate

The paper

"Neural Machine Translation by Jointly Learning to Align and Translate" was submitted to arXiv on 1 September 2014 by Dzmitry Bahdanau of Jacobs University Bremen, and by Kyunghyun Cho and Yoshua Bengio of the Université de Montréal. It was accepted at ICLR 2015 as an oral presentation.^[1]

The problem

Encoder–decoder translation models squeezed the whole source sentence into one fixed-length vector, which the authors call a bottleneck. The performance of a basic encoder–decoder dropped quickly as sentences got longer.^[1] It is a bit like trying to translate a long paragraph after hearing only a one-line summary of it.

The method

The fix is attention. Each time the model generates a target word, it softly searches the source sentence for the positions with the most relevant information, and predicts the word from a context vector built specifically for that word.^[1]

The context vector is a weighted sum of "annotations" of all source words. The weights come from an alignment model that is trained jointly with the rest of the network. This is the attention mechanism.^[1] When the model produces the French word for "cat", for example, it can give the English word "cat" a large weight and the other words smaller weights.

The encoder is a bidirectional RNN, made of a forward and a backward RNN of 1,000 hidden units each. Each word's annotation therefore summarises both the words before it and the words after it.^[1]

Data and training

The models were trained on WMT'14 English–French parallel corpora totalling 850 million words, which data selection reduced to 348 million. Each model was trained for about 5 days.^[1]

Results

On the test set, measured in BLEU, the attention model RNNsearch-50 scored 26.75, against 17.82 for the plain encoder–decoder RNNencdec-50. A longer-trained RNNsearch-50 scored 28.45, and 36.15 on sentences without unknown words. For comparison, the phrase-based Moses system scored 35.63, and it also used an extra 418-million-word monolingual corpus.^[1]

The paper also reports that RNNsearch-50 showed no drop in performance even on sentences of 50 or more words, and that its learned soft alignments between French and English words were linguistically plausible.^[1]

Why it mattered

The model removed the single-vector bottleneck: the decoder can look back at any part of the source at any step, and the alignments it learns can be inspected. In 2025 the paper was runner-up for the ICLR Test of Time Award; the award went to the Adam paper. The organisers said the work fundamentally changed how sequence-to-sequence models process information, and that it became a basis for Transformers and large language models.^[2]

Because the alignments are soft weights rather than hard decisions, the model can show how strongly each source word contributes to each output word. The authors examined the learned French-English alignments in this way and found them linguistically plausible.^[1]