“Attention Is All You Need” is posted to arXiv, introducing the Transformer

Background

Before 2017, the standard way to model language and other sequences, such as sentences being translated, was the recurrent neural network. A recurrent network reads its input one token at a time and carries a running memory forward. That step-by-step design makes it hard to process a whole sequence in parallel on modern hardware. Attention, a way for a network to look back at earlier positions when it produces an output, was already used alongside these models, but only as an add-on rather than the core of the design.

The paper has eight authors, marked as equal contributors with the listing order randomised: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser and Illia Polosukhin.^[1] In the first version, Vaswani, Shazeer and Kaiser were at Google Brain, Parmar, Uszkoreit and Jones at Google Research, and Gomez at the University of Toronto, with his work done while he was at Google Brain.^[2]

What happened

On 12 June 2017 the paper was first submitted to arXiv, the open preprint server, under the identifier arXiv:1706.03762.^[1] It proposed the Transformer, a sequence-transduction architecture, meaning a model that turns one sequence into another, such as a sentence in one language into a sentence in another. The Transformer was built only on attention mechanisms, with no recurrence and no convolutions. Because the computation for all positions can run together, training was more parallelizable and faster than with recurrent designs.^[1]

Key details

The main experiments were on machine translation. On the WMT 2014 English-to-German task the Transformer scored 28.4 BLEU, more than 2 BLEU above the previous best results, including ensembles, which combine several models.^[2] BLEU is a standard score that compares a machine translation with human reference translations; higher is better.

On English-to-French it set a new single-model state of the art, with 41.0 BLEU in the first version and 41.8 in the revised version.^[2][1] Training took 3.5 days on eight GPUs, a small fraction of the training cost of the best earlier models.^[2]

The authors also applied the Transformer to English constituency parsing, which analyses the grammatical structure of sentences. They reported good results with both large and limited training data.^[1]

The paper was published at NIPS 2017, in Advances in Neural Information Processing Systems 30.^[3]

Impact

The main consequence was architectural. Removing recurrence let the computation be spread across many processors at once, which made it practical to train on far larger amounts of text than recurrent models could handle in reasonable time. Most large language models built since then use this design or a close variant, and the same architecture has been applied to images, audio and other kinds of data. The paper's own experiments covered translation and parsing; its wider reach grew over the years that followed.