Attention Is All You Need

The problem

Recurrent networks, a common tool for language at the time, process words one after another, carrying a running memory forward from step to step. This design has two costs. Training is hard to spread across many processors, because each step waits for the one before it. And information from the start of a long sentence has to survive many steps before it can affect later words, which makes long-range dependencies hard to learn. ^[1]

The paper's answer was to ask whether attention alone could carry the whole job. Attention is a mechanism that lets a network weigh different parts of its input according to how relevant they are to the current position.

The approach

The paper proposed the Transformer, a model built entirely on attention, with no recurrence and no convolution. In self-attention, each word is turned into three vectors: a query describing what it is looking for, a key describing what it offers, and a value carrying its content. Each word compares its query with every key in the sentence, and its output is a weighted blend of the values. Because every word can reach every other word in a single step, the whole sentence can be processed together, which suits parallel hardware such as GPUs. ^[1]

The paper credits eight authors: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser and Illia Polosukhin. It notes that they contributed equally and that the author order was random. ^[1]

The paper was submitted to arXiv on 12 June 2017 and published at NeurIPS 2017. ^[1]

Results

On machine translation, the Transformer set new state-of-the-art results: 28.4 BLEU on the WMT 2014 English-to-German task and 41.8 BLEU on English-to-French. BLEU is a standard score that compares a machine translation with human reference translations, and higher is better. The large model trained in 3.5 days on eight GPUs, a fraction of the cost of earlier best models. The authors also tested English constituency parsing, a task that asks a system to recover the grammatical structure of a sentence, to show that the design was not limited to translation. ^[1]

A useful way to see the change is to follow how information travels. In a recurrent network, a word near the end of a long sentence can only hear from earlier words through a chain of intermediate steps. In the attention design, every word is directly connected to every other word, so the path between any two positions is always a single step. Short connections are easier to learn from, and the work for all positions can be carried out at the same time. ^[1]

Why it matters

Most large language models built since are based on this architecture, including OpenAI's GPT series, and it has become the main architecture of a wide variety of AI systems. ^[3] The paper is less a single model than a shared blueprint. Later systems have changed its details in many ways, but the core idea of building around attention has stayed central to the field.

What followed

Several of the eight authors later founded or co-founded AI companies, so the paper's influence reached well beyond academic research. ^[1] For a learner, it is a natural paper to read after understanding how neural networks are trained.