Transformer

What it is

The Transformer is a neural network architecture, a blueprint for how the parts of a model are wired together. It was introduced in the June 2017 paper Attention Is All You Need, written by eight researchers at Google Brain and Google Research.^[1] Its central change was to drop recurrence (the RNN and LSTM designs) and convolution entirely, and to rely on self-attention alone.^[1]

How it works

A model first splits text into tokens and turns each token into a list of numbers called an embedding. Self-attention then lets every token look at every other token in the sequence and decide how much each one matters for its own meaning. Because attention by itself has no sense of word order, the model adds positional encodings to the embeddings so it knows where each token sits. The original paper used sinusoidal encodings.^[1]

A small example makes this concrete. Take the sentence: the cat sat on the mat because it was tired. To work out what "it" refers to, a well-trained attention layer can give the word "cat" a high weight when building the representation of "it", and a lower weight to "mat". This is an illustration of the idea, not a measurement from any particular model.

The original design has an encoder and a decoder, each a stack of 6 identical layers. Every layer contains multi-head self-attention and a position-wise feed-forward network, joined by residual connections and layer normalization. Multi-head attention runs 8 attention operations in parallel in the base model, so the model can capture different kinds of relationships at once. During training, all positions in a sequence are processed in parallel.^[1]

Where it came from

Before the Transformer, a common design for language models was the recurrent network, which reads text one step at a time. The paper proposed building the whole model from attention alone, with no recurrence or convolution.^[1]

The architecture then branched into several families. BERT, from 2018, uses only the encoder. The GPT series, starting in 2018, uses only the decoder. T5, from 2019, keeps the full encoder-decoder design.^[2]

Where it is used today

The architecture has also spread beyond text. The Vision Transformer, from 2020, applies it to images.^[3] The same design has also been used for audio and robotics.^[2]

Limits

The Transformer is a set of design choices rather than a law of nature. It has no built-in sense of order, so it depends on the positional encodings added to it. Its success on text does not show that it is the best fit for every kind of data. Its spread into images, audio and robotics shows that the design can be reused, not that it performs equally well in every field. Understanding a Transformer means knowing which parts are fixed and which can be changed.