Positional encoding

What it is

Positional encoding is a component of sequence models, especially transformers, that encodes the position of each token as a vector ^[1]. It is needed because attention on its own has no notion of order: standard attention is symmetric with respect to position and permutation invariant, so without positional information the input is treated as a bag of words rather than a sequence ^[1]. Two copies of the same sentence with the words shuffled would look identical to such a model.

How it works

The best-known scheme is sinusoidal positional encoding, which builds each position's vector out of cosine and sine functions whose frequencies are spaced geometrically; in the original paper d is 256 and the base b is 10,000 ^[1]. A useful intuition is that each dimension of the vector is a wave running along the sequence, and the wavelength grows from one dimension to the next: low dimensions oscillate quickly and separate neighbouring positions, while high dimensions change slowly and mark out broad regions of the sequence.

A small example makes the point. Suppose the same word appears early in a sentence and again much later. Its token vector is the same both times, but the vector added to it differs, so the layers above can tell the two occurrences apart. Swap two words in a sentence and the token vectors are untouched; only the position vectors move, which is exactly the information attention would otherwise lack.

The original transformer also tested learned absolute position embeddings, an approach that comes from Gehring and colleagues in 2017 ^[1]. In that variant each position has its own trainable vector, looked up by index, so the model learns the encoding rather than computing it from a formula.

Where it came from

Vaswani and colleagues introduced sinusoidal positional encoding in 2017 in Attention Is All You Need ^[1]. The paper, first submitted in June 2017, proposed an architecture based solely on attention mechanisms and dispensing with recurrence and convolutions entirely ^[2]. It reported 28.4 BLEU on WMT 2014 English-to-German, more than 2 BLEU above the existing best results including ensembles ^[2], and on WMT 2014 English-to-French it set a single-model state of the art of 41.8 BLEU after training for 3.5 days on eight GPUs ^[2]. The authors hoped the model could extrapolate to sequences longer than those seen in training ^[1], which is one motivation for using fixed formulas rather than a learned table.

Where it is used today

GPT adopted learned absolute position embeddings in 2018, and BERT in 2018 used learned embeddings for both token position and segment ^[1]. Later work moved toward relative schemes. Rotary position embedding (RoPE), proposed by Su and colleagues in 2021, encodes relative position by rotating the query and key vectors; it was used in GPT-NeoX in 2022 and in Llama with b equal to 10,000 ^[1]. ALiBi, by Press, Smith and Lewis in 2021, takes a different route and adds a linear bias proportional to the negative distance between positions to the attention logits ^[1].

Limitations

Learned absolute embeddings cannot handle context longer than the training length ^[1]. Long contexts cause drift in the positional encoding distribution and in attention weights, which is why extension methods such as YaRN and dynamic scaling are used to train short and test long ^[1].