Transformers & attention

Attention scores how important each element of a sequence is relative to the others and turns those scores into weights, letting a model read any part of its input directly. It is the operation at the core of the transformer.

Attention lets a neural network weigh the parts of its input by relevance. Self-attention applies this idea inside a single sequence, so each token gathers information from every other token. It is the core operation of the Transformer.

Submitted to arXiv in September 2014 by Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio, this paper let a translation model look back at the whole source sentence while producing each word. This mechanism, called attention, later became a basis for Transformer models.

A 2017 paper that introduced the Transformer, a neural network built entirely on attention with no recurrence. It set new machine translation results and became the architecture behind almost every large language model since.

The Transformer is a neural network architecture introduced in 2017 that relies on self-attention instead of recurrence or convolution. Most of today's large language models are built on it.

OpenAI's 2018 paper introducing GPT, a Transformer decoder pre-trained to predict the next token and then fine-tuned for specific tasks. It improved the state of the art on 9 of 12 tasks studied and set the recipe the GPT series followed.

BERT is a language model from Google that reads text in both directions and is pre-trained by filling in masked words. Fine-tuned per task, it set new records on question answering and language-understanding benchmarks and later went into Google Search.

Google's 2018 BERT paper introduced an encoder-only Transformer that reads text in both directions and is pre-trained by hiding words. It set new results on eleven language tasks and made download-and-fine-tune the default workflow.

GPT-3 is an autoregressive language model with 175 billion parameters, described in a May 2020 paper. It could perform new tasks from a few examples in the prompt, without any gradient updates, and it was first offered through OpenAI's API.

A 2020 paper showing that a plain Transformer applied directly to sequences of image patches, with no convolution at all, can match state-of-the-art convolutional networks on image classification.

The NVIDIA H100 is the first data-center GPU built on NVIDIA's Hopper architecture, announced in March 2022 with 80 billion transistors, HBM3 memory, FP8 Tensor Cores and a Transformer Engine aimed at large language models.

RT-1 is a transformer-based model for real-world robot control, trained on over 130,000 episodes collected with 13 robots and reported to perform more than 700 language instructions at a 97% success rate.

Llama 3.1, released on 23 July 2024, added a 405-billion-parameter model to the Llama 3 family and extended its context to 128,000 tokens. Meta called the 405B the largest and most capable openly available foundation model.

A multimodal model handles more than one kind of input or output, such as images together with text, inside a single model. Work on CLIP and Flamingo showed how such models are trained and how they can be steered toward new tasks.

Tokenization splits text into the tokens a language model actually reads; most modern systems do it with byte pair encoding, a 1994 compression method adapted for translation in 2015.

Google's research division, based in Mountain View and usually dated to 2000, produced the Transformer, BERT and T5 — the work underlying modern large language models.

Google Brain was a deep-learning research team started inside Google's X in 2011. Its cat-recognition experiment, TensorFlow, sequence-to-sequence models and the Transformer made it central to modern AI; it merged into Google DeepMind in 2023.

The Tesla P100, announced by NVIDIA in April 2016, was the first data-center GPU on the Pascal architecture, and the first to combine stacked HBM2 memory, half-precision math aimed at deep learning and the NVLink interconnect.

A 2016 paper describing Google's neural machine translation system, GNMT: a deep LSTM encoder-decoder with attention, designed to make end-to-end translation practical by addressing rare words, parallelism and inference cost.

A component of sequence models, especially transformers, that encodes each token's position as a vector, giving order-blind attention a sense of sequence; its forms range from the original transformer's sinusoids to RoPE and ALiBi.

GPT-1 was OpenAI's first generative pre-trained transformer, published in June 2018. It showed that one language model, pre-trained on unlabelled text and then fine-tuned, could improve results on many language tasks.

GPT-2 is a large language model that OpenAI announced in February 2019. Its largest version has 1.5 billion parameters. OpenAI first withheld the full model, then released it in stages through November 2019.

Baidu's first ERNIE pre-trained language model, released in March 2019, added entity- and phrase-level masking to BERT-style pre-training and reported state-of-the-art results on five Chinese NLP tasks.

T5 is a family of Google Transformer models that recasts every language task as text in, text out. Its 11-billion-parameter version set state of the art on several benchmarks, and the code, models and C4 dataset were released openly.

A 2020 paper by Alexey Dosovitskiy and colleagues showing that a pure Transformer applied directly to sequences of image patches — the Vision Transformer, or ViT — matches state-of-the-art convolutional networks on image classification when pre-trained on large amounts of data.

DALL·E is OpenAI's text-to-image model unveiled on 5 January 2021: a 12-billion-parameter transformer trained on 250 million text-image pairs that generates a picture token by token from a caption.

An image-text model from OpenAI that learns to match pictures with their captions. Trained on about 400 million image-caption pairs, it can classify images by comparing them against written prompts, with no task-specific training.

LaMDA is a family of conversational language models that Google announced in 2021 and described in a 2022 paper, notable for its work on safety and factual grounding in dialogue and for later powering Bard.

ERNIE 3.0 is a 10-billion-parameter pre-trained language model from 2021 that trains on plain text plus a large-scale knowledge graph and fuses auto-encoding and auto-regressive networks, so one model handles both language understanding and generation.

PaLM is a 540-billion-parameter dense Transformer language model announced by Google Research in April 2022, trained with the Pathways system across 6,144 TPU v4 chips; it set strong few-shot results on many benchmarks.

A 2022 paper by William Peebles and Saining Xie that replaces the U-Net backbone of diffusion models with a transformer operating on latent patches, showing that image quality improves predictably as compute grows and reaching a state-of-the-art FID of 2.27 on ImageNet 256×256.

PaLM 2 is Google's second-generation Pathways language model, announced on 10 May 2023 in four sizes and used across more than 25 Google products, including Bard.

Mistral AI's first model: a 7.3-billion-parameter language model released in September 2023 under the Apache 2.0 license, built to match much larger models on standard benchmarks.

Grok-1 is xAI's first large language model, announced on 3 November 2023 as the engine of the Grok chatbot. Its 314-billion-parameter weights were released openly in March 2024.

Gemma is a family of lightweight open language models from Google DeepMind, released in February 2024 in 2B and 7B sizes, built from the same research and technology as Gemini and offered for commercial use under Google's terms.

Stable Diffusion 3, announced by Stability AI on 22 February 2024, is a suite of 800 million to 8 billion parameter models that pairs a diffusion transformer with flow matching.

Llama 3 is an open model family from Meta, released on 18 April 2024 at 8 billion and 70 billion parameters. It was pretrained on over 15 trillion tokens of publicly available data.

Tri Dao and Albert Gu's paper introduces state space duality, a framework showing that structured state space models and attention variants are closely related, and uses it to derive Mamba-2, whose core layer runs 2 to 8 times faster than Mamba's.

OpenVLA is a 7-billion-parameter open-source vision-language-action model trained on 970,000 real robot demonstrations, released with checkpoints and code so that other groups can fine-tune and run it on their own robots.

Falcon-H1 is a family of open-weight language models that runs Transformer attention and state space layers in parallel, released in sizes from 0.5B to 34B parameters under the Falcon-LLM Licence.

A KV cache keeps the key and value vectors a model has already computed for earlier tokens, so each new token reuses them instead of recomputing. Its memory footprint is what makes high-throughput serving hard to scale.

Long context is a model's capacity to take in and work over very large inputs — documents, images, recordings — in a single request. Systems now reach millions of tokens, though finding information inside that window is a separate skill.

Noam Shazeer is an AI researcher who co-wrote the Transformer paper and was first author of the paper on sparsely gated mixture-of-experts layers. He co-founded Character.AI, returned to Google in 2024, and announced a move to OpenAI in 2026.

A context window is the full span of text a model can refer to while producing a response, measured in tokens and including the response itself. Its size, and how evenly a model uses it, shape what the model can do.

Mechanistic interpretability explains model behaviour by finding the internal computations and features that produce it. Work on superposition and sparse autoencoders shows how many features can be packed into few neurons.

Llion Jones is one of the eight authors of the 2017 paper "Attention Is All You Need" and a co-founder and CTO of Sakana AI.

Transformers is an open-source Python library that offers pretrained Transformer models through one unified API. Its paper was posted to arXiv on 9 October 2019, and the first v5 release candidate came in December 2025.

Aidan Gomez is a Canadian computer scientist, co-founder and CEO of Cohere, and one of the eight authors of the 2017 paper that introduced the Transformer architecture.

Ashish Vaswani is a computer scientist best known as the first-listed author of “Attention Is All You Need,” the 2017 paper that introduced the Transformer architecture. He also co-founded the AI companies Adept and Essential AI.

Niki Parmar is one of the eight authors of "Attention Is All You Need", the 2017 paper that introduced the Transformer architecture, and is named in Google's own account of that work.

Jakob Uszkoreit is one of the eight authors of “Attention Is All You Need” and the founder of Inceptive, a company using AI to design biological molecules.[^1][^4]

Illia Polosukhin is one of the eight authors of the 2017 Transformer paper and a co-founder of NEAR Protocol. A 2024 NEAR post identifies him as CEO of the NEAR Foundation.[^1][^3][^6]

ERNIE 2.0 is Baidu's pre-trained language model framework, published in 2019, which builds pre-training tasks incrementally through continual multi-task learning and reported results above BERT and XLNet on English and Chinese benchmarks.

A 2019 Google paper that frames every language task as text in, text out, using one encoder-decoder Transformer and one training setup. It also introduced the C4 dataset, and the released T5 models became common research starting points.

Meena is a 2.6-billion-parameter open-domain chatbot from Google, trained end-to-end on filtered social media conversations. Its 2020 paper introduced the SSA human-evaluation metric and linked next-token perplexity to human-judged chat quality. The work was later renamed LaMDA.

Megatron-Turing NLG 530B is a 530-billion-parameter language model announced by NVIDIA and Microsoft on 11 October 2021, then the largest monolithic transformer language model, trained on NVIDIA's Selene supercomputer with Megatron-LM and DeepSpeed combined.

Gopher is DeepMind's 280-billion-parameter language model, announced in December 2021 alongside a study of what scale does to language models and a companion taxonomy of their risks.

AlphaCode is a DeepMind code-generation system for competitive programming, announced in February 2022. On Codeforces contests held after its training data, it reached an estimated rank around the median participant. A Gemini-based AlphaCode 2 followed in December 2023.

OPT-175B is a 175-billion-parameter language model from Meta AI, announced in May 2022. Meta said it was the first model of its size released with both pretrained weights and the code to train and use them.

Speculative decoding speeds up text generation from large autoregressive models: a smaller model proposes several tokens and the large model checks them in parallel, producing identical output faster.

Stable LM is Stability AI's family of open language models, first released in April 2023 as 3B- and 7B-parameter alpha versions under licenses that allow commercial use of the base models and research-only use of the instruction-tuned ones.

RT-2 is a vision-language-action model from Google DeepMind, announced on 28 July 2023, that turns images and text instructions into robot actions written as text tokens. In over 6,000 trials it reached 62% on unseen scenarios, up from 32% for RT-1.

A family of text-to-image models from Stability AI, announced in October 2024: the 8.1-billion-parameter Large, a 4-step Turbo variant and the 2.5-billion-parameter Medium, released under a community licence.

Thomas Wolf is a co-founder and chief science officer of Hugging Face, and the first author of the 2019 paper that introduced the company's open-source Transformers library to researchers and practitioners.

Jacob Devlin is a research scientist and first author of BERT, a language model Google introduced in 2018 that reads text in both directions. He earlier worked on neural machine translation at Microsoft Research.