Home›Papers
Papers
Papers
Papers that changed AI, with a plain-language explainer for each.
Total 12 · Clear filters
- 2024
Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
ICML
Tri Dao and Albert Gu's paper introduces state space duality, a framework showing that structured state space models and attention variants are closely related, and uses it to derive Mamba-2, whose core layer runs 2 to 8 times faster than Mamba's.
- 2022
Scalable Diffusion Models with Transformers (DiT)
arXiv
A 2022 paper by William Peebles and Saining Xie that replaces the U-Net backbone of diffusion models with a transformer operating on latent patches, showing that image quality improves predictably as compute grows and reaching a state-of-the-art FID of 2.27 on ImageNet 256×256.
- 2020
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (ViT)
arXiv
A 2020 paper showing that a plain Transformer applied directly to sequences of image patches, with no convolution at all, can match state-of-the-art convolutional networks on image classification.
- 2020
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
arXiv 2010.11929 (ICLR 2021)
A 2020 paper by Alexey Dosovitskiy and colleagues showing that a pure Transformer applied directly to sequences of image patches — the Vision Transformer, or ViT — matches state-of-the-art convolutional networks on image classification when pre-trained on large amounts of data.
- 2019
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5)
JMLR
A 2019 Google paper that frames every language task as text in, text out, using one encoder-decoder Transformer and one training setup. It also introduced the C4 dataset, and the released T5 models became common research starting points.
- 2018
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
NAACL
Google's 2018 BERT paper introduced an encoder-only Transformer that reads text in both directions and is pre-trained by hiding words. It set new results on eleven language tasks and made download-and-fine-tune the default workflow.
- 2018
Improving Language Understanding by Generative Pre-Training (GPT)
OpenAI
OpenAI's 2018 paper introducing GPT, a Transformer decoder pre-trained to predict the next token and then fine-tuned for specific tasks. It improved the state of the art on 9 of 12 tasks studied and set the recipe the GPT series followed.
- 2017
Attention Is All You Need
NeurIPS
A 2017 paper that introduced the Transformer, a neural network built entirely on attention with no recurrence. It set new machine translation results and became the architecture behind almost every large language model since.
- 2016
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
arXiv
A 2016 paper describing Google's neural machine translation system, GNMT: a deep LSTM encoder-decoder with attention, designed to make end-to-end translation practical by addressing rare words, parallelism and inference cost.
- 2014
Neural Machine Translation by Jointly Learning to Align and Translate
ICLR
Submitted to arXiv in September 2014 by Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio, this paper let a translation model look back at the whole source sentence while producing each word. This mechanism, called attention, later became a basis for Transformer models.
- 1999
The PageRank Citation Ranking: Bringing Order to the Web
Brin and Page's PageRank technical report treats hyperlinks as citations to rank web pages and was used to build Google's search engine; it is one of the heavily cited papers on their early work.
- 1998
The Anatomy of a Large-Scale Hypertextual Web Search Engine
A 1998 conference paper by Sergey Brin and Lawrence Page describing PageRank, an algorithm that ranks web pages by the number and quality of links pointing to them, and the search engine built on that algorithm.