Home›Papers
Papers
Papers
Papers that changed AI, with a plain-language explainer for each.
Total 18 · Clear filters
- 2024
Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
ICML
Tri Dao and Albert Gu's paper introduces state space duality, a framework showing that structured state space models and attention variants are closely related, and uses it to derive Mamba-2, whose core layer runs 2 to 8 times faster than Mamba's.
- 2022
Scalable Diffusion Models with Transformers (DiT)
arXiv
A 2022 paper by William Peebles and Saining Xie that replaces the U-Net backbone of diffusion models with a transformer operating on latent patches, showing that image quality improves predictably as compute grows and reaching a state-of-the-art FID of 2.27 on ImageNet 256×256.
- 2020
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (ViT)
arXiv
A 2020 paper showing that a plain Transformer applied directly to sequences of image patches, with no convolution at all, can match state-of-the-art convolutional networks on image classification.
- 2020
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
arXiv 2010.11929 (ICLR 2021)
A 2020 paper by Alexey Dosovitskiy and colleagues showing that a pure Transformer applied directly to sequences of image patches — the Vision Transformer, or ViT — matches state-of-the-art convolutional networks on image classification when pre-trained on large amounts of data.
- 2019
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5)
JMLR
A 2019 Google paper that frames every language task as text in, text out, using one encoder-decoder Transformer and one training setup. It also introduced the C4 dataset, and the released T5 models became common research starting points.
- 2018
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
NAACL
Google's 2018 BERT paper introduced an encoder-only Transformer that reads text in both directions and is pre-trained by hiding words. It set new results on eleven language tasks and made download-and-fine-tune the default workflow.
- 2018
Improving Language Understanding by Generative Pre-Training (GPT)
OpenAI
OpenAI's 2018 paper introducing GPT, a Transformer decoder pre-trained to predict the next token and then fine-tuned for specific tasks. It improved the state of the art on 9 of 12 tasks studied and set the recipe the GPT series followed.
- 2017
Attention Is All You Need
NeurIPS
A 2017 paper that introduced the Transformer, a neural network built entirely on attention with no recurrence. It set new machine translation results and became the architecture behind almost every large language model since.
- 2016
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
arXiv
A 2016 paper describing Google's neural machine translation system, GNMT: a deep LSTM encoder-decoder with attention, designed to make end-to-end translation practical by addressing rare words, parallelism and inference cost.
- 2014
Neural Machine Translation by Jointly Learning to Align and Translate
ICLR
Submitted to arXiv in September 2014 by Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio, this paper let a translation model look back at the whole source sentence while producing each word. This mechanism, called attention, later became a basis for Transformer models.
- 1969
Perceptrons: An Introduction to Computational Geometry
A 1969 book by Marvin Minsky and Seymour Papert that proved mathematical limits on what single-layer perceptrons can compute — notably that they cannot compute parity or connectedness — and has long been credited with steering research away from neural networks.
- 1966
ELIZA - A Computer Program for the Study of Natural Language Communication Between Man and Machine
Communications of the ACM 9(1)
A January 1966 paper by Joseph Weizenbaum presenting ELIZA, a program on MIT's MAC time-sharing system that used keyword-triggered decomposition and reassembly rules to enable certain natural-language conversations between people and computers.
- 1961
Steps Toward Artificial Intelligence
Proceedings of the IRE 49(1)
Marvin Minsky's 1961 survey in Proceedings of the IRE organized heuristic programming into five problem areas — search, pattern recognition, learning, planning and induction — and argued that machines would need to build models of their environments.
- 1959
Some Studies in Machine Learning Using the Game of Checkers
IJRD
A. L. Samuel's 1959 paper reporting experiments in which a checkers program, given only the rules of the game and an imperfect set of parameters, learned in 8 to 10 hours of playing time to beat the person who wrote it.
- 1959
Programs with Common Sense
PSMTPNPL
John McCarthy's 1959 paper proposing the advice taker, a program that would solve problems by manipulating sentences in formal languages and act when it deduced an imperative — an early statement of logical AI.
- 1950
Computing Machinery and Intelligence
Mind
In 1950 Alan Turing replaced the question "Can machines think?" with the imitation game, now called the Turing test, and answered the main objections to machine intelligence one by one.
- 1943
A Logical Calculus of the Ideas Immanent in Nervous Activity
Bull. Math. Biophys.
In 1943 Warren McCulloch and Walter Pitts described a simplified model of a neuron and showed that networks of such units can carry out logical reasoning, an early link between brain models and computation.
- 1936
On Computable Numbers, with an Application to the Entscheidungsproblem
Proc. London Math. Soc.
Alan Turing's 1936 paper defined computable numbers through an abstract machine, now called the Turing machine, and showed that no general method can decide whether any given mathematical statement is provable.