Home›Papers
Papers
Papers
Papers that changed AI, with a plain-language explainer for each.
Total 112 · Clear filters
- 2020
Denoising Diffusion Probabilistic Models
NeurIPS
A 2020 paper by Jonathan Ho, Ajay Jain and Pieter Abbeel that trained a U-Net to reverse a gradual noising process, making diffusion models a practical rival to GANs and paving the way for DALL-E 2, Imagen and Stable Diffusion.
- 2020
ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
arXiv
A 2020 paper by Clark, Luong, Le and Manning that pre-trains text encoders by detecting replaced tokens instead of filling in masked ones, matching much larger baselines with far less compute.
- 2020
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
arXiv
A 2020 paper by Mildenhall, Srinivasan, Tancik, Barron, Ramamoorthi and Ng that represents a scene as a neural radiance field: a network mapping 5D coordinates to density and colour, optimized from a sparse set of images with known camera poses.
- 2019
Mastering Atari, Go, chess and shogi by planning with a learned model
Nature
MuZero is a DeepMind reinforcement-learning algorithm that learns a model of its environment and plans inside that model, mastering Atari, Go, chess and shogi without being given the rules of those games.
- 2019
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5)
JMLR
A 2019 Google paper that frames every language task as text in, text out, using one encoder-decoder Transformer and one training setup. It also introduced the C4 dataset, and the released T5 models became common research starting points.
- 2019
RoBERTa: A Robustly Optimized BERT Pretraining Approach
arXiv
A replication study of BERT pretraining that measures how much key hyperparameters and training data size matter, finds BERT was significantly undertrained, and shows a properly trained BERT matches or beats later models, reaching state of the art on GLUE, RACE and SQuAD.
- 2019
XLNet: Generalized Autoregressive Pretraining for Language Understanding
arXiv
XLNet is a generalized autoregressive pretraining method from a 2019 paper. It learns bidirectional contexts by maximizing expected likelihood over all factorization-order permutations and outperforms BERT on 20 tasks under comparable settings.
- 2018
A Style-Based Generator Architecture for Generative Adversarial Networks
arXiv
This 2018 paper introduces a style-based GAN generator that separates high-level attributes from stochastic variation and allows scale-specific control, improving distribution quality, interpolation, and disentanglement.
- 2018
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
NAACL
Google's 2018 BERT paper introduced an encoder-only Transformer that reads text in both directions and is pre-trained by hiding words. It set new results on eleven language tasks and made download-and-fine-tune the default workflow.
- 2018
Large Scale GAN Training for High Fidelity Natural Image Synthesis
arXiv
A 2018 paper by Andrew Brock, Jeff Donahue and Karen Simonyan that trained generative adversarial networks at the largest scale attempted at the time, introduced BigGANs and set a new state of the art in class-conditional image synthesis.
- 2018
Glow: Generative Flow with Invertible 1x1 Convolutions
arXiv
A 2018 paper by Diederik P. Kingma and Prafulla Dhariwal presenting Glow, a flow-based generative model built on invertible 1x1 convolutions, which improved log-likelihood on standard benchmarks and could synthesise and manipulate large images.
- 2018
Neural Ordinary Differential Equations
arXiv
A 2018 paper by Chen, Rubanova, Bettencourt and Duvenaud that replaces a fixed sequence of hidden layers with a neural network describing the derivative of the hidden state, so outputs are computed by an ODE solver — yielding continuous-depth models with constant memory cost.[^1]
- 2018
Improving Language Understanding by Generative Pre-Training (GPT)
OpenAI
OpenAI's 2018 paper introducing GPT, a Transformer decoder pre-trained to predict the next token and then fine-tuned for specific tasks. It improved the state of the art on 9 of 12 tasks studied and set the recipe the GPT series followed.
- 2018
Group Normalization
arXiv
Group Normalization is a 2018 paper by Yuxin Wu and Kaiming He that proposes normalizing channels in groups, independent of batch size, to address batch normalization's weakness when batches are small.
- 2018
Deep contextualized word representations
arXiv
A 2018 paper by Matthew E. Peters and five co-authors that represents each word with the internal states of a pre-trained deep bidirectional language model, so vectors change with context; adding them to existing models improved results on six NLP tasks.
- 2018
Universal Language Model Fine-tuning for Text Classification
arXiv
ULMFiT is a 2018 transfer-learning method by Jeremy Howard and Sebastian Ruder for fine-tuning language models across NLP tasks; it beat the state of the art on six text-classification tasks and performed well with few labels.
- 2017
A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play
Science
A 2017 arXiv paper, later published in Science, reporting that a single self-play reinforcement learning algorithm, AlphaZero, mastered chess, shogi and Go and beat strong specialised programs in each.
- 2017
Neural Discrete Representation Learning
arXiv
A 2017 paper by van den Oord, Vinyals and Kavukcuoglu proposing VQ-VAE, a generative model whose latent code is discrete rather than continuous, and whose prior is learned rather than fixed, so that useful representations can be learned without supervision.
- 2017
Progressive Growing of GANs for Improved Quality, Stability, and Variation
arXiv
A 2017 paper by Karras, Aila, Laine and Lehtinen that trains generative adversarial networks by growing the generator and discriminator from low to high resolution, improving stability, quality and variation.
- 2017
mixup: Beyond Empirical Risk Minimization
arXiv
mixup is a training principle from a 2017 paper by Zhang, Cisse, Dauphin and Lopez-Paz that trains networks on convex combinations of pairs of examples and their labels, improving generalisation on image, speech and tabular benchmarks.