Home›Papers
Papers
Papers
Papers that changed AI, with a plain-language explainer for each.
Total 199
- 2018
Group Normalization
arXiv
Group Normalization is a 2018 paper by Yuxin Wu and Kaiming He that proposes normalizing channels in groups, independent of batch size, to address batch normalization's weakness when batches are small.
- 2018
Deep contextualized word representations
arXiv
A 2018 paper by Matthew E. Peters and five co-authors that represents each word with the internal states of a pre-trained deep bidirectional language model, so vectors change with context; adding them to existing models improved results on six NLP tasks.
- 2018
Universal Language Model Fine-tuning for Text Classification
arXiv
ULMFiT is a 2018 transfer-learning method by Jeremy Howard and Sebastian Ruder for fine-tuning language models across NLP tasks; it beat the state of the art on six text-classification tasks and performed well with few labels.
- 2017
A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play
Science
A 2017 arXiv paper, later published in Science, reporting that a single self-play reinforcement learning algorithm, AlphaZero, mastered chess, shogi and Go and beat strong specialised programs in each.
- 2017
Neural Discrete Representation Learning
arXiv
A 2017 paper by van den Oord, Vinyals and Kavukcuoglu proposing VQ-VAE, a generative model whose latent code is discrete rather than continuous, and whose prior is learned rather than fixed, so that useful representations can be learned without supervision.
- 2017
Progressive Growing of GANs for Improved Quality, Stability, and Variation
arXiv
A 2017 paper by Karras, Aila, Laine and Lehtinen that trains generative adversarial networks by growing the generator and discriminator from low to high resolution, improving stability, quality and variation.
- 2017
mixup: Beyond Empirical Risk Minimization
arXiv
mixup is a training principle from a 2017 paper by Zhang, Cisse, Dauphin and Lopez-Paz that trains networks on convex combinations of pairs of examples and their labels, improving generalisation on image, speech and tabular benchmarks.
- 2017
Mastering the game of Go without human knowledge
Nature
AlphaGo Zero learned Go only by playing against itself, starting from random play, and beat the earlier champion-defeating AlphaGo 100 games to 0 after three days of training. The Nature paper appeared in October 2017.
- 2017
Focal Loss for Dense Object Detection
arXiv
A 2017 paper by Lin, Goyal, Girshick, He and Dollar that argues dense one-stage detectors are held back by extreme foreground-background imbalance and proposes focal loss, a reshaped cross-entropy that down-weights well-classified examples.
- 2017
Proximal Policy Optimization Algorithms
arXiv
A 2017 paper by Schulman, Wolski, Dhariwal, Radford and Klimov that introduced proximal policy optimization (PPO), a reinforcement learning method that is much simpler to implement and tune and became OpenAI's default RL algorithm.
- 2017
Deep reinforcement learning from human preferences
NeurIPS
A 2017 paper showing that an AI agent can learn a task from short human comparisons instead of a hand-written reward. The method, reinforcement learning from human feedback, later became central to training chat assistants.
- 2017
Attention Is All You Need
NeurIPS
A 2017 paper that introduced the Transformer, a neural network built entirely on attention with no recurrence. It set new machine translation results and became the architecture behind almost every large language model since.
- 2017
Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks
arXiv
CycleGAN learns image-to-image translation between two domains without aligned pairs, combining an adversarial loss with a cycle-consistency constraint to constrain the mapping.
- 2017
Mask R-CNN
arXiv
Mask R-CNN is a 2017 framework for object instance segmentation. It detects objects and produces a high-quality mask for each instance by adding a parallel mask branch to Faster R-CNN.
- 2016
Neural Architecture Search with Reinforcement Learning
arXiv
A 2016 paper by Barret Zoph and Quoc V. Le that uses a recurrent network trained with reinforcement learning to design neural network architectures automatically, matching human-designed models on CIFAR-10 and beating LSTM on Penn Treebank.
- 2016
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
arXiv
A 2016 paper describing Google's neural machine translation system, GNMT: a deep LSTM encoder-decoder with attention, designed to make end-to-end translation practical by addressing rare words, parallelism and inference cost.
- 2016
WaveNet: A Generative Model for Raw Audio
arXiv
WaveNet is a deep neural network that generates raw audio waveforms one sample at a time, predicting each sample from all the ones before it. Used for text-to-speech, it was rated by listeners as more natural than the systems it was compared with.
- 2016
SQuAD: 100,000+ Questions for Machine Comprehension of Text
arXiv
SQuAD is a reading-comprehension dataset of more than 100,000 crowdworker questions on Wikipedia articles, each answered by a span of its passage. Released in 2016, it gave machine comprehension a large, freely available benchmark with a measured human ceiling.
- 2016
Identity Mappings in Deep Residual Networks
arXiv
A 2016 analysis of residual networks arguing that with identity skip connections and the activation placed after addition, signals travel directly between blocks; a new residual unit built on this eases training, and a 1001-layer network reaches 4.62% error on CIFAR-10.
- 2016
Mastering the game of Go with deep neural networks and tree search
Nature
The 2016 Nature paper describing AlphaGo, which combined deep neural networks with tree search. In October 2015 its distributed version beat the European Go champion Fan Hui 5-0, the first time a program defeated a professional player in full-sized Go without handicap.