Home›Papers
Papers
Papers
Papers that changed AI, with a plain-language explainer for each.
Total 95 · Clear filters
- 2019
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5)
JMLR
A 2019 Google paper that frames every language task as text in, text out, using one encoder-decoder Transformer and one training setup. It also introduced the C4 dataset, and the released T5 models became common research starting points.
- 2019
RoBERTa: A Robustly Optimized BERT Pretraining Approach
arXiv
A replication study of BERT pretraining that measures how much key hyperparameters and training data size matter, finds BERT was significantly undertrained, and shows a properly trained BERT matches or beats later models, reaching state of the art on GLUE, RACE and SQuAD.
- 2019
XLNet: Generalized Autoregressive Pretraining for Language Understanding
arXiv
XLNet is a generalized autoregressive pretraining method from a 2019 paper. It learns bidirectional contexts by maximizing expected likelihood over all factorization-order permutations and outperforms BERT on 20 tasks under comparable settings.
- 2019
HellaSwag: Can a Machine Really Finish Your Sentence?
arXiv
HellaSwag is a 2019 benchmark in which a machine must pick the most plausible continuation of a short event description. Its wrong answers are generated and filtered to fool models rather than people.
- 2019
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
arXiv
SuperGLUE is a benchmark for general-purpose language understanding, released in 2019 as a harder successor to GLUE. It bundles eight tasks, a single-number metric, an analysis toolkit and a public leaderboard.
- 2019
Language Models are Unsupervised Multitask Learners (GPT-2)
OpenAI
OpenAI's 2019 paper on GPT-2, a 1.5-billion-parameter Transformer trained on web text. It showed that a model trained only to predict the next word could perform several tasks when prompted, and its staged release sparked public debate.
- 2018
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
NAACL
Google's 2018 BERT paper introduced an encoder-only Transformer that reads text in both directions and is pre-trained by hiding words. It set new results on eleven language tasks and made download-and-fine-tune the default workflow.
- 2018
Improving Language Understanding by Generative Pre-Training (GPT)
OpenAI
OpenAI's 2018 paper introducing GPT, a Transformer decoder pre-trained to predict the next token and then fine-tuned for specific tasks. It improved the state of the art on 9 of 12 tasks studied and set the recipe the GPT series followed.
- 2018
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
arXiv
GLUE is a benchmark and analysis platform that evaluates natural language understanding models across a diverse set of existing tasks, with an online comparison platform and a hand-crafted diagnostic suite for linguistic analysis.
- 2017
Deep reinforcement learning from human preferences
NeurIPS
A 2017 paper showing that an AI agent can learn a task from short human comparisons instead of a hand-written reward. The method, reinforcement learning from human feedback, later became central to training chat assistants.
- 2017
Attention Is All You Need
NeurIPS
A 2017 paper that introduced the Transformer, a neural network built entirely on attention with no recurrence. It set new machine translation results and became the architecture behind almost every large language model since.
- 2016
SQuAD: 100,000+ Questions for Machine Comprehension of Text
arXiv
SQuAD is a reading-comprehension dataset of more than 100,000 crowdworker questions on Wikipedia articles, each answered by a span of its passage. Released in 2016, it gave machine comprehension a large, freely available benchmark with a measured human ceiling.
- 2014
GloVe: Global Vectors for Word Representation
EMNLP 2014
GloVe is a 2014 method that learns word vectors by factorising global word co-occurrence counts. Released together with pre-trained vectors, it became, alongside word2vec, one of the two standard ways of producing static word embeddings.
- 2014
Sequence to Sequence Learning with Neural Networks
NeurIPS
Ilya Sutskever, Oriol Vinyals and Quoc Le's 2014 paper showed that two LSTMs, one reading the input and one writing the output, could translate sentences and outperform a phrase-based statistical baseline. It set the encoder–decoder pattern behind later translation models.
- 2014
Neural Machine Translation by Jointly Learning to Align and Translate
ICLR
Submitted to arXiv in September 2014 by Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio, this paper let a translation model look back at the whole source sentence while producing each word. This mechanism, called attention, later became a basis for Transformer models.
- 2014
Microsoft COCO: Common Objects in Context
arXiv
A 2014 paper introducing Microsoft COCO, a large dataset of everyday scenes in which common objects are labeled with per-instance segmentations, built to study object recognition as part of scene understanding.
- 2013
Efficient Estimation of Word Representations in Vector Space
arXiv
Submitted to arXiv in January 2013 by Tomas Mikolov and colleagues at Google, this paper introduced CBOW and Skip-gram, two simple architectures that learn word vectors quickly from very large text collections. They are the basis of the word2vec project.
- 2011
Deep Sparse Rectifier Neural Networks
A 2011 paper by Glorot, Bordes and Bengio reporting that deep networks built from rectifying neurons match or beat tanh networks and reach their best performance without unsupervised pre-training on large labelled supervised tasks.
- 2011
Torch7: A Matlab-like Environment for Machine Learning
A 2011 workshop paper by Ronan Collobert, Koray Kavukcuoglu and Clement Farabet describing Torch7, a Lua-based numeric computing framework and machine learning library built for flexible experimentation on fast low-level kernels.
- 2010
Understanding the difficulty of training deep feedforward neural networks
AISTATS / PMLR
A 2010 paper by Xavier Glorot and Yoshua Bengio on why deep feedforward networks are hard to train: sigmoid saturation under random initialization and layer Jacobians whose singular values sit far from 1, plus a new initialization scheme that converges substantially faster.