Home›Papers
Papers
Papers
Papers that changed AI, with a plain-language explainer for each.
Total 51 · Clear filters
- 2021
Evaluating Large Language Models Trained on Code (Codex / HumanEval)
arXiv
A 2021 paper from OpenAI introduces Codex, a GPT model fine-tuned on public GitHub code, and HumanEval, an evaluation set that judges generated Python programs by whether they work rather than by how closely they match a reference.
- 2021
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?
FAccT '21
A 2021 FAccT paper by Bender, Gebru, McMillan-Major and Shmitchell that asks whether language models can be too large and sets out recommendations on costs, data, research goals and alternative directions.
- 2020
The Pile: An 800GB Dataset of Diverse Text for Language Modeling
arXiv
The Pile is an 825 GiB English text dataset of 22 combined sources, released by EleutherAI on 31 December 2020 with an arXiv paper. It became the training data for several of the group's open models.
- 2020
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
arXiv
A 2020 paper by Mildenhall, Srinivasan, Tancik, Barron, Ramamoorthi and Ng that represents a scene as a neural radiance field: a network mapping 5D coordinates to density and colour, optimized from a sparse set of images with known camera poses.
- 2019
Mastering Atari, Go, chess and shogi by planning with a learned model
Nature
MuZero is a DeepMind reinforcement-learning algorithm that learns a model of its environment and plans inside that model, mastering Atari, Go, chess and shogi without being given the rules of those games.
- 2019
On the Measure of Intelligence (ARC-AGI)
arXiv
A 2019 paper by Francois Chollet that defines intelligence as skill-acquisition efficiency and introduces the Abstraction and Reasoning Corpus (ARC), a benchmark of grid tasks that assumes no specialised knowledge or language.
- 2019
C4: Colossal Clean Crawled Corpus (T5 paper)
JMLR
A 2019 paper that casts every language task as text-to-text, systematically compares transfer-learning choices, releases the C4 corpus and models, and reports state-of-the-art results on summarization, question answering, and classification.
- 2019
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5)
JMLR
A 2019 Google paper that frames every language task as text in, text out, using one encoder-decoder Transformer and one training setup. It also introduced the C4 dataset, and the released T5 models became common research starting points.
- 2019
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
arXiv
SuperGLUE is a benchmark for general-purpose language understanding, released in 2019 as a harder successor to GLUE. It bundles eight tasks, a single-number metric, an analysis toolkit and a public leaderboard.
- 2019
Language Models are Unsupervised Multitask Learners (GPT-2)
OpenAI
OpenAI's 2019 paper on GPT-2, a 1.5-billion-parameter Transformer trained on web text. It showed that a model trained only to predict the next word could perform several tasks when prompted, and its staged release sparked public debate.
- 2018
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
arXiv
GLUE is a benchmark and analysis platform that evaluates natural language understanding models across a diverse set of existing tasks, with an online comparison platform and a hand-crafted diagnostic suite for linguistic analysis.
- 2017
A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play
Science
A 2017 arXiv paper, later published in Science, reporting that a single self-play reinforcement learning algorithm, AlphaZero, mastered chess, shogi and Go and beat strong specialised programs in each.
- 2017
Deep reinforcement learning from human preferences
NeurIPS
A 2017 paper showing that an AI agent can learn a task from short human comparisons instead of a hand-written reward. The method, reinforcement learning from human feedback, later became central to training chat assistants.
- 2016
SQuAD: 100,000+ Questions for Machine Comprehension of Text
arXiv
SQuAD is a reading-comprehension dataset of more than 100,000 crowdworker questions on Wikipedia articles, each answered by a span of its passage. Released in 2016, it gave machine comprehension a large, freely available benchmark with a measured human ceiling.
- 2014
GloVe: Global Vectors for Word Representation
EMNLP 2014
GloVe is a 2014 method that learns word vectors by factorising global word co-occurrence counts. Released together with pre-trained vectors, it became, alongside word2vec, one of the two standard ways of producing static word embeddings.
- 2014
Microsoft COCO: Common Objects in Context
arXiv
A 2014 paper introducing Microsoft COCO, a large dataset of everyday scenes in which common objects are labeled with per-instance segmentations, built to study object recognition as part of scene understanding.
- 2011
Deep Sparse Rectifier Neural Networks
A 2011 paper by Glorot, Bordes and Bengio reporting that deep networks built from rectifying neurons match or beat tanh networks and reach their best performance without unsupervised pre-training on large labelled supervised tasks.
- 2009
ImageNet: A Large-Scale Hierarchical Image Database
CVPR
The 2009 paper introduced ImageNet, a large, cleanly labelled image database organised by the WordNet hierarchy. It later became the basis of an annual benchmark, the ImageNet Large Scale Visual Recognition Challenge.
- 2009
Learning Multiple Layers of Features from Tiny Images
UT
A 2009 technical report by Alex Krizhevsky that introduces CIFAR-10 and CIFAR-100 and studies multi-layer generative models for learning features from tiny colour images.
- 2004
MapReduce: Simplified Data Processing on Large Clusters
OSDI
Jeffrey Dean and Sanjay Ghemawat's 2004 OSDI paper presents MapReduce, a programming model in which users write a map function and a reduce function while the run-time system parallelizes the work across large clusters of commodity machines.