Home›Papers
Papers
Papers
Papers that changed AI, with a plain-language explainer for each.
Total 44 · Clear filters
- 2019
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
arXiv 1909.08053
An NVIDIA paper from 2019 presenting an intra-layer model parallelism method, implemented with a few communication operations in native PyTorch, that trained transformer models of up to 8.3 billion parameters and improved several benchmarks.
- 2019
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
arXiv
SuperGLUE is a benchmark for general-purpose language understanding, released in 2019 as a harder successor to GLUE. It bundles eight tasks, a single-number metric, an analysis toolkit and a public leaderboard.
- 2018
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
arXiv
GLUE is a benchmark and analysis platform that evaluates natural language understanding models across a diverse set of existing tasks, with an online comparison platform and a hand-crafted diagnostic suite for linguistic analysis.
- 2016
SQuAD: 100,000+ Questions for Machine Comprehension of Text
arXiv
SQuAD is a reading-comprehension dataset of more than 100,000 crowdworker questions on Wikipedia articles, each answered by a span of its passage. Released in 2016, it gave machine comprehension a large, freely available benchmark with a measured human ceiling.
- 2014
GloVe: Global Vectors for Word Representation
EMNLP 2014
GloVe is a 2014 method that learns word vectors by factorising global word co-occurrence counts. Released together with pre-trained vectors, it became, alongside word2vec, one of the two standard ways of producing static word embeddings.
- 2014
Microsoft COCO: Common Objects in Context
arXiv
A 2014 paper introducing Microsoft COCO, a large dataset of everyday scenes in which common objects are labeled with per-instance segmentations, built to study object recognition as part of scene understanding.
- 2012
Large Scale Distributed Deep Networks
NeurIPS
A 2012 NIPS paper by Jeffrey Dean, Andrew Ng and colleagues that introduced DistBelief, a framework for training deep networks across clusters of thousands of machines, together with the Downpour SGD and Sandblaster optimisation procedures.
- 2012
ImageNet Classification with Deep Convolutional Neural Networks
NeurIPS
The 2012 NeurIPS paper that introduced AlexNet, a large convolutional network trained on GPUs. It won the ILSVRC-2012 image classification competition by a wide margin and is widely seen as a turning point for deep learning in computer vision.
- 2011
Deep Sparse Rectifier Neural Networks
A 2011 paper by Glorot, Bordes and Bengio reporting that deep networks built from rectifying neurons match or beat tanh networks and reach their best performance without unsupervised pre-training on large labelled supervised tasks.
- 2009
ImageNet: A Large-Scale Hierarchical Image Database
CVPR
The 2009 paper introduced ImageNet, a large, cleanly labelled image database organised by the WordNet hierarchy. It later became the basis of an annual benchmark, the ImageNet Large Scale Visual Recognition Challenge.
- 2009
Learning Multiple Layers of Features from Tiny Images
UT
A 2009 technical report by Alex Krizhevsky that introduces CIFAR-10 and CIFAR-100 and studies multi-layer generative models for learning features from tiny colour images.
- 2009
Large-scale Deep Unsupervised Learning using Graphics Processors
ICML
A 2009 ICML paper by Rajat Raina, Anand Madhavan and Andrew Y. Ng that used graphics processors to train deep belief networks and sparse coding models many times faster than multicore CPUs.
- 2004
MapReduce: Simplified Data Processing on Large Clusters
OSDI
Jeffrey Dean and Sanjay Ghemawat's 2004 OSDI paper presents MapReduce, a programming model in which users write a map function and a reduce function while the run-time system parallelizes the work across large clusters of commodity machines.
- 2003
Latent Dirichlet Allocation
JMLR
A generative probabilistic model that represents each document as a finite mixture over topics, fitted with variational approximate inference and EM, and evaluated on document modelling, text classification and collaborative filtering.
- 2001
Random Forests
Machine Learning
Leo Breiman's 2001 paper defines random forests as combinations of tree predictors whose random vectors are sampled independently from the same distribution, analyzes generalization error through tree strength and correlation, and reports error rates competitive with Adaboost and more robust to noise.
- 1999
The PageRank Citation Ranking: Bringing Order to the Web
Brin and Page's PageRank technical report treats hyperlinks as citations to rank web pages and was used to build Google's search engine; it is one of the heavily cited papers on their early work.
- 1998
Gradient-Based Learning Applied to Document Recognition
Proc. IEEE
LeCun, Bottou, Bengio and Haffner’s 1998 review argued that learning systems should replace hand-designed features. It showed that convolutional networks such as LeNet-5 outperformed other methods on handwritten digits, and it introduced the MNIST dataset.
- 1998
The Anatomy of a Large-Scale Hypertextual Web Search Engine
A 1998 conference paper by Sergey Brin and Lawrence Page describing PageRank, an algorithm that ranks web pages by the number and quality of links pointing to them, and the search engine built on that algorithm.
- 1995
Support-vector networks
Machine Learning
Corinna Cortes and Vladimir Vapnik's 1995 paper presents a learning machine for two-group classification that maps inputs nonlinearly into a high-dimensional feature space, constructs a linear decision surface, and extends earlier work from separable to non-separable data.
- 1993
Building a Large Annotated Corpus of English: The Penn Treebank
Computational Linguistics
A 1993 journal article by Mitchell P. Marcus, Beatrice Santorini and Mary Ann Marcinkiewicz describing the Penn Treebank: over one million words of Wall Street Journal text annotated with syntactic bracketing, built as a resource for work on English sentence structure.