Home›Papers
Papers
Papers
Papers that changed AI, with a plain-language explainer for each.
Total 53 · Clear filters
- 2017
mixup: Beyond Empirical Risk Minimization
arXiv
mixup is a training principle from a 2017 paper by Zhang, Cisse, Dauphin and Lopez-Paz that trains networks on convex combinations of pairs of examples and their labels, improving generalisation on image, speech and tabular benchmarks.
- 2017
Mastering the game of Go without human knowledge
Nature
AlphaGo Zero learned Go only by playing against itself, starting from random play, and beat the earlier champion-defeating AlphaGo 100 games to 0 after three days of training. The Nature paper appeared in October 2017.
- 2016
Identity Mappings in Deep Residual Networks
arXiv
A 2016 analysis of residual networks arguing that with identity skip connections and the activation placed after addition, signals travel directly between blocks; a new residual unit built on this eases training, and a 1001-layer network reaches 4.62% error on CIFAR-10.
- 2015
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
arXiv
A 2015 paper by Sergey Ioffe and Christian Szegedy that proposed normalising each layer's inputs per training mini-batch, which allowed higher learning rates, looser initialisation and much faster training of deep networks.
- 2015
Deep learning
Nature
A 2015 Nature review by Yann LeCun, Yoshua Bengio and Geoffrey Hinton explaining deep learning: models of multiple processing layers that learn layered representations of data, and the results this produced in speech, vision and other fields.
- 2014
Adam: A Method for Stochastic Optimization
arXiv (ICLR 2015)
Adam is a stochastic optimisation algorithm from a 2014 paper by Diederik P. Kingma and Jimmy Ba that sets per-parameter step sizes from running estimates of the gradient's first and second moments, needing little memory and little hyper-parameter tuning.
- 2014
Deep Speech: Scaling up end-to-end speech recognition
arXiv
A 2014 paper presenting an end-to-end deep-learning speech recognition system that removes hand-engineered noise modules and the phoneme dictionary, relying instead on recurrent networks trained across multiple GPUs and synthesized data.
- 2014
Dropout: A Simple Way to Prevent Neural Networks from Overfitting
JMLR
A 2014 paper by Srivastava, Hinton, Krizhevsky, Sutskever and Salakhutdinov introducing dropout, a training trick that randomly removes units so large neural networks overfit less.
- 2013
Distributed Representations of Words and Phrases and their Compositionality
arXiv
A 2013 paper by Mikolov, Sutskever, Chen, Corrado and Dean presents continuous Skip-gram as an efficient method for learning distributed word and phrase representations, with extensions that improve vector quality and training speed.
- 2013
Speech Recognition with Deep Recurrent Neural Networks
arXiv
A 2013 paper by Graves, Mohamed and Hinton that examines deep recurrent neural networks for speech recognition, arguing for end-to-end sequence training with methods such as connectionist temporal classification.
- 2013
Maxout Networks
arXiv
A 2013 paper by Goodfellow, Warde-Farley, Mirza, Courville and Bengio introduces maxout units, whose output is the maximum of a set of inputs, designed to work with dropout, and reports state-of-the-art results on four benchmarks.
- 2012
Large Scale Distributed Deep Networks
NeurIPS
A 2012 NIPS paper by Jeffrey Dean, Andrew Ng and colleagues that introduced DistBelief, a framework for training deep networks across clusters of thousands of machines, together with the Downpour SGD and Sandblaster optimisation procedures.
- 2012
Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups
IEEE Signal Processing Magazine
A 2012 article in IEEE Signal Processing Magazine in which four research groups argued that deep neural networks, replacing Gaussian mixture models, gave better acoustic models for speech recognition.
- 2011
Deep Sparse Rectifier Neural Networks
A 2011 paper by Glorot, Bordes and Bengio reporting that deep networks built from rectifying neurons match or beat tanh networks and reach their best performance without unsupervised pre-training on large labelled supervised tasks.
- 2011
Natural Language Processing (Almost) from Scratch
A 2011 Journal of Machine Learning Research paper by Collobert, Weston, Bottou, Karlen, Kavukcuoglu and Kuksa, presenting one neural network architecture and learning algorithm for part-of-speech tagging, chunking, named entity recognition and semantic role labeling, trained on mostly unlabeled data with little task-specific engineering.
- 2011
Real-Time Human Pose Recognition in Parts from a Single Depth Image
A CVPR 2011 Best Paper that recovers 3D body joint positions from a single depth image without temporal information, by classifying pixels into body parts, and runs at 200 frames per second on consumer hardware.
- 2010
Phone recognition with the mean-covariance restricted Boltzmann Machine
NIPS
A 2010 NIPS paper by Dahl, Ranzato, Mohamed and Hinton applies Deep Belief Nets and a mean-covariance RBM to acoustic modeling, reaching 20.5% phone error on speaker-independent TIMIT phone recognition.
- 2010
Understanding the difficulty of training deep feedforward neural networks
AISTATS / PMLR
A 2010 paper by Xavier Glorot and Yoshua Bengio on why deep feedforward networks are hard to train: sigmoid saturation under random initialization and layer Jacobians whose singular values sit far from 1, plus a new initialization scheme that converges substantially faster.
- 2003
Latent Dirichlet Allocation
JMLR
A generative probabilistic model that represents each document as a finite mixture over topics, fitted with variational approximate inference and EM, and evaluated on document modelling, text classification and collaborative filtering.
- 1999
The PageRank Citation Ranking: Bringing Order to the Web
Brin and Page's PageRank technical report treats hyperlinks as citations to rank web pages and was used to build Google's search engine; it is one of the heavily cited papers on their early work.