Science Papers

C. E. Shannon's 1948 paper set out a mathematical framework for communication, defining information in bits, channel capacity, and entropy, and treating messages statistically while setting aside semantic meaning.

A. L. Samuel's 1959 paper reporting experiments in which a checkers program, given only the rules of the game and an imperfect set of parameters, learned in 8 to 10 hours of playing time to beat the person who wrote it.

A 1969 book by Marvin Minsky and Seymour Papert that proved mathematical limits on what single-layer perceptrons can compute — notably that they cannot compute parity or connectedness — and has long been credited with steering research away from neural networks.

A 1982 paper by John J. Hopfield arguing that useful computation can emerge as a collective property of many simple, equivalent units, and describing a model whose collective behaviour gives content-addressable memory, generalisation and error correction.

A 1982 paper by John J. Hopfield showing that large networks of simple neuron-like units can collectively store and recall memories. It became known as the Hopfield network, and the work was recognised by the 2024 Nobel Prize in Physics.

In a 1986 Nature paper, Rumelhart, Hinton and Williams showed that backpropagation lets hidden units in a multilayer network learn useful internal features of a task. The method is standard in most neural networks today.

Richard Sutton's 1988 paper introduced temporal-difference learning, a family of incremental prediction methods that assign credit from changes between successive predictions and are shown to be more economical and accurate than conventional supervised approaches on most prediction problems.

Rodney Brooks's 1990 paper argued for building artificial intelligence from the ground up, with ongoing physical interaction with the environment as the main constraint on design, rather than through the abstract manipulation of symbols that had dominated the field for roughly thirty years.

Published in 1992, this paper by Boser, Guyon and Vapnik presents a training algorithm that maximizes the margin between training patterns and the decision boundary, with experiments on optical character recognition.[^1]

A 1998 conference paper by Sergey Brin and Lawrence Page describing PageRank, an algorithm that ranks web pages by the number and quality of links pointing to them, and the search engine built on that algorithm.

A generative probabilistic model that represents each document as a finite mixture over topics, fitted with variational approximate inference and EM, and evaluated on document modelling, text classification and collaborative filtering.

A 2010 paper by Xavier Glorot and Yoshua Bengio on why deep feedforward networks are hard to train: sigmoid saturation under random initialization and layer Jacobians whose singular values sit far from 1, plus a new initialization scheme that converges substantially faster.

A CVPR 2011 Best Paper that recovers 3D body joint positions from a single depth image without temporal information, by classifying pixels into body parts, and runs at 200 frames per second on consumer hardware.

A 2011 paper by Glorot, Bordes and Bengio reporting that deep networks built from rectifying neurons match or beat tanh networks and reach their best performance without unsupervised pre-training on large labelled supervised tasks.

Adam is a stochastic optimisation algorithm from a 2014 paper by Diederik P. Kingma and Jimmy Ba that sets per-parameter step sizes from running estimates of the gradient's first and second moments, needing little memory and little hyper-parameter tuning.

A 2015 Nature review by Yann LeCun, Yoshua Bengio and Geoffrey Hinton explaining deep learning: models of multiple processing layers that learn layered representations of data, and the results this produced in speech, vision and other fields.

AlphaGo Zero learned Go only by playing against itself, starting from random play, and beat the earlier champion-defeating AlphaGo 100 games to 0 after three days of training. The Nature paper appeared in October 2017.

The Nature paper described AlphaFold's methods, system design, and evaluation for protein-structure prediction.

AlphaFold 3, published in Nature in May 2024, extends structure prediction to biomolecular interactions, reporting greater accuracy than docking tools for protein–ligand binding and than AlphaFold-Multimer v.2.3 for antibody–antigen complexes.

A January 1966 paper by Joseph Weizenbaum presenting ELIZA, a program on MIT's MAC time-sharing system that used keyword-triggered decomposition and reassembly rules to enable certain natural-language conversations between people and computers.

A 1985 Cognitive Science paper by Ackley, Hinton and Sejnowski that presents a general learning rule for connectionist networks and reports simple examples of efficient internal representations.

Gerald Tesauro's 1992 paper tested whether temporal difference learning could master a complex real-world task, using self-play backgammon as the case study.

Brin and Page's PageRank technical report treats hyperlinks as citations to rank web pages and was used to build Google's search engine; it is one of the heavily cited papers on their early work.

A 2010 NIPS paper by Dahl, Ranzato, Mohamed and Hinton applies Deep Belief Nets and a mean-covariance RBM to acoustic modeling, reaching 20.5% phone error on speaker-independent TIMIT phone recognition.

A 2011 Journal of Machine Learning Research paper by Collobert, Weston, Bottou, Karlen, Kavukcuoglu and Kuksa, presenting one neural network architecture and learning algorithm for part-of-speech tagging, chunking, named entity recognition and semantic role labeling, trained on mostly unlabeled data with little task-specific engineering.

A 2012 article in IEEE Signal Processing Magazine in which four research groups argued that deep neural networks, replacing Gaussian mixture models, gave better acoustic models for speech recognition.

A 2013 paper by Mikolov, Sutskever, Chen, Corrado and Dean presents continuous Skip-gram as an efficient method for learning distributed word and phrase representations, with extensions that improve vector quality and training speed.

A 2014 paper by Srivastava, Hinton, Krizhevsky, Sutskever and Salakhutdinov introducing dropout, a training trick that randomly removes units so large neural networks overfit less.

A 2014 paper presenting an end-to-end deep-learning speech recognition system that removes hand-engineered noise modules and the phoneme dictionary, relying instead on recurrent networks trained across multiple GPUs and synthesized data.

A 2015 paper by Sergey Ioffe and Christian Szegedy that proposed normalising each layer's inputs per training mini-batch, which allowed higher learning rates, looser initialisation and much faster training of deep networks.

A 2016 analysis of residual networks arguing that with identity skip connections and the activation placed after addition, signals travel directly between blocks; a new residual unit built on this eases training, and a 1001-layer network reaches 4.62% error on CIFAR-10.

mixup is a training principle from a 2017 paper by Zhang, Cisse, Dauphin and Lopez-Paz that trains networks on convex combinations of pairs of examples and their labels, improving generalisation on image, speech and tabular benchmarks.

A 2017 arXiv paper, later published in Science, reporting that a single self-play reinforcement learning algorithm, AlphaZero, mastered chess, shogi and Go and beat strong specialised programs in each.

A 2018 paper by Chen, Rubanova, Bettencourt and Duvenaud that replaces a fixed sequence of hidden layers with a neural network describing the derivative of the hidden state, so outputs are computed by an ODE solver — yielding continuous-depth models with constant memory cost.[^1]

MuZero is a DeepMind reinforcement-learning algorithm that learns a model of its environment and plans inside that model, mastering Atari, Go, chess and shogi without being given the rules of those games.

A 2022 Nature paper by Degrave, Felici and colleagues showing that a deep reinforcement learning controller can learn on its own to command all of a tokamak's magnetic control coils, producing and holding a range of plasma shapes.

GraphCast is a graph neural network trained on four decades of reanalysis data that produces 10-day global weather forecasts at 0.25 degree resolution in under a minute, outperforming the leading operational deterministic systems on most verification targets.

GPQA is a benchmark of 448 expert-written multiple-choice questions in biology, physics and chemistry, built so that skilled non-experts cannot look the answers up, and hard enough that the strongest GPT-4 baseline scored 39% while domain experts reached 65%.

A 2023 Nature paper from researchers at Google DeepMind presenting a scaled-up deep-learning approach to predicting stable crystal structures, screening 2.2 million candidates and reporting a large jump in stability-prediction accuracy.

FrontierMath is a benchmark of hundreds of original, unpublished mathematics problems written and vetted by expert mathematicians, introduced by Epoch AI to test advanced mathematical reasoning in AI. At release, state-of-the-art models solved under 2% of the problems.[^1][^2]

A 2013 paper by Goodfellow, Warde-Farley, Mirza, Courville and Bengio introduces maxout units, whose output is the maximum of a set of inputs, designed to work with dropout, and reports state-of-the-art results on four benchmarks.

A 2013 paper by Graves, Mohamed and Hinton that examines deep recurrent neural networks for speech recognition, arguing for end-to-end sequence training with methods such as connectionist temporal classification.

Group Normalization is a 2018 paper by Yuxin Wu and Kaiming He that proposes normalizing channels in groups, independent of batch size, to address batch normalization's weakness when batches are small.