Research history Papers

Alan Turing's 1936 paper defined computable numbers through an abstract machine, now called the Turing machine, and showed that no general method can decide whether any given mathematical statement is provable.

In 1943 Warren McCulloch and Walter Pitts described a simplified model of a neuron and showed that networks of such units can carry out logical reasoning, an early link between brain models and computation.

C. E. Shannon's 1948 paper set out a mathematical framework for communication, defining information in bits, channel capacity, and entropy, and treating messages statistically while setting aside semantic meaning.

In 1950 Alan Turing replaced the question "Can machines think?" with the imitation game, now called the Turing test, and answered the main objections to machine intelligence one by one.

In 1958 Frank Rosenblatt described the perceptron, a model whose connection strengths change with training so that it learns to classify stimuli by trial and error. It became the starting point for trainable neural networks.

John McCarthy's 1959 paper proposing the advice taker, a program that would solve problems by manipulating sentences in formal languages and act when it deduced an imperative — an early statement of logical AI.

A. L. Samuel's 1959 paper reporting experiments in which a checkers program, given only the rules of the game and an imperfect set of parameters, learned in 8 to 10 hours of playing time to beat the person who wrote it.

Marvin Minsky's 1961 survey in Proceedings of the IRE organized heuristic programming into five problem areas — search, pattern recognition, learning, planning and induction — and argued that machines would need to build models of their environments.

A 1969 book by Marvin Minsky and Seymour Papert that proved mathematical limits on what single-layer perceptrons can compute — notably that they cannot compute parity or connectedness — and has long been credited with steering research away from neural networks.

A 1982 paper by John J. Hopfield showing that large networks of simple neuron-like units can collectively store and recall memories. It became known as the Hopfield network, and the work was recognised by the 2024 Nobel Prize in Physics.

A 1982 paper by John J. Hopfield arguing that useful computation can emerge as a collective property of many simple, equivalent units, and describing a model whose collective behaviour gives content-addressable memory, generalisation and error correction.

In a 1986 Nature paper, Rumelhart, Hinton and Williams showed that backpropagation lets hidden units in a multilayer network learn useful internal features of a task. The method is standard in most neural networks today.

Richard Sutton's 1988 paper introduced temporal-difference learning, a family of incremental prediction methods that assign credit from changes between successive predictions and are shown to be more economical and accurate than conventional supervised approaches on most prediction problems.

Rodney Brooks's 1990 paper argued for building artificial intelligence from the ground up, with ongoing physical interaction with the environment as the main constraint on design, rather than through the abstract manipulation of symbols that had dominated the field for roughly thirty years.

Published in 1992, this paper by Boser, Guyon and Vapnik presents a training algorithm that maximizes the margin between training patterns and the decision boundary, with experiments on optical character recognition.[^1]

Published in 1997 by Sepp Hochreiter and Jürgen Schmidhuber, the LSTM paper introduced a memory cell with gates and a "constant error carousel" that lets recurrent networks learn across long time gaps. It shaped the recurrent models that came before Transformers.

A 1998 conference paper by Sergey Brin and Lawrence Page describing PageRank, an algorithm that ranks web pages by the number and quality of links pointing to them, and the search engine built on that algorithm.

A generative probabilistic model that represents each document as a finite mixture over topics, fitted with variational approximate inference and EM, and evaluated on document modelling, text classification and collaborative filtering.

A 2006 paper by Hinton, Osindero and Teh that uses complementary priors and a greedy layer-by-layer algorithm to train deep belief nets, then fine-tunes them; the resulting three-hidden-layer model both modelled and classified handwritten digits well.

A 2009 ICML paper by Rajat Raina, Anand Madhavan and Andrew Y. Ng that used graphics processors to train deep belief networks and sparse coding models many times faster than multicore CPUs.

The 2009 paper introduced ImageNet, a large, cleanly labelled image database organised by the WordNet hierarchy. It later became the basis of an annual benchmark, the ImageNet Large Scale Visual Recognition Challenge.

A 2010 paper by Xavier Glorot and Yoshua Bengio on why deep feedforward networks are hard to train: sigmoid saturation under random initialization and layer Jacobians whose singular values sit far from 1, plus a new initialization scheme that converges substantially faster.

A CVPR 2011 Best Paper that recovers 3D body joint positions from a single depth image without temporal information, by classifying pixels into body parts, and runs at 200 frames per second on consumer hardware.

A 2011 paper by Glorot, Bordes and Bengio reporting that deep networks built from rectifying neurons match or beat tanh networks and reach their best performance without unsupervised pre-training on large labelled supervised tasks.

The 2012 NeurIPS paper that introduced AlexNet, a large convolutional network trained on GPUs. It won the ILSVRC-2012 image classification competition by a wide margin and is widely seen as a turning point for deep learning in computer vision.

Goodfellow, Bengio, and co-authors proposed a generative-model framework with adversarially trained generator and discriminator.

Submitted to arXiv in September 2014 by Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio, this paper let a translation model look back at the whole source sentence while producing each word. This mechanism, called attention, later became a basis for Transformer models.

A 2014 study showing that stacking many small 3×3 convolution filters, up to 16–19 weight layers, improves large-scale image recognition; its models took first place in localisation and second in classification at ImageNet Challenge 2014 and were released publicly.

Ilya Sutskever, Oriol Vinyals and Quoc Le's 2014 paper showed that two LSTMs, one reading the input and one writing the output, could translate sentences and outperform a phrase-based statistical baseline. It set the encoder–decoder pattern behind later translation models.

The 2014 paper that introduced the Inception architecture and its 22-layer GoogLeNet network, which took first place in the ImageNet 2014 classification and detection challenge with a 6.67% top-5 error.

GloVe is a 2014 method that learns word vectors by factorising global word co-occurrence counts. Released together with pre-trained vectors, it became, alongside word2vec, one of the two standard ways of producing static word embeddings.

Adam is a stochastic optimisation algorithm from a 2014 paper by Diederik P. Kingma and Jimmy Ba that sets per-parameter step sizes from running estimates of the gradient's first and second moments, needing little memory and little hyper-parameter tuning.

A 2015 Nature review by Yann LeCun, Yoshua Bengio and Geoffrey Hinton explaining deep learning: models of multiple processing layers that learn layered representations of data, and the results this produced in speech, vision and other fields.

A 2015 Nature paper by Mnih, Kavukcuoglu, Silver and colleagues showed that one deep Q-network agent, given only screen pixels and the game score, learned to play 49 Atari 2600 games at a level comparable to a professional human tester.

A 2016 paper by Barret Zoph and Quoc V. Le that uses a recurrent network trained with reinforcement learning to design neural network architectures automatically, matching human-designed models on CIFAR-10 and beating LSTM on Penn Treebank.

Mask R-CNN is a 2017 framework for object instance segmentation. It detects objects and produces a high-quality mask for each instance by adding a parallel mask branch to Faster R-CNN.

A 2017 paper that introduced the Transformer, a neural network built entirely on attention with no recurrence. It set new machine translation results and became the architecture behind almost every large language model since.

AlphaGo Zero learned Go only by playing against itself, starting from random play, and beat the earlier champion-defeating AlphaGo 100 games to 0 after three days of training. The Nature paper appeared in October 2017.

A 2018 paper by Matthew E. Peters and five co-authors that represents each word with the internal states of a pre-trained deep bidirectional language model, so vectors change with context; adding them to existing models improved results on six NLP tasks.

OpenAI's 2018 paper introducing GPT, a Transformer decoder pre-trained to predict the next token and then fine-tuned for specific tasks. It improved the state of the art on 9 of 12 tasks studied and set the recipe the GPT series followed.

Google's 2018 BERT paper introduced an encoder-only Transformer that reads text in both directions and is pre-trained by hiding words. It set new results on eleven language tasks and made download-and-fine-tune the default workflow.

An NVIDIA paper from 2019 presenting an intra-layer model parallelism method, implemented with a few communication operations in native PyTorch, that trained transformer models of up to 8.3 billion parameters and improved several benchmarks.

A 2019 paper by Francois Chollet that defines intelligence as skill-acquisition efficiency and introduces the Abstraction and Reasoning Corpus (ARC), a benchmark of grid tasks that assumes no specialised knowledge or language.

A 2020 paper by Mildenhall, Srinivasan, Tancik, Barron, Ramamoorthi and Ng that represents a scene as a neural radiance field: a network mapping 5D coordinates to density and colour, optimized from a sparse set of images with known camera poses.

A 2020 paper that pairs a pre-trained seq2seq model with a dense Wikipedia index fetched by a neural retriever, setting state of the art on three open-domain QA tasks and producing more factual text than a parametric-only baseline.

A benchmark released in 2020 that tests language models across 57 subjects, from elementary mathematics to law, and showed that even the largest models of the time were far from expert-level.

A 2020 paper showing that a plain Transformer applied directly to sequences of image patches, with no convolution at all, can match state-of-the-art convolutional networks on image classification.

A 2021 paper by Fedus, Zoph and Shazeer that simplifies Mixture-of-Experts routing so that language models can hold up to a trillion parameters while keeping per-example computation roughly constant.

A 2021 FAccT paper by Bender, Gebru, McMillan-Major and Shmitchell that asks whether language models can be too large and sets out recommendations on costs, data, research goals and alternative directions.

LoRA is a fine-tuning method that freezes a pre-trained model's weights and trains small low-rank matrices injected into each layer, cutting trainable parameters and memory by orders of magnitude.

A 2021 paper from OpenAI introduces Codex, a GPT model fine-tuned on public GitHub code, and HumanEval, an evaluation set that judges generated Python programs by whether they work rather than by how closely they match a reference.

The Nature paper described AlphaFold's methods, system design, and evaluation for protein-structure prediction.

A 2021 paper showing that finetuning a language model on many tasks described by instructions — instruction tuning — makes it much better at unseen tasks without examples, surpassing a larger model on most benchmarks tested.

A 2021 paper by Rombach, Blattmann, Lorenz, Esser and Ommer that runs diffusion models inside the latent space of pretrained autoencoders, cutting the compute needed for high-resolution image synthesis while adding cross-attention for text and bounding-box control.

FlashAttention is an exact attention algorithm built around GPU memory traffic: by tiling computation so intermediates stay on chip, it cuts reads and writes to high bandwidth memory, training Transformers faster and enabling longer sequences.

ReAct is a prompting method that has a language model interleave written reasoning with calls to outside tools or environments, improving question answering, fact verification and interactive decision-making.

LLaMA is a family of foundation language models from 7B to 65B parameters, trained only on public datasets; its 13B model beat GPT-3 on most benchmarks, and all models were released to researchers.

OpenAI's technical report on GPT-4, submitted to arXiv in March 2023, describes a large multimodal Transformer model, reports human-level results on several professional benchmarks, and explains how the team made scaling behaviour predictable.

A 2023 paper that introduces the Segment Anything project: a promptable segmentation task, a model, and a dataset of over 1 billion masks on 11 million images, all released to advance foundation models for computer vision.

A 2023 paper by Rafailov and colleagues that rewrites the reinforcement-learning-from-human-feedback objective so a language model can be aligned to preferences with a simple classification loss, without a separate reward model or reinforcement learning.

A July 2023 paper from Meta led by Hugo Touvron that introduces and releases Llama 2, a family of pretrained and fine-tuned language models from 7 to 70 billion parameters, together with a detailed account of how the chat versions were tuned and made safer.

A benchmark of 2,294 real software engineering problems taken from GitHub issues and the pull requests that resolved them, in which a model must edit a codebase to fix an issue; on release the best model solved under 2% of them.

Mamba is a sequence-model architecture introduced by Albert Gu and Tri Dao that replaces attention with selective state space models, giving linear-time scaling, faster inference, and language-model quality competitive with Transformers twice its size.

Mixtral 8x7B is a sparse mixture-of-experts language model from Mistral AI. A router sends each token to two of eight experts per layer, so the model holds 47B parameters but uses 13B during inference.

A report from Google's Gemini Team introducing the Gemini 1.5 family, multimodal models built around very long context, with Gemini 1.5 Pro recalling information across up to 10 million tokens.[^1]

Meta's 2024 paper describing Llama 3 as a family of foundation models — 8B, 70B and 405B parameters — trained on about 15 trillion tokens and reported to match leading models such as GPT-4 across a wide range of tasks.

A 2025 paper introducing ARC-AGI-2, a benchmark of input-output puzzle tasks built to measure abstract reasoning in frontier AI systems, with human testing as a baseline and a task set designed to give a more granular signal of progress.

A January 1966 paper by Joseph Weizenbaum presenting ELIZA, a program on MIT's MAC time-sharing system that used keyword-triggered decomposition and reassembly rules to enable certain natural-language conversations between people and computers.

A 1985 Cognitive Science paper by Ackley, Hinton and Sejnowski that presents a general learning rule for connectionist networks and reports simple examples of efficient internal representations.

A 3-layer back-propagation network that takes camera and laser range finder images and outputs the direction to steer, trained on simulated road images and tested on Carnegie Mellon's autonomous navigation test vehicle.

In 1989 LeCun and colleagues at AT&T Bell Laboratories trained a convolutional network with backpropagation to read handwritten zip-code digits from U.S. mail, reaching 5.0% test error. The work showed that the convolutional design could be learned end to end on a real task.

A 1992 technical note by Watkins and Dayan that set out Q-learning, an incremental and computationally light way for an agent to learn optimal action values, and proved it converges to the optimum with probability 1.

Gerald Tesauro's 1992 paper tested whether temporal difference learning could master a complex real-world task, using self-play backgammon as the case study.

Corinna Cortes and Vladimir Vapnik's 1995 paper presents a learning machine for two-group classification that maps inputs nonlinearly into a high-dimensional feature space, constructs a linear decision surface, and extends earlier work from separable to non-separable data.

MNIST is a database of 60,000 training and 10,000 test images of handwritten digits, normalized to 28×28 pixels and described in the 1998 paper by LeCun and colleagues as a shared benchmark for document recognition.

LeCun, Bottou, Bengio and Haffner’s 1998 review argued that learning systems should replace hand-designed features. It showed that convolutional networks such as LeNet-5 outperformed other methods on handwritten digits, and it introduced the MNIST dataset.

Brin and Page's PageRank technical report treats hyperlinks as citations to rank web pages and was used to build Google's search engine; it is one of the heavily cited papers on their early work.

Leo Breiman's 2001 paper defines random forests as combinations of tree predictors whose random vectors are sampled independently from the same distribution, analyzes generalization error through tree strength and correlation, and reports error rates competitive with Adaboost and more robust to noise.

Published in 2003 by Yoshua Bengio and colleagues, this paper proposed learning a vector for every word together with a neural network that predicts the next word. It introduced the core idea behind word embeddings and reached lower test perplexity than n-gram models.

Jeffrey Dean and Sanjay Ghemawat's 2004 OSDI paper presents MapReduce, a programming model in which users write a map function and a reduce function while the run-time system parallelizes the work across large clusters of commodity machines.

CIFAR-10 and CIFAR-100 are labeled image datasets drawn from the 80 million tiny images collection, documented in a 2009 technical report by Alex Krizhevsky, Vinod Nair and Geoffrey Hinton. CIFAR-10 holds 60,000 32×32 colour images in 10 classes.

A 2009 technical report by Alex Krizhevsky that introduces CIFAR-10 and CIFAR-100 and studies multi-layer generative models for learning features from tiny colour images.

A 2010 NIPS paper by Dahl, Ranzato, Mohamed and Hinton applies Deep Belief Nets and a mean-covariance RBM to acoustic modeling, reaching 20.5% phone error on speaker-independent TIMIT phone recognition.

A 2011 workshop paper by Ronan Collobert, Koray Kavukcuoglu and Clement Farabet describing Torch7, a Lua-based numeric computing framework and machine learning library built for flexible experimentation on fast low-level kernels.

A 2011 Journal of Machine Learning Research paper by Collobert, Weston, Bottou, Karlen, Kavukcuoglu and Kuksa, presenting one neural network architecture and learning algorithm for part-of-speech tagging, chunking, named entity recognition and semantic role labeling, trained on mostly unlabeled data with little task-specific engineering.

In 2012, Quoc Le, Andrew Ng and colleagues trained a large unsupervised neural network on 10 million internet images and showed it could detect faces without a single labelled face.

A 2012 article in IEEE Signal Processing Magazine in which four research groups argued that deep neural networks, replacing Gaussian mixture models, gave better acoustic models for speech recognition.

Submitted to arXiv in January 2013 by Tomas Mikolov and colleagues at Google, this paper introduced CBOW and Skip-gram, two simple architectures that learn word vectors quickly from very large text collections. They are the basis of the word2vec project.

A 2013 paper by Mikolov, Sutskever, Chen, Corrado and Dean presents continuous Skip-gram as an efficient method for learning distributed word and phrase representations, with extensions that improve vector quality and training speed.

R-CNN, posted to arXiv by Girshick, Donahue, Darrell and Malik in November 2013, applied convolutional neural networks to region proposals for object detection and lifted mean average precision on PASCAL VOC 2012 to 53.3%, more than 30% above the previous best[^1].

Zeiler and Fergus ask why large convolutional networks perform so well on ImageNet, introduce a visualization technique for intermediate layers, and use ablation to find an architecture that outperforms Krizhevsky et al. The model also transfers to Caltech-101 and Caltech-256.

A 2013 paper by Mnih, Kavukcuoglu, Silver and colleagues showed a convolutional neural network trained with a Q-learning variant learning to play seven Atari 2600 games directly from raw pixels, beating prior methods on six and human experts on three.

A 2013 paper by Diederik P. Kingma and Max Welling that introduced a stochastic variational inference and learning algorithm for directed latent-variable models with intractable posteriors, making approximate inference practical on large datasets.

DeepFace is a 2014 face verification pipeline that rebuilt the alignment and representation stages of the standard face pipeline, reaching 97.35% accuracy on Labeled Faces in the Wild with a nine-layer network of over 120 million parameters.

A 2014 paper by Srivastava, Hinton, Krizhevsky, Sutskever and Salakhutdinov introducing dropout, a training trick that randomly removes units so large neural networks overfit less.

A 2014 paper introducing Microsoft COCO, a large dataset of everyday scenes in which common objects are labeled with per-instance segmentations, built to study object recognition as part of scene understanding.

A 2014 paper from Google researchers that introduced the Inception architecture, whose GoogLeNet model set a new state of the art in ILSVRC 2014 by widening and deepening a network within a fixed computational budget.

A 2014 paper by Graves, Wayne and Danihelka that couples a neural network to an external memory it reads and writes through attention, making the whole machine differentiable and trainable by gradient descent.

A 2014 paper presenting an end-to-end deep-learning speech recognition system that removes hand-engineered noise modules and the phoneme dictionary, relying instead on recurrent networks trained across multiple GPUs and synthesized data.

A 2015 paper by Sergey Ioffe and Christian Szegedy that proposed normalising each layer's inputs per training mini-batch, which allowed higher learning rates, looser initialisation and much faster training of deep networks.

A convolutional network for biomedical image segmentation, posted to arXiv on May 18, 2015, that trains end to end from very few annotated images and won the ISBI cell tracking challenge 2015 in its categories.

A 2015 paper that introduced a Region Proposal Network sharing full-image convolutional features with the detection network, making proposals nearly cost-free and enabling end-to-end training, 5 fps GPU detection, and state-of-the-art accuracy.

The 2015 YOLO paper recast object detection as a single neural network's regression over bounding boxes and class probabilities, achieving real-time detection while reducing false positives on empty background.

The 2015 paper introduced YOLO, which treats object detection as a single regression from image pixels to bounding boxes and class probabilities. One neural network makes predictions in one evaluation, reaching real-time speed.

The 2015 paper introduced residual networks, which add each layer's input back to its output through a shortcut connection. This made networks of over a hundred layers trainable and won first place in several ILSVRC and COCO tasks in 2015.

The 2016 Nature paper describing AlphaGo, which combined deep neural networks with tree search. In October 2015 its distributed version beat the European Go champion Fan Hui 5-0, the first time a program defeated a professional player in full-sized Go without handicap.

A 2016 analysis of residual networks arguing that with identity skip connections and the activation placed after addition, signals travel directly between blocks; a new residual unit built on this eases training, and a 1001-layer network reaches 4.62% error on CIFAR-10.

SQuAD is a reading-comprehension dataset of more than 100,000 crowdworker questions on Wikipedia articles, each answered by a span of its passage. Released in 2016, it gave machine comprehension a large, freely available benchmark with a measured human ceiling.

A 2016 paper describing Google's neural machine translation system, GNMT: a deep LSTM encoder-decoder with attention, designed to make end-to-end translation practical by addressing rare words, parallelism and inference cost.

A 2017 paper showing that an AI agent can learn a task from short human comparisons instead of a hand-written reward. The method, reinforcement learning from human feedback, later became central to training chat assistants.

A 2017 paper by Schulman, Wolski, Dhariwal, Radford and Klimov that introduced proximal policy optimization (PPO), a reinforcement learning method that is much simpler to implement and tune and became OpenAI's default RL algorithm.

A 2017 paper by Lin, Goyal, Girshick, He and Dollar that argues dense one-stage detectors are held back by extreme foreground-background imbalance and proposes focal loss, a reshaped cross-entropy that down-weights well-classified examples.

mixup is a training principle from a 2017 paper by Zhang, Cisse, Dauphin and Lopez-Paz that trains networks on convex combinations of pairs of examples and their labels, improving generalisation on image, speech and tabular benchmarks.

A 2017 paper by van den Oord, Vinyals and Kavukcuoglu proposing VQ-VAE, a generative model whose latent code is discrete rather than continuous, and whose prior is learned rather than fixed, so that useful representations can be learned without supervision.

A 2017 arXiv paper, later published in Science, reporting that a single self-play reinforcement learning algorithm, AlphaZero, mastered chess, shogi and Go and beat strong specialised programs in each.

ULMFiT is a 2018 transfer-learning method by Jeremy Howard and Sebastian Ruder for fine-tuning language models across NLP tasks; it beat the state of the art on six text-classification tasks and performed well with few labels.

GLUE is a benchmark and analysis platform that evaluates natural language understanding models across a diverse set of existing tasks, with an online comparison platform and a hand-crafted diagnostic suite for linguistic analysis.

A 2018 paper by Chen, Rubanova, Bettencourt and Duvenaud that replaces a fixed sequence of hidden layers with a neural network describing the derivative of the hidden state, so outputs are computed by an ODE solver — yielding continuous-depth models with constant memory cost.[^1]

SuperGLUE is a benchmark for general-purpose language understanding, released in 2019 as a harder successor to GLUE. It bundles eight tasks, a single-number metric, an analysis toolkit and a public leaderboard.

XLNet is a generalized autoregressive pretraining method from a 2019 paper. It learns bidirectional contexts by maximizing expected likelihood over all factorization-order permutations and outperforms BERT on 20 tasks under comparable settings.

A replication study of BERT pretraining that measures how much key hyperparameters and training data size matter, finds BERT was significantly undertrained, and shows a properly trained BERT matches or beats later models, reaching state of the art on GLUE, RACE and SQuAD.

A 2019 paper that casts every language task as text-to-text, systematically compares transfer-learning choices, releases the C4 corpus and models, and reports state-of-the-art results on summarization, question answering, and classification.

MuZero is a DeepMind reinforcement-learning algorithm that learns a model of its environment and plans inside that model, mastering Atari, Go, chess and shogi without being given the rules of those games.

A 2020 paper by Clark, Luong, Le and Manning that pre-trains text encoders by detecting replaced tokens instead of filling in masked ones, matching much larger baselines with far less compute.

GShard, submitted to arXiv in June 2020, is a set of annotation APIs and an XLA compiler extension from Google that lets existing model code express parallel computation, scaling a sparsely gated mixture-of-experts translation model past 600 billion parameters.

A 2020 paper by Alexey Dosovitskiy and colleagues showing that a pure Transformer applied directly to sequences of image patches — the Vision Transformer, or ViT — matches state-of-the-art convolutional networks on image classification when pre-trained on large amounts of data.

The Pile is an 825 GiB English text dataset of 22 combined sources, released by EleutherAI on 31 December 2020 with an arXiv paper. It became the training data for several of the group's open models.

A 2021 OpenAI paper describing a single 12-billion-parameter autoregressive transformer trained on 250 million image-text pairs, which generates images from text and, evaluated zero-shot, is competitive with domain-specific models.

MATH is a dataset of 12,500 challenging competition mathematics problems, each with a full step-by-step solution. Released in 2021, it was built to measure and train mathematical reasoning in models.

A 2021 arXiv paper by Austin, Odena, Nye and eight other authors that introduced MBPP, a benchmark of 974 short Python programming tasks, and measured how language models from 244M to 137B parameters turn natural-language descriptions into working code.

GSM8K is an OpenAI benchmark of 8.5K grade school math word problems, released with a paper showing that even the largest Transformer models scored poorly, and that training verifiers to select among many candidate solutions improved results.

Imagen is a text-to-image diffusion model that uses a large pretrained language model to encode prompts; it reached a zero-shot FID of 7.27 on COCO and introduced the DrawBench benchmark.

BIG-bench is a large community-built benchmark of 204 tasks designed to test language models on problems believed to lie beyond their current abilities, described in a paper posted in June 2022.

A 2022 paper arguing that some abilities of large language models appear only once models are large enough, and cannot be predicted by extrapolating from smaller models — so further scaling may keep widening what these models can do.

A 2022 paper by Percy Liang, Rishi Bommasani, Tony Lee and many co-authors that ran 30 language models through the same 42 scenarios and seven metrics, turning scattered benchmarks into one standardized evaluation.

Constitutional AI, a 2022 paper from Anthropic, proposes training a harmless AI assistant with AI-generated feedback guided by a written set of principles, instead of human labels of harmful outputs.

A 2022 paper by William Peebles and Saining Xie that replaces the U-Net backbone of diffusion models with a transformer operating on latent patches, showing that image quality improves predictably as compute grows and reaching a state-of-the-art FID of 2.27 on ImageNet 256×256.

A 2023 paper by Schick, Dwivedi-Yu, Dessì and co-authors showing that a language model can teach itself when to call a calculator, search engine, Q&A system, translation system or calendar, and how to use the results.

A March 2023 paper by Microsoft researchers testing an early GPT-4, arguing it shows strikingly general, near-human performance across many domains and could be seen as an early but incomplete step toward AGI.

QLoRA is a finetuning method that trains a 65B-parameter language model on a single 48GB GPU while matching full 16-bit finetuning quality, using 4-bit quantization, double quantization and paged optimizers.

FlashAttention-2 is a 2023 revision of the FlashAttention attention kernel that reorganizes work across GPU thread blocks and warps, roughly doubling speed and reaching up to 73% of the A100's theoretical peak throughput.

WebArena is a reproducible environment of four fully functional websites — e-commerce, a social forum, collaborative software development and content management — built for testing language-guided web agents. Its best GPT-4 agent completed 14.41% of tasks; humans completed 78.24%.[^1]

GPQA is a benchmark of 448 expert-written multiple-choice questions in biology, physics and chemistry, built so that skilled non-experts cannot look the answers up, and hard enough that the strongest GPT-4 baseline scored 39% while domain experts reached 65%.

GAIA is a benchmark of real-world questions that are easy for humans but hard for AI assistants, testing reasoning, multimodality, web browsing, and tool use. It compares AI progress against average human performance.

A benchmark for code LLMs that keeps collecting fresh contest problems from LeetCode, AtCoder and CodeForces, so that scores reflect ability rather than memorised test data, and that also tests code repair and execution reasoning.

OSWorld is a benchmark and environment for testing multimodal agents on 369 real computer tasks in Ubuntu, Windows and macOS; humans complete over 72.36% of the tasks, while the best model reaches only 12.24%.

Tri Dao and Albert Gu's paper introduces state space duality, a framework showing that structured state space models and attention variants are closely related, and uses it to derive Mamba-2, whose core layer runs 2 to 8 times faster than Mamba's.

MMLU-Pro is a 2024 benchmark that makes the widely used MMLU knowledge test harder: its questions are reasoning-focused and each has ten answer options instead of four.

τ-bench is a benchmark, submitted to arXiv in June 2024, that tests language agents in dynamic conversations with a simulated user, with domain-specific API tools and policy rules, and scores them by database state rather than by the text of their replies.

FlashAttention-3 is an attention algorithm for NVIDIA Hopper (H100) GPUs that reaches up to 75% utilization in FP16 and close to 1.2 PFLOPs/s in FP8, using asynchronous execution and low-precision techniques.

SAM 2 is a Meta FAIR foundation model that segments objects in images and videos from simple prompts such as points, boxes or masks, released together with a large video segmentation dataset and training code.

FrontierMath is a benchmark of hundreds of original, unpublished mathematics problems written and vetted by expert mathematicians, introduced by Epoch AI to test advanced mathematical reasoning in AI. At release, state-of-the-art models solved under 2% of the problems.[^1][^2]

Humanity's Last Exam (HLE) is a 2,500-question, multi-modal benchmark spanning dozens of academic subjects, built by experts worldwide so that frontier language models can still be measured after saturating older tests.

A paper showing that a 32B model fine-tuned on 1,000 carefully chosen questions and run with a simple "budget forcing" trick can beat o1-preview on competition maths by up to 27%, with model, data and code released.

A 2025 paper proposing Absolute Zero, a reinforcement-learning paradigm in which one model writes its own code-reasoning tasks and then solves them, with a code executor as the only judge, reaching state of the art on coding and maths without external training data.

Terminal-Bench 2.0 is a curated benchmark of 89 hard tasks carried out in computer terminal environments, built to measure how well AI agents handle realistic command-line work; frontier models and agents score below 65% on it.

A 1993 journal article by Mitchell P. Marcus, Beatrice Santorini and Mary Ann Marcinkiewicz describing the Penn Treebank: over one million words of Wall Street Journal text annotated with syntactic bracketing, built as a resource for work on English sentence structure.

A 2013 paper by Goodfellow, Warde-Farley, Mirza, Courville and Bengio introduces maxout units, whose output is the maximum of a set of inputs, designed to work with dropout, and reports state-of-the-art results on four benchmarks.

A 2013 paper by Graves, Mohamed and Hinton that examines deep recurrent neural networks for speech recognition, arguing for end-to-end sequence training with methods such as connectionist temporal classification.

A 2014 paper by Mehdi Mirza and Simon Osindero that adds conditioning data to both the generator and the discriminator, enabling a GAN to generate samples such as MNIST digits conditioned on class labels.

A 2015 arXiv paper by Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho and co-authors that introduced an image-captioning model with visual attention, able to show where it is looking while writing each word.

Group Normalization is a 2018 paper by Yuxin Wu and Kaiming He that proposes normalizing channels in groups, independent of batch size, to address batch normalization's weakness when batches are small.

A 2018 paper by Diederik P. Kingma and Prafulla Dhariwal presenting Glow, a flow-based generative model built on invertible 1x1 convolutions, which improved log-likelihood on standard benchmarks and could synthesise and manipulate large images.

HellaSwag is a 2019 benchmark in which a machine must pick the most plausible continuation of a short event description. Its wrong answers are generated and filtered to fool models rather than people.