Deep learning

In 1943 Warren McCulloch and Walter Pitts described a simplified model of a neuron and showed that networks of such units can carry out logical reasoning, an early link between brain models and computation.

An artificial neural network is a computing system of many simple connected units whose behaviour is set by connection weights. Its roots lie in 1943 models of neurons, and its learning versions began with Frank Rosenblatt's perceptron in 1958.

Geoffrey Hinton is a computer scientist born in London whose research on neural networks, including back-propagation and deep convolutional networks, is central to deep learning. He shared the 2018 ACM A.M. Turing Award and the 2024 Nobel Prize in Physics.

In 1958 Frank Rosenblatt described the perceptron, a model whose connection strengths change with training so that it learns to classify stimuli by trial and error. It became the starting point for trainable neural networks.

The Mark I Perceptron was the first hardware implementation of the perceptron algorithm: a custom analog computer built at Cornell Aeronautical Laboratory for image recognition, with photocells as input and motor-driven potentiometers as learnable weights.

Yann LeCun is a French computer scientist known for convolutional neural networks, which learn to recognise images directly from pixels. He co-authored the 1998 paper that introduced LeNet-5 and shared the 2018 ACM A.M. Turing Award.

Canadian computer scientist whose work on sequence models, word embeddings and attention helped lay foundations for modern language models. He shared the 2018 ACM A.M. Turing Award with Geoffrey Hinton and Yann LeCun.

A 1969 book by Marvin Minsky and Seymour Papert that proved mathematical limits on what single-layer perceptrons can compute — notably that they cannot compute parity or connectedness — and has long been credited with steering research away from neural networks.

A recurrent neural network introduced by John Hopfield in 1982 that stores patterns as minima of an energy function and recalls them from partial or distorted input, bringing statistical mechanics into the study of neural networks.

A 1982 paper by John J. Hopfield showing that large networks of simple neuron-like units can collectively store and recall memories. It became known as the Hopfield network, and the work was recognised by the 2024 Nobel Prize in Physics.

Autonomous driving is the operation of a road vehicle by a computer system instead of a person. Its six automation levels, its two main technical traditions and its long line of research vehicles define how the field is discussed today.

A two-volume MIT Press work by Rumelhart, McClelland and the PDP Research Group, published in 1986, that set out the parallel distributed processing framework — networks of simple units with hidden layers and sigmoid activation — and became a landmark of connectionism.

Israeli-Canadian computer scientist born in 1986 who co-built AlexNet, co-founded OpenAI and served as its chief scientist, and later founded Safe Superintelligence Inc.

In a 1986 Nature paper, Rumelhart, Hinton and Williams showed that backpropagation lets hidden units in a multilayer network learn useful internal features of a task. The method is standard in most neural networks today.

Backpropagation is the procedure that lets a multilayer neural network learn by sending output errors backward to adjust its weights, using gradient descent. A 1986 Nature paper made it widely known, and it is standard in most neural networks today.

Imitation learning trains an agent by supervised learning on expert demonstrations rather than by trial and error. Its simplest form is behaviour cloning, and its best-known weakness is distribution shift.

ALVINN was a 1988 Carnegie Mellon neural network that learned to steer the Navlab van by watching a human driver, an early demonstration of learned end-to-end driving.

A convolutional neural network applies the same small pattern detectors across a whole image, so a feature is found wherever it appears. Trained with backpropagation in 1989 and refined as LeNet-5 in 1998, it became standard in computer vision.

Published in 1992, this paper by Boser, Guyon and Vapnik presents a training algorithm that maximizes the margin between training patterns and the decision boundary, with experiments on optical character recognition.[^1]

NVIDIA is the company that created the GPU in 1999 and, through the CUDA architecture and the chips behind AlexNet, became a central supplier of computing power for modern AI.

Published in 1997 by Sepp Hochreiter and Jürgen Schmidhuber, the LSTM paper introduced a memory cell with gates and a "constant error carousel" that lets recurrent networks learn across long time gaps. It shaped the recurrent models that came before Transformers.

A 2006 paper by Hinton, Osindero and Teh that uses complementary priors and a greedy layer-by-layer algorithm to train deep belief nets, then fine-tunes them; the resulting three-hidden-layer model both modelled and classified handwritten digits well.

A 2009 ICML paper by Rajat Raina, Anand Madhavan and Andrew Y. Ng that used graphics processors to train deep belief networks and sparse coding models many times faster than multicore CPUs.

CIFAR-10 is a labelled image dataset of 60000 32×32 colour images in 10 mutually exclusive classes, split into 50000 training and 10000 test images; it was created by Krizhevsky, Nair and Hinton for training and comparing image classifiers.

The 2009 paper introduced ImageNet, a large, cleanly labelled image database organised by the WordNet hierarchy. It later became the basis of an annual benchmark, the ImageNet Large Scale Visual Recognition Challenge.

A 2010 paper by Xavier Glorot and Yoshua Bengio on why deep feedforward networks are hard to train: sigmoid saturation under random initialization and layer Jacobians whose singular values sit far from 1, plus a new initialization scheme that converges substantially faster.

A London-headquartered AI research organisation formed in 2023 by merging DeepMind with Google's Brain team, known for AlphaGo, AlphaFold and the Gemini model family.

A CVPR 2011 Best Paper that recovers 3D body joint positions from a single depth image without temporal information, by classifying pixels into body parts, and runs at 200 frames per second on consumer hardware.

A 2011 paper by Glorot, Bordes and Bengio reporting that deep networks built from rectifying neurons match or beat tanh networks and reach their best performance without unsupervised pre-training on large labelled supervised tasks.

The 2012 NeurIPS paper that introduced AlexNet, a large convolutional network trained on GPUs. It won the ILSVRC-2012 image classification competition by a wide margin and is widely seen as a turning point for deep learning in computer vision.

Attention scores how important each element of a sequence is relative to the others and turns those scores into weights, letting a model read any part of its input directly. It is the operation at the core of the transformer.

A generative adversarial network pits a generator that makes fake data against a discriminator that judges it, both trained in a zero-sum game; proposed by Ian Goodfellow and colleagues in 2014, it led to a series of architectures for generating images.

Goodfellow, Bengio, and co-authors proposed a generative-model framework with adversarially trained generator and discriminator.

Submitted to arXiv in September 2014 by Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio, this paper let a translation model look back at the whole source sentence while producing each word. This mechanism, called attention, later became a basis for Transformer models.

Attention lets a neural network weigh the parts of its input by relevance. Self-attention applies this idea inside a single sequence, so each token gathers information from every other token. It is the core operation of the Transformer.

A 2014 study showing that stacking many small 3×3 convolution filters, up to 16–19 weight layers, improves large-scale image recognition; its models took first place in localisation and second in classification at ImageNet Challenge 2014 and were released publicly.

Ilya Sutskever, Oriol Vinyals and Quoc Le's 2014 paper showed that two LSTMs, one reading the input and one writing the output, could translate sentences and outperform a phrase-based statistical baseline. It set the encoder–decoder pattern behind later translation models.

The 2014 paper that introduced the Inception architecture and its 22-layer GoogLeNet network, which took first place in the ImageNet 2014 classification and detection challenge with a 6.67% top-5 error.

GloVe is a 2014 method that learns word vectors by factorising global word co-occurrence counts. Released together with pre-trained vectors, it became, alongside word2vec, one of the two standard ways of producing static word embeddings.

Adam is a stochastic optimisation algorithm from a 2014 paper by Diederik P. Kingma and Jimmy Ba that sets per-parameter step sizes from running estimates of the gradient's first and second moments, needing little memory and little hyper-parameter tuning.

A residual connection adds a block's input to its output, so the block learns only the change it needs to make. This makes very deep networks trainable, and it is a standard part of today's transformers.

Google's first-generation Tensor Processing Unit is a custom chip built to speed up neural-network inference in its data centers. Deployed in 2015 and revealed in 2016, it showed that dedicated accelerators could beat CPUs and GPUs on cost-performance.

A 2015 Nature review by Yann LeCun, Yoshua Bengio and Geoffrey Hinton explaining deep learning: models of multiple processing layers that learn layered representations of data, and the results this produced in speech, vision and other fields.

A 2015 Nature paper by Mnih, Kavukcuoglu, Silver and colleagues showed that one deep Q-network agent, given only screen pixels and the game score, learned to play 49 Atari 2600 games at a level comparable to a professional human tester.

TensorFlow is Google's second-generation machine learning system, released as an open-source library under the Apache 2.0 license on 9 November 2015, succeeding the earlier DistBelief system.

AlphaGo is a Go-playing program from DeepMind that combined deep neural networks with Monte Carlo tree search. In 2016 it beat top professional Lee Sedol in a widely watched match, a landmark for learned systems.

A 2016 paper by Barret Zoph and Quoc V. Le that uses a recurrent network trained with reinforcement learning to design neural network architectures automatically, matching human-designed models on CIFAR-10 and beating LSTM on Penn Treebank.

PyTorch is an open-source deep-learning framework released by Meta in 2017, now governed by the PyTorch Foundation under the Linux Foundation. It is used both for research and for training and running large models in production.

Mask R-CNN is a 2017 framework for object instance segmentation. It detects objects and produces a high-quality mask for each instance by adding a parallel mask branch to Faster R-CNN.

Google TPU v2 is the second-generation Tensor Processing Unit announced in May 2017. It added floating-point support and the bfloat16 format, and scaled to 256-chip pods rated at 11.5 petaFLOPS.

NVIDIA's first data-center GPU built on its Volta architecture, announced in May 2017. It added Tensor Cores, dedicated matrix-multiply units; OpenAI later trained its GPT-3 models on V100s.

The Transformer is a neural network architecture introduced in 2017 that relies on self-attention instead of recurrence or convolution. Most of today's large language models are built on it.

A 2017 paper that introduced the Transformer, a neural network built entirely on attention with no recurrence. It set new machine translation results and became the architecture behind almost every large language model since.

AlphaGo Zero learned Go only by playing against itself, starting from random play, and beat the earlier champion-defeating AlphaGo 100 games to 0 after three days of training. The Nature paper appeared in October 2017.

A 2018 paper by Matthew E. Peters and five co-authors that represents each word with the internal states of a pre-trained deep bidirectional language model, so vectors change with context; adding them to existing models improved results on six NLP tasks.

Pre-training trains a large model on huge amounts of unlabeled text, and fine-tuning then adapts it to a specific task with a small labeled dataset. This two-stage recipe is the basis of the term foundation model.

OpenAI's 2018 paper introducing GPT, a Transformer decoder pre-trained to predict the next token and then fine-tuned for specific tasks. It improved the state of the art on 9 of 12 tasks studied and set the recipe the GPT series followed.

BERT is a language model from Google that reads text in both directions and is pre-trained by filling in masked words. Fine-tuned per task, it set new records on question answering and language-understanding benchmarks and later went into Google Search.

Google's 2018 BERT paper introduced an encoder-only Transformer that reads text in both directions and is pre-trained by hiding words. It set new results on eleven language tasks and made download-and-fine-tune the default workflow.

GPT-3 is an autoregressive language model with 175 billion parameters, described in a May 2020 paper. It could perform new tasks from a few examples in the prompt, without any gradient updates, and it was first offered through OpenAI's API.

A 2020 paper showing that a plain Transformer applied directly to sequences of image patches, with no convolution at all, can match state-of-the-art convolutional networks on image classification.

Isomorphic Labs is an AI drug discovery company launched out of DeepMind and dated to 2021, building the IsoDDE platform and working with Novartis, Eli Lilly and Johnson & Johnson.

A 2021 paper by Fedus, Zoph and Shazeer that simplifies Mixture-of-Experts routing so that language models can hold up to a trillion parameters while keeping per-example computation roughly constant.

RoseTTAFold is a three-track neural network for protein structure prediction, published by the Baker lab in 2021; its accuracy approached AlphaFold2's at CASP14, it models protein–protein complexes from sequence alone, and it is distributed as open code with a public web server.

The NVIDIA H100 is the first data-center GPU built on NVIDIA's Hopper architecture, announced in March 2022 with 80 billion transistors, HBM3 memory, FP8 Tensor Cores and a Transformer Engine aimed at large language models.

FlashAttention is an exact attention algorithm built around GPU memory traffic: by tiling computation so intermediates stay on chip, it cuts reads and writes to high bandwidth memory, training Transformers faster and enabling longer sequences.

RT-1 is a transformer-based model for real-world robot control, trained on over 130,000 episodes collected with 13 robots and reported to perform more than 700 language instructions at a 97% success rate.

A 2023 paper by Rafailov and colleagues that rewrites the reinforcement-learning-from-human-feedback objective so a language model can be aligned to preferences with a simple classification loss, without a separate reward model or reinforcement learning.

Mamba is a sequence-model architecture introduced by Albert Gu and Tri Dao that replaces attention with selective state space models, giving linear-time scaling, faster inference, and language-model quality competitive with Transformers twice its size.

Cerebras's third-generation wafer-scale AI chip, announced in March 2024: 4 trillion transistors on one 46,225-square-millimetre piece of silicon, built to keep very large models in a single memory space.

AlphaFold 3 is a structure-prediction model from Google DeepMind and Isomorphic Labs, announced on 8 May 2024. It predicts the structures and interactions of proteins, DNA, RNA, ligands and chemical modifications, extending structure prediction beyond proteins alone.[^1]

AlphaFold 3, published in Nature in May 2024, extends structure prediction to biomolecular interactions, reporting greater accuracy than docking tools for protein–ligand binding and than AlphaFold-Multimer v.2.3 for antibody–antigen complexes.

Llama 3.1, released on 23 July 2024, added a 405-billion-parameter model to the Llama 3 family and extended its context to 128,000 tokens. Meta called the 405B the largest and most capable openly available foundation model.

SK hynix's 12-layer HBM3E, announced in mass production on 26 September 2024, was the first 12-die high-bandwidth memory stack: 36 GB in a package no thicker than the eight-layer version, running at 9.6 Gbps.

π0 is a generalist robot policy from Physical Intelligence, built by adapting a 3-billion-parameter vision-language model with flow matching so one model can output fast, continuous motor commands for dexterous manipulation.

TPU7x (Ironwood) is Google's seventh-generation TPU: 192 GB of high-bandwidth memory per chip, FP8 matrix units, 9,216 chips per pod, and a programming model that exposes two devices per chip.

Huawei's Ascend 910C is a data-center AI chip in the 910 series, built on the Da Vinci architecture. It reached the market in the first quarter of 2025 with 800 TFLOPS of FP16 compute and 128 GB of HBM.

Huawei CloudMatrix384 is a Huawei Cloud service instance built on Atlas 900 A3 SuperPoDs, each linking up to 384 Ascend 910C chips so that they work as one computer.

SK hynix said on 19 March 2025 that it was the first to supply 12-layer HBM4 samples, a stack offering more than 2 TB of data per second and 36 GB of capacity, over 60% faster than HBM3E.

Huawei's Ascend 950 is a two-chip AI series announced in September 2025, pairing a shared die with Huawei's own high bandwidth memory, up to 2 PFLOPS in low precision formats and 2 TB/s interconnect.

Quantization stores a language model's weights and activations at lower numerical precision, cutting the memory and bandwidth that inference requires. Methods such as LLM.int8() and GPTQ made very large models runnable on far smaller hardware.

JAX is a Python library for accelerator-oriented array computation and program transformation, aimed at high-performance numerical computing and large-scale machine learning. It pairs a NumPy-style API with composable transformations for compilation, batching, differentiation and parallelization.

Reinforcement learning is the branch of machine learning and optimal control in which an agent learns, by trial and error, which actions in a dynamic environment maximise a reward signal.

Unsupervised learning trains algorithms on data that carry no labels, letting them find structure such as clusters, low-dimensional representations or generative models. It underlies clustering tools and neural generative models from the Hopfield network and Boltzmann machine to the variational autoencoder.

A multimodal model handles more than one kind of input or output, such as images together with text, inside a single model. Work on CLIP and Flamingo showed how such models are trained and how they can be steered toward new tasks.

Supervised learning trains a model to map inputs to outputs using paired labelled examples, so that it can predict answers for new, unseen data. Its central measure of success is how well the model generalises beyond the training set.

A vision-language-action model (VLA) is a robot policy that takes in camera images and a language instruction and outputs actions directly, usually by adapting a pretrained vision-language model to control.

A public research university in Toronto where Geoffrey Hinton has been affiliated since 1987 and where AlexNet, the 2012 deep convolutional network, was developed.

MIT, the Massachusetts Institute of Technology in Cambridge, Massachusetts, hosted the AI lab that Marvin Minsky and John McCarthy co-founded, which later became CSAIL. The lab also invented Lisp machines, and Minsky co-wrote the book Perceptrons.

The University of California, Berkeley is a public land-grant research university founded in 1868, the first land-grant university in California, known for a long Nobel record, a history of student activism and top rankings among public universities.

Stanford University is a private research university in California, home to the Stanford AI Lab (SAIL) and the Institute for Human-Centered AI, which publishes the AI Index.

Carnegie Mellon University in Pittsburgh has one of the earliest US computer science departments. Its researchers built the Logic Theorist, Sphinx speech recognition and the Deep Blue lineage, and won the 2007 DARPA Urban Challenge.

Bell Labs was the research arm of the Bell System, where the transistor, Unix and the C language were created. Yann LeCun's convolutional networks for handwritten digits were developed there, which made it an early home of neural-network research.

Marvin Minsky (1927–2016) was an American scientist who co-founded MIT’s AI laboratory and co-signed the 1955 Dartmouth proposal. With Seymour Papert he wrote the 1969 book Perceptrons, which analysed the limits of simple neural networks.

Frank Rosenblatt (1928–1971) was an American psychologist who developed the perceptron, an early learning machine that adjusted its weights from examples. His Mark I Perceptron is now held by the Smithsonian Institution.

American physicist born in 1933 who invented the Hopfield network in 1982, a model that stores and recalls patterns using ideas from spin physics, and who received half of the 2024 Nobel Prize in Physics.

David Rumelhart (1942–2011) was an American cognitive psychologist. His 1986 work with colleagues helped bring back-propagation and neural-network models to a wide audience.

MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL), formed in 2003 by merging the MIT AI Lab with the Laboratory for Computer Science, is the largest laboratory on the MIT campus.

Protein scientist at the University of Washington in Seattle, known for computational protein design: he designed a protein unlike any in nature in 2003 and received one half of the 2024 Nobel Prize in Chemistry.

German computer scientist and co-author, with Sepp Hochreiter, of the 1997 LSTM paper. He has directed the Swiss AI lab IDSIA since 1995 and has publicly argued that some contributions to AI deserve more credit.

A supervised maximum-margin model for classification and regression that separates classes by the widest possible margin and uses kernel functions to handle non-linear data.

German computer scientist who co-developed the long short-term memory (LSTM) network, a recurrent neural network design for learning from sequences such as text and speech. He led the xLSTM work in 2024.

American computer scientist known for Google's core infrastructure — MapReduce, Bigtable and Spanner — and for leading work on Google Brain, TensorFlow and, from 2023, the role of Google's chief scientist.

Stanford computer scientist Fei-Fei Li created ImageNet, the large-scale image dataset and challenge that gave data-driven computer vision a common benchmark, and has led AI research in both academia and industry.

British computer scientist born in 1976 who specialises in reinforcement learning. He led DeepMind's AlphaGo project, co-authored the 2013 deep Q-network paper and later founded Ineffable Intelligence.

Demis Hassabis is a British AI researcher and entrepreneur who co-founded DeepMind and was the last author of the 2016 AlphaGo paper. He shared the 2024 Nobel Prize in Chemistry for work on predicting protein structures.

ICML is the annual International Conference on Machine Learning, held every year since 1980 and organized by the International Machine Learning Society. It is the oldest of the field's main conferences and one of its three most influential venues.

Czech computer scientist and lead author of word2vec, the 2013 methods that learn word vectors from text. His later work includes the fastText library and research on mathematical models that grow in complexity.

CVPR is the annual Conference on Computer Vision and Pattern Recognition, first held in 1983 and usually held in June in North America. It is known for selective, double-blind review and open-access papers.

A Boltzmann machine is a neural network whose configurations are scored by a global energy and trained by a stochastic learning algorithm. Introduced by Geoffrey Hinton and Terry Sejnowski, it brought energy-based models into cognitive science and machine learning.

American scientist (born 1985) who worked on AlphaFold at Google DeepMind and shared half of the 2024 Nobel Prize in Chemistry with Demis Hassabis for protein structure prediction.

A recurrent neural network (RNN) is a neural network with feedback connections that carry information from one step of a sequence to the next. Its long short-term memory variant (LSTM) was designed to keep useful memories over longer spans.

AI researcher and educator, born in Bratislava and raised partly in Toronto, who co-founded OpenAI, led AI at Tesla, taught Stanford's CS231n course, and later joined Anthropic's pre-training team.

NETtalk is an artificial neural network built by Terrence Sejnowski and Charles Rosenberg that learns to convert written English into phonemes and audible speech, showing that a network trained on examples can generalize to unseen words.

NeurIPS is an annual December conference on machine learning and computational neuroscience, founded in 1987 and counted among the most influential meetings in AI research.

In 1989 LeCun and colleagues at AT&T Bell Laboratories trained a convolutional network with backpropagation to read handwritten zip-code digits from U.S. mail, reaching 5.0% test error. The work showed that the convolutional design could be learned end to end on a real task.

Microsoft's corporate research division, founded in 1991, whose researchers produced work such as the ResNet image-recognition architecture and helped build features used in Kinect, Bing, HoloLens, Cortana and Microsoft Translator.

A 1992 technical note by Watkins and Dayan that set out Q-learning, an incremental and computationally light way for an agent to learn optimal action values, and proved it converges to the optimum with probability 1.

Chelsea Finn is a computer scientist and assistant professor at Stanford University whose research spans meta-learning, deep reinforcement learning and robot learning; she is a co-founder of Physical Intelligence.

Mila, the Quebec AI Institute, is a Montreal deep-learning institute founded in 1993 by Yoshua Bengio as LISA; its graduates include Ian Goodfellow and Hugo Larochelle.

Tokenization splits text into the tokens a language model actually reads; most modern systems do it with byte pair encoding, a 1994 compression method adapted for translation in 2015.

Corinna Cortes and Vladimir Vapnik's 1995 paper presents a learning machine for two-group classification that maps inputs nonlinearly into a high-dimensional feature space, constructs a linear decision surface, and extends earlier work from separable to non-separable data.

AdaBoost, short for Adaptive Boosting, is a statistical classification meta-algorithm introduced by Yoav Freund and Robert Schapire in 1995. It reweights training examples, trains weak classifiers in sequence and combines their outputs into one weighted classifier.

MNIST is a database of 60,000 training and 10,000 test images of handwritten digits, normalized to 28×28 pixels and described in the 1998 paper by LeCun and colleagues as a shared benchmark for document recognition.

LeCun, Bottou, Bengio and Haffner’s 1998 review argued that learning systems should replace hand-designed features. It showed that convolutional networks such as LeNet-5 outperformed other methods on handwritten digits, and it introduced the MNIST dataset.

Google's research division, based in Mountain View and usually dated to 2000, produced the Transformer, BERT and T5 — the work underlying modern large language models.

Leo Breiman's 2001 paper defines random forests as combinations of tree predictors whose random vectors are sampled independently from the same distribution, analyzes generalization error through tree strength and correlation, and reports error rates competitive with Adaboost and more robust to noise.

Torch is a scientific computing framework for machine learning built on the LuaJIT scripting language with a C/CUDA core, designed to put GPUs first and surrounded by a large ecosystem of community packages.

A word embedding represents each word as a list of numbers learned from text, so that words used in similar ways get similar lists. Embeddings are a basic building block of modern language models.

Published in 2003 by Yoshua Bengio and colleagues, this paper proposed learning a vector for every word together with a neural network that predicts the next word. It introduced the core idea behind word embeddings and reached lower test perplexity than n-gram models.

Connectionist temporal classification (CTC) is a method for training recurrent neural networks to label unsegmented sequences, such as speech or handwriting, without frame-by-frame alignment. Introduced in 2006, it let CTC-trained LSTMs win the first international contests for recurrent networks.

Theano is a Python library and optimizing compiler for mathematical expressions, especially matrix-valued ones, developed at the Montreal Institute for Learning Algorithms. Released in 2007, it combined symbolic differentiation, GPU execution and C code generation for numerically heavy research and teaching.

CIFAR-10 and CIFAR-100 are labeled image datasets drawn from the 80 million tiny images collection, documented in a 2009 technical report by Alex Krizhevsky, Vinod Nair and Geoffrey Hinton. CIFAR-10 holds 60,000 32×32 colour images in 10 classes.

A 2009 technical report by Alex Krizhevsky that introduces CIFAR-10 and CIFAR-100 and studies multi-layer generative models for learning features from tiny colour images.

Deep neural network acoustic modeling uses multilayer networks to map speech to units such as phones. A 2010 NIPS paper used Deep Belief Nets and a mean-covariance RBM to extract speech features, reaching 20.5% phone error on TIMIT.

Graphics processing units (GPUs) run many arithmetic operations in parallel, which suits neural-network training. Their use from 2010 onward made deep learning practical at much larger scales.

Xavier (Glorot) initialization is a weight-initialization scheme from Glorot and Bengio's 2010 paper that sets each layer's starting weight scale so signals stay stable across layers, giving substantially faster convergence in deep feedforward networks.

A consumer graphics card NVIDIA released in November 2010 on its Fermi architecture. Two of them, each with 3 GB of memory, later trained AlexNet, the network that won ILSVRC-2012 and which NVIDIA's timeline credits with sparking the era of modern AI.

A 2010 NIPS paper by Dahl, Ranzato, Mohamed and Hinton applies Deep Belief Nets and a mean-covariance RBM to acoustic modeling, reaching 20.5% phone error on speaker-independent TIMIT phone recognition.

A 2011 workshop paper by Ronan Collobert, Koray Kavukcuoglu and Clement Farabet describing Torch7, a Lua-based numeric computing framework and machine learning library built for flexible experimentation on fast low-level kernels.

Google Brain was a deep-learning research team started inside Google's X in 2011. Its cat-recognition experiment, TensorFlow, sequence-to-sequence models and the Transformer made it central to modern AI; it merged into Google DeepMind in 2023.

A 2011 Journal of Machine Learning Research paper by Collobert, Weston, Bottou, Karlen, Kavukcuoglu and Kuksa, presenting one neural network architecture and learning algorithm for part-of-speech tagging, chunking, named entity recognition and semantic role labeling, trained on mostly unlabeled data with little task-specific engineering.

AlexNet is the eight-layer convolutional network that won the ILSVRC2012 image classification task by a wide margin, showing that a large network trained on a large labelled dataset with GPUs could beat hand-designed features.

ICLR is an annual machine learning conference held in late April or early May, founded in 2012 by Yann LeCun and Yoshua Bengio, and known for its open peer review process.

A research institute at the University of Washington, founded in 2012 and led by David Baker, that designs new proteins for medicine and technology; its labs produced RoseTTAFold and, it says, the first computationally designed vaccine.

In 2012, Quoc Le, Andrew Ng and colleagues trained a large unsupervised neural network on 10 million internet images and showed it could detect faces without a single labelled face.

A 2012 article in IEEE Signal Processing Magazine in which four research groups argued that deep neural networks, replacing Gaussian mixture models, gave better acoustic models for speech recognition.

A 2012 NIPS paper by Jeffrey Dean, Andrew Ng and colleagues that introduced DistBelief, a framework for training deep networks across clusters of thousands of machines, together with the Downpour SGD and Sandblaster optimisation procedures.

A variational autoencoder is a generative model whose encoder maps an input to a distribution over a latent space and whose decoder reconstructs the input from a sample; training maximises a lower bound on the data likelihood.

Submitted to arXiv in January 2013 by Tomas Mikolov and colleagues at Google, this paper introduced CBOW and Skip-gram, two simple architectures that learn word vectors quickly from very large text collections. They are the basis of the word2vec project.

A 2013 paper by Mikolov, Sutskever, Chen, Corrado and Dean presents continuous Skip-gram as an efficient method for learning distributed word and phrase representations, with extensions that improve vector quality and training speed.

Recursion is a drug discovery company that pairs automated laboratory experiments with computation; its Recursion OS platform spans target identification to clinical trial enrolment. It listed on Nasdaq in 2021 and combined with Exscientia in 2024.

R-CNN, posted to arXiv by Girshick, Donahue, Darrell and Malik in November 2013, applied convolutional neural networks to region proposals for object detection and lifted mean average precision on PASCAL VOC 2012 to 53.3%, more than 30% above the previous best[^1].

Zeiler and Fergus ask why large convolutional networks perform so well on ImageNet, introduce a visualization technique for intermediate layers, and use ablation to find an architecture that outperforms Krizhevsky et al. The model also transfers to Caltech-101 and Caltech-256.

A DeepMind reinforcement-learning agent that combined Q-learning with a convolutional neural network to play Atari 2600 games straight from raw pixels, reaching human-level performance on many titles without being told the rules.

A 2013 paper by Mnih, Kavukcuoglu, Silver and colleagues showed a convolutional neural network trained with a Q-learning variant learning to play seven Atari 2600 games directly from raw pixels, beating prior methods on six and human experts on three.

A 2013 paper by Diederik P. Kingma and Max Welling that introduced a stochastic variational inference and learning algorithm for directed latent-variable models with intractable posteriors, making approximate inference practical on large datasets.

A 2014 paper by Srivastava, Hinton, Krizhevsky, Sutskever and Salakhutdinov introducing dropout, a training trick that randomly removes units so large neural networks overfit less.

DeepFace is a 2014 face verification pipeline that rebuilt the alignment and representation stages of the standard face pipeline, reaching 97.35% accuracy on Labeled Faces in the Wild with a nine-layer network of over 120 million parameters.

Preferred Networks, Inc. is a Tokyo technology company founded in 2014 that builds AI processors, runs its own supercomputer and develops Japanese large language models, with major Japanese manufacturers among its shareholders.

Caffe is a BSD-licensed C++ library with Python and MATLAB bindings for training and deploying convolutional neural networks, described in a 2014 paper by Yangqing Jia and colleagues.

A 2014 paper from Google researchers that introduced the Inception architecture, whose GoogLeNet model set a new state of the art in ILSVRC 2014 by widening and deepening a network within a fixed computational budget.

A Chinese chip designer founded in 2014 that builds x86-compatible processors and DCU accelerators, listed on the STAR Market in 2022 and positioning itself around a self-developed C86 plus GPGPU product line.

A 2014 paper by Graves, Wayne and Danihelka that couples a neural network to an external memory it reads and writes through attention, making the whole machine differentiable and trainable by gradient descent.

A 2014 paper by Vinyals and colleagues that trained a single deep network to turn a photograph into a sentence, reporting a BLEU-1 score of 59 on Pascal against 25 for the previous state of the art.

A 2014 paper presenting an end-to-end deep-learning speech recognition system that removes hand-engineered noise modules and the phoneme dictionary, relying instead on recurrent networks trained across multiple GPUs and synthesized data.

Knowledge distillation trains a small model to reproduce the outputs of a larger model or an ensemble, so that much of the predictive performance is kept at lower compute and deployment cost.

A technique that re-centres each layer's inputs to zero and rescales them to unit variance over a mini-batch, making deep networks faster and more stable to train.

A 2015 paper by Sergey Ioffe and Christian Szegedy that proposed normalising each layer's inputs per training mini-batch, which allowed higher learning rates, looser initialisation and much faster training of deep networks.

Keras is a deep learning API written in Python that runs on top of JAX, TensorFlow or PyTorch. Since 2015 it has offered a high-level way to define, train and export neural networks without tying a project to one framework.

A 2015 paper that introduced a Region Proposal Network sharing full-image convolutional features with the detection network, making proposals nearly cost-free and enabling end-to-end training, 5 fps GPU detection, and state-of-the-art accuracy.

The 2015 paper introduced YOLO, which treats object detection as a single regression from image pixels to bounding boxes and class probabilities. One neural network makes predictions in one evaluation, reaching real-time speed.

The 2015 YOLO paper recast object detection as a single neural network's regression over bounding boxes and class probabilities, achieving real-time detection while reducing false positives on empty background.

The 2015 paper introduced residual networks, which add each layer's input back to its output through a shortcut connection. This made networks of over a hundred layers trainable and won first place in several ILSVRC and COCO tasks in 2015.

The 2016 Nature paper describing AlphaGo, which combined deep neural networks with tree search. In October 2015 its distributed version beat the European Go champion Fan Hui 5-0, the first time a program defeated a professional player in full-sized Go without handicap.

A Chinese AI chip company founded in 2016 and listed on the STAR Market in 2020. It designs cloud, edge and endpoint chips, and in 2025 it turned profitable as revenue rose sharply.

A 2016 analysis of residual networks arguing that with identity skip connections and the activation placed after addition, signals travel directly between blocks; a new residual unit built on this eases training, and a 1001-layer network reaches 4.62% error on CIFAR-10.

The Tesla P100, announced by NVIDIA in April 2016, was the first data-center GPU on the Pascal architecture, and the first to combine stacked HBM2 memory, half-precision math aimed at deep learning and the NVLink interconnect.

NVIDIA's DGX-1, unveiled in April 2016, packed eight Tesla P100 GPUs into a single chassis and was billed as the world's first deep learning supercomputer, turning accelerator count into one purchasable machine.

PaddlePaddle (飞桨) is Baidu's open-source deep learning framework, released in September 2016 out of an internal platform begun in 2013; at launch it was described as China's first open-source deep learning platform.

A DeepMind generative model that predicts raw audio one sample at a time, bringing neural text-to-speech close to human quality and later powering Google Assistant and Cloud Text-to-Speech voices.

WaveNet is a deep neural network that generates raw audio waveforms one sample at a time, predicting each sample from all the ones before it. Used for text-to-speech, it was rated by listeners as more natural than the systems it was compared with.

A 2016 paper describing Google's neural machine translation system, GNMT: a deep LSTM encoder-decoder with attention, designed to make end-to-end translation practical by addressing rare words, parallelism and inference cost.

A component of sequence models, especially transformers, that encodes each token's position as a vector, giving order-blind attention a sense of sequence; its forms range from the original transformer's sinusoids to RoPE and ALiBi.

Tensor Cores are specialised units inside NVIDIA GPUs that perform a small matrix multiply-and-add in one step, making the matrix arithmetic behind neural-network training and inference dramatically faster.

A 2017 paper by Schulman, Wolski, Dhariwal, Radford and Klimov that introduced proximal policy optimization (PPO), a reinforcement learning method that is much simpler to implement and tune and became OpenAI's default RL algorithm.

A 2017 paper by Lin, Goyal, Girshick, He and Dollar that argues dense one-stage detectors are held back by extreme foreground-background imbalance and proposes focal loss, a reshaped cross-entropy that down-weights well-classified examples.

ONNX is an open format for representing machine learning models, announced by Microsoft and Facebook in September 2017 so that developers can move models between frameworks instead of converting them between incompatible formats.

A DeepMind Go program that learned the game from scratch by playing against itself, using no human game records, and beat earlier versions of AlphaGo within days.

mixup is a training principle from a 2017 paper by Zhang, Cisse, Dauphin and Lopez-Paz that trains networks on convex combinations of pairs of examples and their labels, improving generalisation on image, speech and tabular benchmarks.

AlphaZero is a general game-playing system from DeepMind, described in a preprint in December 2017, that reached a superhuman level in chess, shogi and Go within 24 hours starting from random play, using only the rules of each game.

A 2017 arXiv paper, later published in Science, reporting that a single self-play reinforcement learning algorithm, AlphaZero, mastered chess, shogi and Go and beat strong specialised programs in each.

Google's third-generation Tensor Processing Unit, introduced in 2018, with doubled HBM capacity, higher peak throughput, lower power draw and liquid cooling for denser pods.

ULMFiT is a 2018 transfer-learning method by Jeremy Howard and Sebastian Ruder for fine-tuning language models across NLP tasks; it beat the state of the art on six text-classification tasks and performed well with few labels.

OpenAI Five was a team of five reinforcement-learning bots that played the video game Dota 2; in April 2019 it beat OG, the reigning world champions, in a best-of-three series.

GPT-1 was OpenAI's first generative pre-trained transformer, published in June 2018. It showed that one language model, pre-trained on unlabelled text and then fine-tuned, could improve results on many language tasks.

A 2018 paper by Chen, Rubanova, Bettencourt and Duvenaud that replaces a fixed sequence of hidden layers with a neural network describing the derivative of the hidden state, so outputs are computed by an ODE solver — yielding continuous-depth models with constant memory cost.[^1]

DeepMind's first protein-structure prediction system, which placed first overall in the 2018 CASP13 blind assessment by estimating probability distributions of distances between residues.

AlphaStar is a DeepMind system that plays StarCraft II. In 2019 it reached Grandmaster level with all three races, ranking above 99.8% of active players on Battle.net.

GPT-2 is a large language model that OpenAI announced in February 2019. Its largest version has 1.5 billion parameters. OpenAI first withheld the full model, then released it in stages through November 2019.

Baidu's first ERNIE pre-trained language model, released in March 2019, added entity- and phrase-level masking to BERT-style pre-training and reported state-of-the-art results on five Chinese NLP tasks.

XLNet is a generalized autoregressive pretraining method from a 2019 paper. It learns bidirectional contexts by maximizing expected likelihood over all factorization-order permutations and outperforms BERT on 20 tasks under comparable settings.

A replication study of BERT pretraining that measures how much key hyperparameters and training data size matter, finds BERT was significantly undertrained, and shows a properly trained BERT matches or beats later models, reaching state of the art on GLUE, RACE and SQuAD.

Huawei's Ascend 910 is an AI training processor launched in 2019 alongside the MindSpore framework, rated at 256 teraflops FP16 and used to train the PanGu-α language models on clusters of thousands of chips.

Chinese chip designer founded in 2019 that builds general-purpose GPU (GPGPU) chips and intelligent computing solutions on its self-developed BILI series hardware and BIRENSUPA software, for AI training and inference from cloud to edge.

T5 is a family of Google Transformer models that recasts every language task as text in, text out. Its 11-billion-parameter version set state of the art on several benchmarks, and the code, models and C4 dataset were released openly.

MuZero is a DeepMind reinforcement-learning algorithm that learns a model of its environment and plans inside that model, mastering Atari, Go, chess and shogi without being given the rules of those games.

Chinese GPU company Moore Threads, founded in 2020 by former Nvidia China head Zhang Jianzhong, sells AI accelerator cards, servers and clusters built on its MUSA architecture. It listed on the STAR Market in 2025 and is growing fast while still loss-making.

Shenmai FFR is a software package that applies deep learning to CT coronary angiograms to estimate coronary flow reserve, helping clinicians judge whether a narrowing will cause myocardial ischaemia. It holds medical device approvals in China, the EU and the US.

A 2020 paper by Clark, Luong, Le and Manning that pre-trains text encoders by detecting replaced tokens instead of filling in masked ones, matching much larger baselines with far less compute.

NVIDIA's third-generation DGX system, unveiled on 14 May 2020: eight A100 GPUs, 320 GB of GPU memory, five petaflops of rated AI performance, and the ability to split into as many as 56 instances.

Diffusion models learn to reverse a gradual noising process, so they can turn random noise into a new image by removing noise step by step. Latent diffusion, which runs this in a compressed space, became the basis of Stable Diffusion.

A 2020 paper by Jonathan Ho, Ajay Jain and Pieter Abbeel that trained a U-Net to reverse a gradual noising process, making diffusion models a practical rival to GANs and paving the way for DALL-E 2, Imagen and Stable Diffusion.

A 2020 paper by Alexey Dosovitskiy and colleagues showing that a pure Transformer applied directly to sequences of image patches — the Vision Transformer, or ViT — matches state-of-the-art convolutional networks on image classification when pre-trained on large amounts of data.

AlphaFold2 is DeepMind's protein structure prediction system, announced in 2020. It reached accuracy competitive with experimental methods in many cases, and work in this area received half of the 2024 Nobel Prize in Chemistry.

Cambricon's third-generation cloud AI chip, released in 2021: a 7 nm chiplet design with two compute dies, 256 TOPS at INT8, LPDDR5 memory and multi-chip interconnect, positioned for cost-effective inference.

A state space model represents a sequence as a hidden state that evolves with the input. S4 brought the idea into deep learning in 2021, and Mamba made its parameters input-dependent, enabling linear-time modeling of long sequences.

An image-text model from OpenAI that learns to match pictures with their captions. Trained on about 400 million image-caption pairs, it can classify images by comparing them against written prompts, with no task-specific training.

DALL·E is OpenAI's text-to-image model unveiled on 5 January 2021: a 12-billion-parameter transformer trained on 250 million text-image pairs that generates a picture token by token from a caption.

Kunlunxin is the AI chip company spun out of Baidu in April 2021, continuing Baidu's Kunlun chip line. It designs processors for AI workloads in data centres and is one of China's largest domestic AI chip suppliers.

LaMDA is a family of conversational language models that Google announced in 2021 and described in a 2022 paper, notable for its work on safety and factual grounding in dialogue and for later powering Bard.

Triton is OpenAI's open-source, Python-like language and compiler for writing GPU kernels. It lets researchers with no CUDA experience produce efficient GPU code, often close to what an expert would write.

ERNIE 3.0 is a 10-billion-parameter pre-trained language model from 2021 that trains on plain text plus a large-scale knowledge graph and fuses auto-encoding and auto-regressive networks, so one model handles both language understanding and generation.

AMD's data-center GPU accelerator launched on 8 November 2021, built on the CDNA 2 architecture with 220 compute units, 128 GB of HBM2e memory, and strong double-precision throughput alongside low-precision AI formats.

NVIDIA's A800 is a data-center GPU that went into production in 2022 as a China-market alternative to the A100, with a reduced chip-to-chip transfer rate to meet US export-control thresholds.

A 2022 Nature paper by Degrave, Felici and colleagues showing that a deep reinforcement learning controller can learn on its own to command all of a tokamak's magnetic control coils, producing and holding a range of plasma shapes.

PaLM is a 540-billion-parameter dense Transformer language model announced by Google Research in April 2022, trained with the Pathways system across 6,144 TPU v4 chips; it set strong few-shot results on many benchmarks.

Biren's general-purpose GPU, released in August 2022, packages two dies on a 7 nm process with 77 billion transistors; the company claimed PFLOPS-level single-chip compute, a first for a Chinese firm.

Whisper is OpenAI's open-source speech recognition system, released in September 2022 and trained on 680,000 hours of audio, which transcribes speech in many languages and translates it into English.

A 2022 paper by William Peebles and Saining Xie that replaces the U-Net backbone of diffusion models with a transformer operating on latent patches, showing that image quality improves predictably as compute grows and reaching a state-of-the-art FID of 2.27 on ImageNet 256×256.

GraphCast is a graph neural network trained on four decades of reanalysis data that produces 10-day global weather forecasts at 0.25 degree resolution in under a minute, outperforming the leading operational deterministic systems on most verification targets.

NVIDIA's China-specific data-center GPU, built to stay inside US export controls. Cut down from the H100 in core count and performance, it was still the most advanced NVIDIA chip legally sold in China until April 2025.

Cambricon's MLU 590 is a 7 nm cloud AI chip released in 2023 for both training and inference on models with hundreds of billions of parameters, and the centrepiece of the company's fifth-generation MLUarch line.

Skild AI is a robotics company founded in 2023 that builds a general-purpose robotic foundation model, intended to act as a single brain across many robot bodies and tasks such as manipulation, locomotion and navigation.

Huawei's Ascend 910B is a 2023 volume-production AI chip rated at 320 TFLOPS of FP16, with 32 GB of HBM2e memory, 1.6 TB/s of bandwidth and a 310 W envelope on a 7-nanometer domestic process.

The Segment Anything Model (SAM) is a promptable image segmentation model Meta AI released in April 2023. Given a point, a box or a mask, it outlines the object the prompt refers to.

PaLM 2 is Google's second-generation Pathways language model, announced on 10 May 2023 in four sizes and used across more than 25 Google products, including Bard.

Pangu-Weather is a three-dimensional deep-learning model for medium-range global weather forecasting, published in Nature in 2023; on reanalysis data it produced stronger forecasts than ECMWF's operational system across all tested variables.

FlashAttention-2 is a 2023 revision of the FlashAttention attention kernel that reorganizes work across GPU thread blocks and warps, roughly doubling speed and reaching up to 73% of the A100's theoretical peak throughput.

Mistral AI's first model: a 7.3-billion-parameter language model released in September 2023 under the Apache 2.0 license, built to match much larger models on standard benchmarks.

Grok-1 is xAI's first large language model, announced on 3 November 2023 as the engine of the Grok chatbot. Its 314-billion-parameter weights were released openly in March 2024.

A 2023 Nature paper from researchers at Google DeepMind presenting a scaled-up deep-learning approach to predicting stable crystal structures, screening 2.2 million candidates and reporting a large jump in stability-prediction accuracy.

Google TPU v5p is a 2023 TPU generation with 459 teraflops of peak BF16 and FP8 compute per chip, 95 GiB of HBM, and 8,960 chips per pod.

MLX is an open-source machine learning framework from Apple's machine learning research team, released in December 2023 and built primarily for Apple silicon. Its NumPy-like Python API and unified-memory design let the CPU and GPU work on the same data.

FunSearch is a method that uses a pretrained large language model inside an evolutionary loop to search for programs that solve mathematical and computational problems, rather than for answers directly. It produced new results on the cap set problem and on online bin packing.

The Kunlunxin P800 is a Chinese AI accelerator launched in 2024, the third generation of Kunlunxin's line, built on its own XPU-P architecture and used in large training clusters and in customer deployments.

Intel Gaudi 3 is an AI accelerator introduced in 2024, built from two 5 nm compute dies with 128 GB of HBM2e and 1,835 TFLOPS of BF16 and FP8 matrix throughput, plus integrated 200GbE networking on its OAM module.

Gemma is a family of lightweight open language models from Google DeepMind, released in February 2024 in 2B and 7B sizes, built from the same research and technology as Gemini and offered for commercial use under Google's terms.

Stable Diffusion 3, announced by Stability AI on 22 February 2024, is a suite of 800 million to 8 billion parameter models that pairs a diffusion transformer with flow matching.

Meta's second-generation in-house AI chip, published on 10 April 2024, built on TSMC's 5-nanometer process and aimed at the ranking and recommendation models Meta runs in its own data centers.

Llama 3 is an open model family from Meta, released on 18 April 2024 at 8 billion and 70 billion parameters. It was pretrained on over 15 trillion tokens of publicly available data.

Tri Dao and Albert Gu's paper introduces state space duality, a framework showing that structured state space models and attention variants are closely related, and uses it to derive Mamba-2, whose core layer runs 2 to 8 times faster than Mamba's.

OpenVLA is a 7-billion-parameter open-source vision-language-action model trained on 970,000 real robot demonstrations, released with checkpoints and code so that other groups can fine-tune and run it on their own robots.

FlashAttention-3 is an attention algorithm for NVIDIA Hopper (H100) GPUs that reaches up to 75% utilization in FP16 and close to 1.2 PFLOPs/s in FP8, using asynchronous execution and low-precision techniques.

SAM 2 is a Meta FAIR foundation model that segments objects in images and videos from simple prompts such as points, boxes or masks, released together with a large video segmentation dataset and training code.

Google TPU v6e, also called Trillium, is a Google Cloud AI accelerator whose published specifications cover per-chip BF16 and FP8 compute, high-bandwidth HBM, and pod-scale interconnects for large training jobs.

Hygon's Shengsuan No. 3 is a GPGPU-architecture deep computing processor that reached the market in 2025, built on a 7 nm chiplet design with 1.8 TB/s of HBM bandwidth and a CUDA-like software stack.

Moore Threads' flagship training-and-inference GPU card, built on the PH100 chip and the Pinghu architecture, rated at 1,000 TFLOPS of dense AI compute with 80 GB of memory, and the company's main shipping product in 2025.

A paper showing that a 32B model fine-tuned on 1,000 carefully chosen questions and run with a simple "budget forcing" trick can beat o1-preview on competition maths by up to 27%, with model, data and code released.

Helix is Figure's vision-language-action model for humanoid control, pairing a 7-billion-parameter vision-language model with a fast 80-million-parameter action policy, trained on about 500 hours of teleoperated data and run entirely on onboard GPUs.

π0.5 is a vision-language-action model from Physical Intelligence, released in April 2025, built to generalize to unfamiliar homes and trained by co-training on many kinds of robot and web data.

Falcon-H1 is a family of open-weight language models that runs Transformer attention and state space layers in parallel, released in sizes from 0.5B to 34B parameters under the Falcon-LLM Licence.

A 2025 paper proposing Absolute Zero, a reinforcement-learning paradigm in which one model writes its own code-reasoning tasks and then solves them, with a code executor as the only judge, reaching state of the art on coding and maths without external training data.

Aurora is a foundation model for the Earth system, trained on more than a million hours of geophysical data. It outperforms operational forecasts for weather, air quality, ocean waves and tropical cyclone tracks at far lower computational cost.

Huawei's Ascend 970 is a planned AI chip announced in September 2025 as the last entry on the company's published roadmap, targeted for the fourth quarter of 2028 with roughly double the compute and wider interconnect and memory bandwidth than the Ascend 960.

The Ascend 960 is a Huawei AI chip on the company's three-year roadmap, slated for the fourth quarter of 2027 with twice the compute, memory and interconnect ports of the Ascend 950 and support for Huawei's HiF4 4-bit format.

Cambricon's flagship MLU 690, in mass production since early 2026, is a dual-die accelerator with 196 GB of HBM3 memory and more than 700 TFLOPS of FP16 compute, aimed at very large-scale training workloads.

Zhenwu 810E is an AI processor designed by Alibaba's chip team T-Head, listed in January 2026 with 96 GB of high-bandwidth memory, aimed at training, inference and autonomous driving, and already deployed in large clusters on Alibaba Cloud.

Jakob Uszkoreit is one of the eight authors of “Attention Is All You Need” and the founder of Inceptive, a company using AI to design biological molecules.[^1][^4]

Sim-to-real transfer is the practice of training robot policies in simulation and then adapting them to physical robots, using synthetic data to supplement costly real-world demonstrations.

A loss function turns the cost of a prediction into a single number, and training a model means making that number as small as possible. Its choice decides which mistakes a model is punished for.

Overfitting is when a model fits its training data so closely that it performs poorly on new data. It is one half of the bias–variance trade-off and is usually spotted by comparing training and validation performance.

Regularization is the practice of making a problem's answer simpler, usually by adding a penalty term to the objective, so that ill-posed problems become solvable and models avoid overfitting.

A function applied to each node's weighted sum, giving neural networks their non-linearity. Sigmoid, tanh and ReLU shaped early deep learning; GELU now dominates transformers.

Transformers is an open-source Python library that offers pretrained Transformer models through one unified API. Its paper was posted to arXiv on 9 October 2019, and the first v5 release candidate came in December 2025.

Chris Olah is a Canadian machine learning researcher and co-founder of Anthropic, known for work on mechanistic interpretability, the effort to understand the internal workings of neural networks.

Embodied AI is the study of agents that learn by interacting with an environment, using egocentric perception, rather than only from datasets curated from the internet. Simulators serve as virtual testbeds where such agents are trained and evaluated.

Mechanistic interpretability explains model behaviour by finding the internal computations and features that produce it. Work on superposition and sparse autoencoders shows how many features can be packed into few neurons.

Reward hacking is what happens when a system optimises a proxy reward that only imperfectly captures what we want: it scores well on the proxy while performing badly on the true objective.

A KV cache keeps the key and value vectors a model has already computed for earlier tokens, so each new token reuses them instead of recomputing. Its memory footprint is what makes high-throughput serving hard to scale.

Long context is a model's capacity to take in and work over very large inputs — documents, images, recordings — in a single request. Systems now reach millions of tokens, though finding information inside that window is a separate skill.

A context window is the full span of text a model can refer to while producing a response, measured in tokens and including the response itself. Its size, and how evenly a model uses it, shape what the model can do.

AWS Trainium is a chip built by Amazon Web Services for deep learning training, sold through EC2 Trn1 instances that AWS says cut training costs by up to 50% compared with comparable instances.

AWS Inferentia2 is Amazon Web Services' second-generation chip built for inference, offering 32 GB of high-bandwidth memory, up to 190 TFLOPS in FP16 and up to 12 chips per EC2 Inf2 instance.

Google TPU v5e is a 2023 machine-learning accelerator chip with one TensorCore per chip, 197 TFLOP of bfloat16 peak compute, and 256-chip pods, positioned for both training and serving.

The GH200 Grace Hopper Superchip is an NVIDIA module that puts a 72-core Arm CPU and a Hopper GPU in one package joined by NVLink-C2C, pairing fast HBM with up to 480 GB of CPU memory for AI and technical computing.

SK hynix is a Korea-headquartered supplier of DRAM and NAND flash memory, and the company that released the first high-bandwidth memory (HBM) and says it has developed and shipped every HBM generation since.

Andrew G. Barto is a computer scientist who helped establish reinforcement learning as a general framework and shared the 2024 ACM A.M. Turing Award with Richard Sutton.

Richard Sutton is a computer scientist at the University of Alberta known for his work on reinforcement learning; he co-authored the field's standard textbook and shared the 2024 ACM A.M. Turing Award with Andrew Barto.

Baidu Research is the research arm of Baidu, built around the Institute of Deep Learning founded in 2013, with laboratories in Beijing and Silicon Valley; in 2014 Andrew Ng became chief scientist in charge of the whole organisation.

Koray Kavukcuoglu is a DeepMind computer scientist who co-authored the first deep Q-network paper and later, as Google DeepMind's CTO and Google's chief AI architect, fronted successive Gemini model launches.

Durk Kingma is a machine learning researcher and co-author of the variational autoencoder and the Adam optimizer; he has worked at OpenAI, Google Brain / DeepMind and Anthropic.

Ian Goodfellow is a machine learning researcher and first author of the 2014 paper that introduced generative adversarial networks, which train a generator and a discriminator against each other. He has worked at OpenAI, Google Brain, Apple and Google DeepMind.

Jensen Huang is a founder of NVIDIA and has served as its president, CEO and board member since the company was founded in 1993. NVIDIA credits the company's invention of the GPU and its work in accelerated computing to that era.

Quoc Le is a machine-learning researcher whose 2013 Stanford Ph.D. thesis was on scalable feature learning. He co-authored papers on large-scale unsupervised learning, sequence-to-sequence models, neural architecture search and EfficientNet; the sequence-to-sequence paper won a NeurIPS 2024 Test of Time award.

Niki Parmar is one of the eight authors of "Attention Is All You Need", the 2017 paper that introduced the Transformer architecture, and is named in Google's own account of that work.

Llion Jones is one of the eight authors of the 2017 paper "Attention Is All You Need" and a co-founder and CTO of Sakana AI.

Illia Polosukhin is one of the eight authors of the 2017 Transformer paper and a co-founder of NEAR Protocol. A 2024 NEAR post identifies him as CEO of the NEAR Foundation.[^1][^3][^6]

Christopher Manning is a Stanford professor of linguistics and computer science, former SAIL director, and former ACL president, recognized with fellowships and academy memberships.

Pieter Abbeel is a UC Berkeley professor who directs the Berkeley Robot Learning Lab and co-directs BAIR; his honors include the ACM Prize in Computing and IEEE Fellow status.

Noam Shazeer is an AI researcher who co-wrote the Transformer paper and was first author of the paper on sparsely gated mixture-of-experts layers. He co-founded Character.AI, returned to Google in 2024, and announced a move to OpenAI in 2026.

Ashish Vaswani is a computer scientist best known as the first-listed author of “Attention Is All You Need,” the 2017 paper that introduced the Transformer architecture. He also co-founded the AI companies Adept and Essential AI.

Kaiming He is a computer-vision and deep-learning researcher, first author of the 2015 ResNet paper that introduced residual connections, now a tenured associate professor at MIT.

Aidan Gomez is a Canadian computer scientist, co-founder and CEO of Cohere, and one of the eight authors of the 2017 paper that introduced the Transformer architecture.

Sergey Levine is a computer scientist who studies how robots can learn control behaviour from data. He is a professor at UC Berkeley and a co-founder of Physical Intelligence.

University College London (UCL) is a public research university in central London. Geoffrey Hinton led its Gatsby Computational Neuroscience Unit, and Demis Hassabis, a Gatsby postdoctoral researcher, later co-founded DeepMind.

New York University, chartered in 1831 and centred on Washington Square Park, is a private research university whose Courant Institute and Center for Data Science made it a home for machine learning research and data science training.

A private research university in Ithaca, New York, founded in 1865. It is also where Frank Rosenblatt studied and worked, and where the perceptron — an early learning machine and the first neural network — was developed.

A public technical university in Munich, founded in 1868, where Sepp Hochreiter wrote the 1991 thesis that produced the first LSTM results, and where Jürgen Schmidhuber studied and later taught.

A French-language public research university in Montreal, founded in 1878 and autonomous since 1919, known today as a centre of deep-learning research and the home institution of Yoshua Bengio and Mila.

k-means clustering splits n observations into k groups by repeatedly sending each point to its nearest centroid and recomputing each centroid as its cluster's mean. It is simple and scales to large data, but k must be fixed in advance.

A public research university in La Jolla, San Diego, founded in 1960, where connectionist neural-network research flourished in the 1980s and where data science and computing were later organised into dedicated institutes and a school.

Tang Xiaoou (1968–2023) was a Chinese computer vision researcher at the Chinese University of Hong Kong and co-founder and chairman of SenseTime, one of China's best-known AI companies.

A model validation technique that repeatedly splits data into folds, trains on some parts and validates on others, then averages the results to estimate how well a model or statistical result will generalise to unseen data.

Spanish computer scientist (born 1983) who co-invented sequence-to-sequence learning, led DeepMind's AlphaStar research group, and became a technical lead on Gemini before leaving to start Discovery Loop.

A Swiss artificial intelligence institute founded in 1988 and now a joint institute of USI and SUPSI, known for LSTM and other neural-network research.

Canadian machine-learning researcher who was one of Geoffrey Hinton's earliest hires at Google Brain's Toronto office, co-founded the natural language processing company Cohere in Toronto in 2019, and is lead vocalist of the indie rock band Good Kid.

Microsoft Research Asia is Microsoft's research laboratory in the Asia-Pacific region, founded in Beijing in November 1998. It works on artificial intelligence, human-computer interaction and related fields, and its researchers have included the first author of the ResNet paper.

Graph neural networks are neural networks built for graph-structured data, in which nodes repeatedly exchange messages with their neighbours. Since the model was formulated in 2009, they have become a standard way of learning on graphs.

A regularisation technique for neural networks: during training it randomly zeroes the inputs or outputs of some neurons so that units cannot co-adapt, which reduces overfitting.

DNNresearch Inc. was a University of Toronto start-up founded in 2012 by Geoffrey Hinton and two of his graduate students; Google bought it for its deep neural network research.

Huawei's artificial intelligence research centre, founded in 2012 and part of the company's 2012 Laboratories, with sites in China, Hong Kong, Europe and Canada and research spanning vision, language, search and recommendation, and decision-making.

A 2013 paper by Goodfellow, Warde-Farley, Mirza, Courville and Bengio introduces maxout units, whose output is the maximum of a set of inputs, designed to work with dropout, and reports state-of-the-art results on four benchmarks.

A 2013 paper by Graves, Mohamed and Hinton that examines deep recurrent neural networks for speech recognition, arguing for end-to-end sequence training with methods such as connectionist temporal classification.

NVIDIA's Tesla K80 is a dual-GPU data-center accelerator announced in November 2014, putting two GK210 chips, 24 GB of memory and up to 8.74 teraflops of single-precision performance on one 300 W board.

A 2014 paper by Mehdi Mirza and Simon Osindero that adds conditioning data to both the generator and the discriminator, enabling a GAN to generate samples such as MNIST digits conditioned on class labels.

A 2015 arXiv paper by Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho and co-authors that introduced an image-captioning model with visual attention, able to show where it is looking while writing each word.

Apache MXNet is an open-source deep learning library that offers both symbolic and imperative programming, runs from mobile devices to GPU clusters, and ships bindings for eight languages. Its paper was posted in December 2015 by Tianqi Chen, Mu Li and colleagues.

Chinese medical-AI company founded in 2016 that focuses on AI for diagnosing and treating coronary heart disease, and says it holds China's first class III AI medical device certificate.

An independent, non-profit AI research institute in Toronto, launched in 2017 with provincial and corporate funding, with Geoffrey Hinton among its founding scientists.

Group Normalization is a 2018 paper by Yuxin Wu and Kaiming He that proposes normalizing channels in groups, independent of batch size, to address batch normalization's weakness when batches are small.

NVIDIA's DGX-2, announced in March 2018, packed 16 Tesla V100 GPUs and a new NVSwitch interconnect into one server that NVIDIA described as the first to reach two petaflops for deep learning.

ERNIE 2.0 is Baidu's pre-trained language model framework, published in 2019, which builds pre-training tasks incrementally through continual multi-task learning and reported results above BERT and XLNet on English and Chinese benchmarks.

An AI processor from Cerebras Systems that keeps an entire silicon wafer as one chip, holding more than 1.2 trillion transistors and 400,000 cores; the first of a line that reached WSE-3 in 2024.

A 2019 Google paper that frames every language task as text in, text out, using one encoder-decoder Transformer and one training setup. It also introduced the C4 dataset, and the released T5 models became common research starting points.

MuZero is a DeepMind reinforcement-learning algorithm that pairs tree-based search with a learned model. It reached superhuman play in Atari, Go, chess and shogi without being given the environment's rules.

Meena is a 2.6-billion-parameter open-domain chatbot from Google, trained end-to-end on filtered social media conversations. Its 2020 paper introduced the SSA human-evaluation metric and linked next-token perplexity to human-judged chat quality. The work was later renamed LaMDA.

MindSpore is Huawei's open-source AI computing framework, announced in March 2020, built to run across device, edge and cloud, co-optimized with Huawei's Ascend processors and shipped with tools for training visualization and model security.

Megatron-Turing NLG 530B is a 530-billion-parameter language model announced by NVIDIA and Microsoft on 11 October 2021, then the largest monolithic transformer language model, trained on NVIDIA's Selene supercomputer with Megatron-LM and DeepSpeed combined.

Gopher is DeepMind's 280-billion-parameter language model, announced in December 2021 alongside a study of what scale does to language models and a companion taxonomy of their risks.

AlphaCode is a DeepMind code-generation system for competitive programming, announced in February 2022. On Codeforces contests held after its training data, it reached an estimated rank around the median participant. A Gemini-based AlphaCode 2 followed in December 2023.

OPT-175B is a 175-billion-parameter language model from Meta AI, announced in May 2022. Meta said it was the first model of its size released with both pretrained weights and the code to train and use them.

Speculative decoding speeds up text generation from large autoregressive models: a smaller model proposes several tokens and the large model checks them in parallel, producing identical output faster.

Stable LM is Stability AI's family of open language models, first released in April 2023 as 3B- and 7B-parameter alpha versions under licenses that allow commercial use of the base models and research-only use of the instruction-tuned ones.

MTIA v1 is Meta's first-generation in-house AI inference accelerator, built on TSMC's 7 nm process with 128 MB of on-chip memory and designed for deep learning recommendation models.

RT-2 is a vision-language-action model from Google DeepMind, announced on 28 July 2023, that turns images and text instructions into robot actions written as text tokens. In over 6,000 trials it reached 62% on unseen scenarios, up from 32% for RT-1.

GraphCast is a Google DeepMind machine-learning model that makes global weather forecasts up to 10 days ahead, producing a 10-day forecast in under a minute on a single TPU and outperforming ECMWF's HRES on most tested variables.

GNoME is a Google DeepMind deep-learning tool, announced on 29 November 2023, that predicts new inorganic crystals; it produced 2.2 million candidate structures, of which 380,000 were predicted to be stable.

DeepSeekMoE is a mixture-of-experts architecture from DeepSeek, released in January 2024, that splits experts finely and keeps a few shared experts always active so that each expert holds focused knowledge. Its 16B open-weights model matched larger dense models at a fraction of the computation.

Octo is an open-source generalist robot policy: a transformer-based diffusion model trained on 800,000 robot trajectories that can drive a range of robots out of the box and be fine-tuned to new setups.

A family of text-to-image models from Stability AI, announced in October 2024: the 8.1-billion-parameter Large, a 4-step Turbo variant and the 2.5-billion-parameter Medium, released under a community licence.

Skild Brain is Skild AI's general-purpose foundation model for robots, trained in a universe of 100,000 different machines and meant to be fitted to existing hardware rather than built for one task.

Qwen3-Next is an open-weight model from Alibaba's Qwen team, released in September 2025 under Apache 2.0. It has 80B total parameters, only 3B active per token, and a hybrid attention design later used in Qwen3.5.

Ai2's fully open 7B model, which mixes Transformer attention with Gated DeltaNet linear recurrent layers and matches its predecessor's MMLU accuracy using 49% fewer training tokens.

Intel Gaudi 2 is a data-centre AI accelerator with 96 GB of HBM2e memory, 432 TFLOPS of BF16 compute and 24 tensor processor cores, built to run the matrix arithmetic behind large neural networks.

Karol Hausman is a roboticist working on robot learning, sim-to-real transfer and embodied AI. He was a senior research scientist at Google Brain and Google DeepMind, is an adjunct professor at Stanford, and co-founded Physical Intelligence.

The AMD Instinct MI100 is a GPU accelerator built on AMD's first-generation CDNA architecture. It pairs 32 GB of HBM2 memory with peak throughput ratings for double-, single-, and half-precision math.

Shun-ichi Amari (born 1936) is a Japanese mathematical engineer who founded information geometry and built theoretical foundations for machine learning and neural networks; he received the 2025 Kyoto Prize in Advanced Technology.

Cerebras's second-generation wafer-scale chip, presented as the largest AI processor ever made: 2.6 trillion transistors, 850,000 cores and 40 GB of on-wafer memory on a single wafer built on a 7-nanometer process.

An autoencoder is an unsupervised neural network that compresses its input into a low-dimensional code and reconstructs it back, learning a latent representation useful for dimensionality reduction, denoising and anomaly detection.

AWS Inferentia is a custom chip designed by AWS for running trained machine-learning models, sold to customers through EC2 Inf1 instances. It aims to make inference cheaper and faster than general-purpose instances.

Jacob Devlin is a research scientist and first author of BERT, a language model Google introduced in 2018 that reads text in both directions. He earlier worked on neural machine translation at Microsoft Research.

The NVIDIA L40S is a data-centre GPU on Nvidia's Ada Lovelace architecture, pairing 48 GB of ECC memory with AI and graphics throughput in a passive, dual-slot PCIe Gen4 card that omits NVLink and MIG.

The NVIDIA Tesla T4 is a 70-watt, single-slot GPU built on the Turing architecture and aimed at AI inference, pairing Tensor Cores with dedicated video transcoding engines in a card that slots into ordinary servers.

The NVIDIA A10 is a data-center GPU built on the Ampere architecture that combines graphics, video and AI inference in a single-slot PCIe card with 24 GB of memory, aimed at mainstream enterprise servers.

Thomas Wolf is a co-founder and chief science officer of Hugging Face, and the first author of the 2019 paper that introduced the company's open-source Transformers library to researchers and practitioners.

Shane Legg is a New Zealand AI researcher and co-founder of DeepMind, known for helping popularise the term "artificial general intelligence" and for leading the company's AGI safety work.

Sun Jian was a computer-vision scientist who spent 13 years at Microsoft, co-created ResNet, and later became chief scientist at Megvii. He died aged 45 after a sudden illness in June 2022.

Decision tree learning builds a classification or regression tree from data by repeatedly splitting it into more homogeneous groups. It is a widely used supervised method whose rules can be read directly, and whose main weaknesses are instability and a tendency to overfit.

Feature engineering is the preprocessing step that turns raw data into more useful inputs for machine learning and statistical models, shaping, cleaning and trimming the attributes a model learns from.

The NVIDIA Tesla P40 is a data-center GPU accelerator described in a September 2016 NVIDIA datasheet. Built on the Pascal architecture and aimed at deep learning inference, it pairs 24 GB of memory with 47 TOPS of INT8 throughput in a 250 W PCIe card.

A data-center GPU on NVIDIA's Ampere architecture, aimed at AI inference and mainstream enterprise workloads, with 24 GB of HBM2 memory, a 165 W power ceiling and the ability to split into as many as four MIG instances.