2017–2022 · 154 key events · 134 related people, organisations and concepts
Large models
The Transformer and pre-training at scale lead to large language models.
2017
SoftBank agrees to acquire Boston Dynamics from Alphabet
On 9 June 2017 SoftBank Group announced that a subsidiary had agreed to buy Boston Dynamics from Alphabet, together with the Japanese bipedal-robotics company Schaft. The terms were not disclosed.
“Attention Is All You Need” is posted to arXiv, introducing the Transformer
On 12 June 2017 a team of researchers, most of them at Google, posted the paper Attention Is All You Need to arXiv. It introduced the Transformer, a neural network built only on attention, which became the basis for most later large language models.
Attention Is All You Need
A 2017 paper that introduced the Transformer, a neural network built entirely on attention with no recurrence. It set new machine translation results and became the architecture behind almost every large language model since.
Deep reinforcement learning from human preferences
A 2017 paper showing that an AI agent can learn a task from short human comparisons instead of a hand-written reward. The method, reinforcement learning from human feedback, later became central to training chat assistants.
Proximal Policy Optimization Algorithms
A 2017 paper by Schulman, Wolski, Dhariwal, Radford and Klimov that introduced proximal policy optimization (PPO), a reinforcement learning method that is much simpler to implement and tune and became OpenAI's default RL algorithm.
Focal Loss for Dense Object Detection
A 2017 paper by Lin, Goyal, Girshick, He and Dollar that argues dense one-stage detectors are held back by extreme foreground-background imbalance and proposes focal loss, a reshaped cross-entropy that down-weights well-classified examples.
AlphaGo Zero learns Go from scratch
In 2017, DeepMind announced AlphaGo Zero, a Go system that started from random play, learned only by self-play, and outperformed earlier AlphaGo versions with one neural network and far less hardware.
Mastering the game of Go without human knowledge
AlphaGo Zero learned Go only by playing against itself, starting from random play, and beat the earlier champion-defeating AlphaGo 100 games to 0 after three days of training. The Nature paper appeared in October 2017.
mixup: Beyond Empirical Risk Minimization
mixup is a training principle from a 2017 paper by Zhang, Cisse, Dauphin and Lopez-Paz that trains networks on convex combinations of pairs of examples and their labels, improving generalisation on image, speech and tabular benchmarks.
Progressive Growing of GANs for Improved Quality, Stability, and Variation
A 2017 paper by Karras, Aila, Laine and Lehtinen that trains generative adversarial networks by growing the generator and discriminator from low to high resolution, improving stability, quality and variation.
Neural Discrete Representation Learning
A 2017 paper by van den Oord, Vinyals and Kavukcuoglu proposing VQ-VAE, a generative model whose latent code is discrete rather than continuous, and whose prior is learned rather than fixed, so that useful representations can be learned without supervision.
DeepMind's AlphaZero masters chess, shogi and Go
A preprint posted to arXiv by DeepMind on 5 December 2017 described one general reinforcement-learning algorithm that, given only the rules, learned chess, shogi and Go by self-play and reached superhuman level in all three within 24 hours.
A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play
A 2017 arXiv paper, later published in Science, reporting that a single self-play reinforcement learning algorithm, AlphaZero, mastered chess, shogi and Go and beat strong specialised programs in each.
2018
Machine reading comprehension surpasses human baseline on SQuAD
In January 2018, Microsoft Research Asia's R-NET and then an Alibaba model scored above the human baseline on SQuAD, a machine reading comprehension benchmark — the first time a system had exceeded human exact-match performance there.
Universal Language Model Fine-tuning for Text Classification
ULMFiT is a 2018 transfer-learning method by Jeremy Howard and Sebastian Ruder for fine-tuning language models across NLP tasks; it beat the state of the art on six text-classification tasks and performed well with few labels.
Deep contextualized word representations
A 2018 paper by Matthew E. Peters and five co-authors that represents each word with the internal states of a pre-trained deep bidirectional language model, so vectors change with context; adding them to existing models improved results on six NLP tasks.
Uber self-driving test car kills a pedestrian in Tempe
On 18 March 2018 a prototype Uber self-driving car based on a Volvo XC90 struck and killed Elaine Herzberg, 49, as she pushed a bicycle across a road in Tempe, Arizona. It is recorded as the first pedestrian fatality involving a self-driving car.
Group Normalization
Group Normalization is a 2018 paper by Yuxin Wu and Kaiming He that proposes normalizing channels in groups, independent of batch size, to address batch normalization's weakness when batches are small.
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
GLUE is a benchmark and analysis platform that evaluates natural language understanding models across a diverse set of existing tasks, with an online comparison platform and a hand-crafted diagnostic suite for linguistic analysis.
The European Commission sets out its approach to artificial intelligence
A European Commission communication adopted on 25 April 2018 that set out three strands of EU policy on artificial intelligence — industrial capacity, preparing people for economic change, and an ethical and legal framework — alongside an investment target of at least EUR 20 billion.
Google announces Duplex at I/O 2018
On 8 May 2018, at its I/O developer conference, Google announced Duplex, a system that holds natural-sounding phone conversations to complete narrow tasks such as scheduling certain appointments.
Google publishes its AI principles
On 7 June 2018 Google published a set of seven objectives to guide its AI research and products, together with four kinds of application it said it would not pursue.
Improving Language Understanding by Generative Pre-Training (GPT)
OpenAI's 2018 paper introducing GPT, a Transformer decoder pre-trained to predict the next token and then fine-tuned for specific tasks. It improved the state of the art on 9 of 12 tasks studied and set the recipe the GPT series followed.
OpenAI publishes the first GPT paper on generative pre-training
On 11 June 2018 OpenAI published a paper on generative pre-training: a language model first learns from unlabeled text and is then adapted to specific tasks. The approach improved the state of the art on 9 of the 12 tasks studied.
Neural Ordinary Differential Equations
A 2018 paper by Chen, Rubanova, Bettencourt and Duvenaud that replaces a fixed sequence of hidden layers with a neural network describing the derivative of the hidden state, so outputs are computed by an ODE solver — yielding continuous-depth models with constant memory cost.[^1]
Glow: Generative Flow with Invertible 1x1 Convolutions
A 2018 paper by Diederik P. Kingma and Prafulla Dhariwal presenting Glow, a flow-based generative model built on invertible 1x1 convolutions, which improved log-likelihood on standard benchmarks and could synthesise and manipulate large images.
First World Artificial Intelligence Conference opens in Shanghai
The first World Artificial Intelligence Conference opened in Shanghai on September 17, 2018, bringing together experts, entrepreneurs and more than 200 leading AI companies from nearly 40 countries and regions, and it has been held in the city every year since.
Large Scale GAN Training for High Fidelity Natural Image Synthesis
A 2018 paper by Andrew Brock, Jeff Donahue and Karen Simonyan that trained generative adversarial networks at the largest scale attempted at the time, introduced BigGANs and set a new state of the art in class-conditional image synthesis.
Google researchers publish BERT on arXiv
On 11 October 2018 Google researchers posted the BERT paper to arXiv. BERT uses context from both directions in every layer, and a single added output layer is enough to adapt it to tasks such as question answering.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Google's 2018 BERT paper introduced an encoder-only Transformer that reads text in both directions and is pre-trained by hiding words. It set new results on eleven language tasks and made download-and-fine-tune the default workflow.
NIPS changes its abbreviation to NeurIPS
In 2018 the Conference on Neural Information Processing Systems changed its long-standing abbreviation from NIPS to NeurIPS, after critics argued the old acronym encouraged sexism and read as a slur against Japanese.
AlphaFold tops the CASP13 protein-structure assessment
In December 2018 DeepMind's AlphaFold placed first overall at CASP13, a blind protein-structure prediction assessment, giving the best prediction for 25 of the 43 hardest targets.
A Style-Based Generator Architecture for Generative Adversarial Networks
This 2018 paper introduces a style-based GAN generator that separates high-level attributes from stochastic variation and allows scale-specific control, improving distribution quality, interpolation, and disentanglement.
2019
The United States issues the executive order creating the American AI Initiative
On 11 February 2019 the US president signed an executive order creating the American AI Initiative, a coordinated federal strategy that made AI a budget priority and set agencies deadlines on data access, standards and regulation.
Language Models are Unsupervised Multitask Learners (GPT-2)
OpenAI's 2019 paper on GPT-2, a 1.5-billion-parameter Transformer trained on web text. It showed that a model trained only to predict the next word could perform several tasks when prompted, and its staged release sparked public debate.
OpenAI announces GPT-2 and holds back the full model
On 14 February 2019 OpenAI announced GPT-2, a 1.5-billion-parameter Transformer that performed several language tasks without task-specific training. It did not release the full model at first, instead releasing it in stages.
Baidu releases ERNIE, a knowledge-enhanced Chinese language model
In March 2019 Baidu released ERNIE, a Chinese language model that models entities and phrases directly, and reported that it outperformed Google's BERT on several Chinese language tasks.
OpenAI creates the capped-profit OpenAI LP
On 11 March 2019 OpenAI announced OpenAI LP, a "capped-profit" company that let it raise outside capital while limiting investor returns. The structure lasted until October 2025.
Bengio, Hinton and LeCun receive the 2018 ACM Turing Award
In March 2019 ACM named Yoshua Bengio, Geoffrey Hinton and Yann LeCun the recipients of the 2018 A.M. Turing Award, citing breakthroughs that made deep neural networks a critical component of computing.
The EU High-Level Expert Group on AI presents its Ethics Guidelines for Trustworthy AI
In April 2019 the EU High-Level Expert Group on AI published ethics guidelines that define trustworthy AI as lawful, ethical and robust, and set out seven requirements for AI systems, later turned into a piloted assessment list.
OpenAI Five beats Dota 2 world champions OG
On 13 April 2019, OpenAI Five — a team of reinforcement-learning bots — beat OG, the reigning Dota 2 world champions, in a best-of-three in San Francisco, and days later won 99.4% of games in a public online arena.
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
SuperGLUE is a benchmark for general-purpose language understanding, released in 2019 as a harder successor to GLUE. It bundles eight tasks, a single-number metric, an analysis toolkit and a public leaderboard.
HellaSwag: Can a Machine Really Finish Your Sentence?
HellaSwag is a 2019 benchmark in which a machine must pick the most plausible continuation of a short event description. Its wrong answers are generated and filtered to fool models rather than people.
2018 Turing Award ceremony in San Francisco
The 2018 ACM A.M. Turing Award honored Bengio, Hinton and LeCun for foundational and engineering work in deep learning; they formally received the prize on 15 June 2019 in San Francisco.
XLNet: Generalized Autoregressive Pretraining for Language Understanding
XLNet is a generalized autoregressive pretraining method from a 2019 paper. It learns bidirectional contexts by maximizing expected likelihood over all factorization-order permutations and outperforms BERT on 20 tasks under comparable settings.
Microsoft invests $1 billion in OpenAI and becomes its cloud partner
In July 2019 Microsoft invested $1 billion in OpenAI and became its exclusive computing partner, with OpenAI's services to run on Microsoft Azure.
RoBERTa: A Robustly Optimized BERT Pretraining Approach
A replication study of BERT pretraining that measures how much key hyperparameters and training data size matter, finds BERT was significantly undertrained, and shows a properly trained BERT matches or beats later models, reaching state of the art on GLUE, RACE and SQuAD.
Baidu releases ERNIE 2.0, beating BERT and XLNet on 16 tasks
In July 2019 Baidu released ERNIE 2.0, a continual pre-training framework that adds training tasks incrementally and reported beating BERT and XLNet on 16 tasks. The work was published at AAAI 2020.
Spot becomes commercially available
In September 2019 Boston Dynamics made its quadruped robot Spot commercially available, first through an Early Adopter Program for companies that could put it to practical use and build custom payloads; broader sales followed in 2020.
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
An NVIDIA paper from 2019 presenting an intra-layer model parallelism method, implemented with a few communication operations in native PyTorch, that trained transformer models of up to 8.3 billion parameters and improved several benchmarks.
US adds 28 Chinese entities including AI firms to the Entity List
On 7 October 2019 the US Commerce Department added 28 Chinese organisations, including facial-recognition and AI firms such as Hikvision, Dahua, iFlytek, Megvii and SenseTime, to its Entity List, restricting their ability to buy US products.
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5)
A 2019 Google paper that frames every language task as text in, text out, using one encoder-decoder Transformer and one training setup. It also introduced the C4 dataset, and the released T5 models became common research starting points.
C4: Colossal Clean Crawled Corpus (T5 paper)
A 2019 paper that casts every language task as text-to-text, systematically compares transfer-learning choices, releases the C4 corpus and models, and reports state-of-the-art results on summarization, question answering, and classification.
AlphaStar reaches Grandmaster level in StarCraft II
On 30 October 2019, DeepMind announced that AlphaStar reached Grandmaster level in StarCraft II for all three races, ranking above 99.8% of active players and playing under human-like interface limits.[^1]
OpenAI releases the full 1.5-billion-parameter GPT-2
On 5 November 2019 OpenAI released the full 1.5-billion-parameter GPT-2, with code and model weights, ending its staged release. The model card warned that the model cannot tell fact from fiction.
On the Measure of Intelligence (ARC-AGI)
A 2019 paper by Francois Chollet that defines intelligence as skill-acquisition efficiency and introduces the Abstraction and Reasoning Corpus (ARC), a benchmark of grid tasks that assumes no specialised knowledge or language.
DeepMind describes MuZero, which plans without knowing the rules
On 19 November 2019, DeepMind posted a MuZero preprint to arXiv: it plans with a learned model rather than rules, matched AlphaZero in Go, chess and shogi, and set a state of the art on 57 Atari games.
Mastering Atari, Go, chess and shogi by planning with a learned model
MuZero is a DeepMind reinforcement-learning algorithm that learns a model of its environment and plans inside that model, mastering Atari, Go, chess and shogi without being given the rules of those games.
2020
China grants its first class III certificate to an AI medical device
On 14 January 2020, Keya Medical's Shenmai FFR software became the first AI medical device to receive a class III certificate in China, opening a registration path for diagnostic software and later reaching billable clinical use.
Scaling Laws for Neural Language Models
A 2020 paper finding that language model loss falls predictably, as a power law, with model size, data and compute. It gave labs a quantitative reason to build much larger models and shaped the scale-up that followed.
Google describes the Meena chatbot
On 27 January 2020, Google researchers posted a paper describing Meena, a 2.6-billion-parameter open-domain chatbot that scored 79% on a new chat-quality metric called the Sensibleness and Specificity Average.
The European Commission publishes its White Paper on Artificial Intelligence
The European Commission's February 2020 white paper set out a policy to promote the uptake of AI while limiting the risks of certain uses, built on a risk-based approach, an ecosystem of excellence and an ecosystem of trust.
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
A 2020 paper by Mildenhall, Srinivasan, Tancik, Barron, Ramamoorthi and Ng that represents a scene as a neural radiance field: a network mapping 5D coordinates to density and colour, optimized from a sparse set of images with known camera poses.
ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
A 2020 paper by Clark, Luong, Le and Manning that pre-trains text encoders by detecting replaced tokens instead of filling in masked ones, matching much larger baselines with far less compute.
NVIDIA launches the A100 GPU
On 14 May 2020 NVIDIA announced the A100, its first GPU built on the Ampere architecture, claiming up to 20 times the AI performance of its predecessors. It shipped in full production alongside new systems, cloud partners and supercomputer customers.
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (RAG)
A 2020 paper that pairs a pre-trained seq2seq model with a dense Wikipedia index fetched by a neural retriever, setting state of the art on three open-domain QA tasks and producing more factual text than a parametric-only baseline.
OpenAI publishes the GPT-3 paper
In May 2020 OpenAI published the GPT-3 paper, describing a 175-billion-parameter language model that performs new tasks from a few examples given in its text input, without additional training.
Language Models are Few-Shot Learners (GPT-3)
The GPT-3 paper showed that a language model with 175 billion parameters could perform many tasks from a few examples in its prompt, with no fine-tuning, and that this ability grew with model size.
OpenAI launches its API for accessing its models
In June 2020 OpenAI announced an API that let outside developers access its AI models. By March 2021 more than 300 applications were using GPT-3 through it.
Denoising Diffusion Probabilistic Models
A 2020 paper by Jonathan Ho, Ajay Jain and Pieter Abbeel that trained a U-Net to reverse a gradual noising process, making diffusion models a practical rival to GANs and paving the way for DALL-E 2, Imagen and Stable Diffusion.
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
GShard, submitted to arXiv in June 2020, is a set of annotation APIs and an XLA compiler extension from Google that lets existing model code express parallel computation, scaling a sparsely gated mixture-of-experts translation model past 600 billion parameters.
Measuring Massive Multitask Language Understanding (MMLU)
A benchmark released in 2020 that tests language models across 57 subjects, from elementary mathematics to law, and showed that even the largest models of the time were far from expert-level.
NVIDIA agrees to acquire Arm from SoftBank for $40 billion
On 13 September 2020, NVIDIA agreed to buy Arm Limited from SoftBank and the SoftBank Vision Fund for $40 billion in shares and cash, with Arm to keep its open-licensing model and Cambridge headquarters, subject to regulatory approval.
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
A 2020 paper by Alexey Dosovitskiy and colleagues showing that a pure Transformer applied directly to sequences of image patches — the Vision Transformer, or ViT — matches state-of-the-art convolutional networks on image classification when pre-trained on large amounts of data.
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (ViT)
A 2020 paper showing that a plain Transformer applied directly to sequences of image patches, with no convolution at all, can match state-of-the-art convolutional networks on image classification.
AlphaFold2 achieves near-experimental accuracy at CASP14
On 30 November 2020 DeepMind announced that the organisers of CASP14 had recognised its AlphaFold system as a solution to the 50-year-old protein-folding problem, with accuracy approaching that of experimental methods.
Uber sells its self-driving unit ATG to Aurora
On 7 December 2020, Aurora announced it was acquiring Uber's self-driving unit, the Advanced Technologies Group, with Uber investing 400 million dollars in Aurora and its chief executive joining Aurora's board.
The Pile: An 800GB Dataset of Diverse Text for Language Modeling
The Pile is an 825 GiB English text dataset of 22 combined sources, released by EleutherAI on 31 December 2020 with an arXiv paper. It became the training data for several of the group's open models.
EleutherAI releases The Pile training dataset
On 31 December 2020 EleutherAI released The Pile, an openly available 825 GiB English training corpus assembled from 22 smaller datasets, with a paper showing models trained on it beat both comparison corpora on every component.
2021
Anthropic is founded
Anthropic, an AI company, was incorporated in January 2021 by former employees of OpenAI and set up as a public benefit corporation. Dario Amodei became its chief executive and Daniela Amodei its president.
OpenAI introduces DALL·E
On 5 January 2021 OpenAI announced DALL·E, a neural network that creates images from text captions, and CLIP, which learns visual concepts from natural-language supervision.
Learning Transferable Visual Models From Natural Language Supervision (CLIP)
CLIP is an OpenAI model trained on 400 million image-text pairs to map pictures and sentences into one shared space, so it can classify images zero-shot from a written description.
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
A 2021 paper by Fedus, Zoph and Shazeer that simplifies Mixture-of-Experts routing so that language models can hold up to a trillion parameters while keeping per-example computation roughly constant.
Zero-Shot Text-to-Image Generation (DALL·E)
A 2021 OpenAI paper describing a single 12-billion-parameter autoregressive transformer trained on 250 million image-text pairs, which generates images from text and, evaluated zero-shot, is competitive with domain-specific models.
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?
A 2021 FAccT paper by Bender, Gebru, McMillan-Major and Shmitchell that asks whether language models can be too large and sets out recommendations on costs, data, research goals and alternative directions.
Measuring Mathematical Problem Solving With the MATH Dataset
MATH is a dataset of 12,500 challenging competition mathematics problems, each with a full step-by-step solution. Released in 2021, it was built to measure and train mathematical reasoning in models.
EleutherAI releases GPT-Neo
On 21 March 2021 EleutherAI released GPT-Neo, a family of language models at 125M, 1.3B and 2.7B parameters whose weights anyone could download, in contrast to OpenAI's API-only GPT-3.
Boston Dynamics unveils Stretch
On 29 March 2021 Boston Dynamics announced Stretch, a mobile robot aimed at warehouse work such as truck unloading, built on a small omni-directional base so warehouses can adopt it without costly fixed automation infrastructure.
European Commission proposes the AI Act
In April 2021 the European Commission put forward its proposal for an Artificial Intelligence Act, submitted by Internal Market Commissioner Thierry Breton. It opened a negotiation in which the Council of the EU adopted its own position in December 2022.
Google unveils LaMDA at I/O
On 18 May 2021 Google announced LaMDA, a language model built for conversation, at its I/O keynote. It grew out of the Meena chatbot, was followed by LaMDA 2 and the AI Test Kitchen, and later powered Bard.
EleutherAI releases GPT-J-6B
GPT-J-6B is a six-billion-parameter language model released by EleutherAI in June 2021 with open weights under Apache 2.0, trained on The Pile and reported to be close to the 6.7B GPT-3 on zero-shot tasks.
Unitree launches Go1
In June 2021 Unitree released the Go1, a consumer quadruped robot priced from 16,000 yuan, weighing 12 kg and running at up to 4.7 metres per second; it sold nearly 1,000 units in its first year.
LoRA: Low-Rank Adaptation of Large Language Models
LoRA is a fine-tuning method that freezes a pre-trained model's weights and trains small low-rank matrices injected into each layer, cutting trainable parameters and memory by orders of magnitude.
Hyundai Motor Group completes acquisition of Boston Dynamics
In June 2021 Hyundai Motor Group completed its purchase of a controlling 80 percent stake in Boston Dynamics from SoftBank, in a deal that valued the robot maker at US$1.1 billion.
GitHub Copilot technical preview is announced
On 29 June 2021 GitHub announced GitHub Copilot as a technical preview, an AI pair programmer that suggests lines and whole functions as developers write code. It was powered by OpenAI Codex.
Baidu introduces the 10-billion-parameter ERNIE 3.0
On 5 July 2021 Baidu researchers posted the ERNIE 3.0 paper, describing a 10-billion-parameter model trained on plain text plus a knowledge graph that combines auto-encoding and auto-regressive networks, topping 54 Chinese tasks and SuperGLUE.
Evaluating Large Language Models Trained on Code (Codex / HumanEval)
A 2021 paper from OpenAI introduces Codex, a GPT model fine-tuned on public GitHub code, and HumanEval, an evaluation set that judges generated Python programs by whether they work rather than by how closely they match a reference.
AlphaFold2 structure-prediction research appears in Nature
In July 2021 the journal Nature published the AlphaFold2 paper from DeepMind. The AI system predicts a protein's three-dimensional shape from its amino-acid sequence, and DeepMind then released its code and a free structure database.
Highly accurate protein structure prediction with AlphaFold
The Nature paper described AlphaFold's methods, system design, and evaluation for protein-structure prediction.
Baker lab publishes RoseTTAFold and opens it to all
On 15 July 2021 the Institute for Protein Design released RoseTTAFold as code on GitHub and a public web server, with lead author Minkyung Baek describing structures computable in minutes on a gaming computer.
AI21 Labs launches AI21 Studio and the 178B-parameter Jurassic-1
In August 2021 AI21 Labs launched AI21 Studio and its Jurassic-1 language models. The largest, Jumbo, had 178 billion parameters, and developers could use the models through an API or a web playground.
OpenAI opens Codex to developers in its API
On 10 August 2021 OpenAI released an improved Codex, a GPT model fine-tuned on public code, through its API in private beta. The model powered GitHub Copilot and came with the new HumanEval benchmark.
Xiaomi unveils CyberDog
On 10 August 2021 Xiaomi unveiled CyberDog, a 14 kg quadruped robot sold as a 9,999-yuan engineering exploration edition limited to 1,000 units, with a first beta round for 400 users.
Program Synthesis with Large Language Models (MBPP)
A 2021 arXiv paper by Austin, Odena, Nye and eight other authors that introduced MBPP, a benchmark of 974 short Python programming tasks, and measured how language models from 244M to 137B parameters turn natural-language descriptions into working code.
Tesla announces its humanoid robot at AI Day
On 19 August 2021 Tesla used its first AI Day to unveil a concept for a humanoid robot meant to do dangerous, repetitive or boring work, alongside its pure-vision driving software, the D1 training chip and the Dojo supercomputer.
Finetuned Language Models Are Zero-Shot Learners
A 2021 paper showing that finetuning a language model on many tasks described by instructions — instruction tuning — makes it much better at unseen tasks without examples, surpassing a larger model on most benchmarks tested.
NVIDIA and Microsoft unveil the 530-billion-parameter Megatron-Turing NLG
On 11 October 2021 NVIDIA and Microsoft announced Megatron-Turing NLG 530B, a language model with 530 billion parameters that they called the largest and most powerful of its kind. The full technical report followed in January 2022.
Training Verifiers to Solve Math Word Problems (GSM8K)
GSM8K is an OpenAI benchmark of 8.5K grade school math word problems, released with a paper showing that even the largest Transformer models scored poorly, and that training verifiers to select among many candidate solutions improved results.
DeepMind publishes the 280-billion-parameter Gopher
On 8 December 2021, DeepMind published three language-model papers, including Gopher, a 280-billion-parameter Transformer, alongside a survey of 21 ethical and social risks and RETRO, a retrieval-based model.
High-Resolution Image Synthesis with Latent Diffusion Models
A 2021 paper by Rombach, Blattmann, Lorenz, Esser and Ommer that runs diffusion models inside the latent space of pretrained autoencoders, cutting the compute needed for high-resolution image synthesis while adding cross-attention for text and bounding-box control.
Hugging Face acquires Gradio
On 21 December 2021 Hugging Face acquired Gradio, a Python library for building web demos of machine-learning models that Abubakar Abid had started in 2019 and that had been used to build more than 300,000 demos.
Baidu and Peng Cheng Lab describe the 260-billion-parameter ERNIE 3.0 Titan
On 23 December 2021, authors from Baidu and Peng Cheng Laboratory posted a paper on ERNIE 3.0 Titan, a 260-billion-parameter Chinese pre-trained model they called the largest of its kind, reporting gains on 68 NLP datasets.
2022
OpenAI introduces InstructGPT and makes it the default in its API
On 27 January 2022 OpenAI announced InstructGPT, a version of GPT-3 trained with human feedback to follow user instructions more closely. OpenAI then made these models the default in its API.
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
A 2022 paper by Jason Wei and colleagues at Google Research showing that prompting a large language model with examples that include intermediate reasoning steps, rather than answers alone, substantially improves its performance on arithmetic, commonsense and symbolic reasoning tasks.
DeepMind introduces AlphaCode for competitive programming
On 2 February 2022, DeepMind introduced AlphaCode, a system that writes computer programs at a competitive level and, on average, ranked within the top 54% of participants across 10 Codeforces contests.
NVIDIA and SoftBank terminate the $40 billion Arm acquisition
On 7 February 2022, NVIDIA and SoftBank Group called off NVIDIA's planned acquisition of Arm Limited, citing regulatory obstacles. SoftBank kept a $1.25 billion prepayment and began preparing Arm for a public offering, while NVIDIA retained its 20-year Arm licence.
EleutherAI releases GPT-NeoX-20B
EleutherAI announced GPT-NeoX-20B on 2 February 2022 and released its weights on 9 February. The 20-billion-parameter model was trained on CoreWeave GPUs and released under Apache 2.0, and the group called it a research artifact rather than a production system.
Magnetic control of tokamak plasmas through deep reinforcement learning
A 2022 Nature paper by Degrave, Felici and colleagues showing that a deep reinforcement learning controller can learn on its own to command all of a tokamak's magnetic control coils, producing and holding a range of plasma shapes.
China's algorithm recommendation rules take effect
China's rules on algorithm-generated recommendation, jointly issued by four departments, took effect on 1 March 2022. They impose registration, content review and data-protection duties on providers and ban models that induce addiction or manipulate rankings and search results.
Training language models to follow instructions with human feedback (InstructGPT)
InstructGPT used human-written demonstrations, human rankings of outputs and reinforcement learning to make GPT-3 follow user instructions better. Labelers preferred its outputs even though it was far smaller than GPT-3.
NVIDIA announces the Hopper architecture and H100
At GTC on 22 March 2022, NVIDIA announced the Hopper architecture and its first Hopper GPU, the H100, headlined by a Transformer Engine and paired with new HBM3 memory and NVLink, with cloud instances planned from the third quarter of 2022.
Training Compute-Optimal Large Language Models (Chinchilla)
A DeepMind study showing that, for a fixed computing budget, language models should be trained on far more data than was then standard; its 70-billion-parameter Chinchilla beat larger contemporaries.
ASIMO's final demonstration and retirement
Honda's humanoid robot ASIMO formally retired on 31 March 2022, after a final demonstration at Honda's Tokyo headquarters; Honda said it would focus on remotely controlled avatar robots instead.
Google announces the 540-billion-parameter PaLM
On 4 April 2022, Google Research announced PaLM, a 540-billion-parameter language model trained on 6,144 TPU v4 chips with the Pathways system, and reported strong multi-step reasoning and above-average-human results on BIG-bench.
OpenAI announces DALL·E 2
On 6 April 2022 OpenAI announced DALL·E 2, its second text-to-image system, which generated images from CLIP embeddings with a diffusion decoder and was opened to the public in stages that year.
Meta releases OPT-175B to researchers
On 3 May 2022 Meta AI announced OPT-175B, a 175-billion-parameter language model, and released its pretrained weights and training code to researchers under a noncommercial licence. Meta said it was the first release of a model of that size with both weights and code.
Hugging Face raises $100 million at a $2 billion valuation
On 9 May 2022 Hugging Face announced a $100 million Series C round led by Lux Capital, valuing the company at $2 billion; the round was framed as building the GitHub of machine learning.
Google Research unveils Imagen
On 23 May 2022, Google Research presented Imagen, a text-to-image diffusion model built on a large frozen T5-XXL text encoder, which beat DALL-E 2 in human ratings on the new DrawBench benchmark but was never released publicly.
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (Imagen)
Imagen is a text-to-image diffusion model that uses a large pretrained language model to encode prompts; it reached a zero-shot FID of 7.27 on COCO and introduced the DrawBench benchmark.
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
FlashAttention is an exact attention algorithm built around GPU memory traffic: by tiling computation so intermediates stay on chip, it cuts reads and writes to high bandwidth memory, training Transformers faster and enabling longer sequences.
Beyond the Imitation Game (BIG-bench)
BIG-bench is a large community-built benchmark of 204 tasks designed to test language models on problems believed to lie beyond their current abilities, described in a paper posted in June 2022.
Emergent Abilities of Large Language Models
A 2022 paper arguing that some abilities of large language models appear only once models are large enough, and cannot be predicted by extrapolating from smaller models — so further scaling may keep widening what these models can do.
Hinton, LeCun, Bengio and Hassabis win the 2022 Princess of Asturias Award
In June 2022, the Princess of Asturias Award for Technical and Scientific Research was awarded jointly to Geoffrey Hinton, Yann LeCun, Yoshua Bengio and Demis Hassabis for advancing artificial intelligence and deep learning.
GitHub Copilot becomes generally available to individual developers
On 21 June 2022 GitHub made Copilot generally available to individual developers at 10 US dollars a month, after a preview that drew more than 1.2 million developers. It marked the arrival of an AI coding assistant inside mainstream editors.
Midjourney opens its public beta
On 12 July 2022 Midjourney opened its text-to-image generator to the public as an open beta, run entirely through a bot on Discord, with subscriptions costing $10 or $30 a month.
BigScience releases BLOOM, a 176B-parameter open multilingual model
On 12 July 2022 the BigScience workshop, coordinated by Hugging Face, released BLOOM, a 176-billion-parameter language model covering 46 natural and 13 programming languages.
AlphaFold database expands to over 200 million protein structures
On 28 July 2022, DeepMind and EMBL-EBI expanded the AlphaFold Protein Structure Database from nearly 1 million to over 200 million predicted protein structures, covering nearly all catalogued proteins known to science.
Midjourney image wins the Colorado State Fair digital art prize
In August 2022 an image made with Midjourney by Jason Allen, Théâtre D'opéra Spatial, took first place in the digital art category at the Colorado State Fair, setting off a widely reported argument about AI-generated art.
Xiaomi unveils CyberOne
On 11 August 2022 Xiaomi unveiled CyberOne, a 1.77-metre humanoid robot with 21 degrees of freedom, at Lei Jun's annual speech. The news lifted Chinese smart-machine stocks, but the robot had not entered mass production.
Stability AI publicly releases Stable Diffusion
On 22 August 2022 Stability AI publicly released Stable Diffusion, a latent text-to-image diffusion model whose weights, code and model card were published on Hugging Face.
Meta moves PyTorch to the new PyTorch Foundation
On 12 September 2022 Meta announced that PyTorch would move to the Linux Foundation as the PyTorch Foundation, with a governing board that included AMD, AWS, Google Cloud, Microsoft Azure and NVIDIA. PyTorch stayed under the BSD-3-Clause licence.
OpenAI open-sources the Whisper speech-recognition model
On 21 September 2022 OpenAI released Whisper, an open-source speech-recognition system trained on 680,000 hours of audio that could also translate speech into English.
Tsinghua and Zhipu publish the open bilingual GLM-130B
In October 2022 Tsinghua University's KEG group and Zhipu AI published GLM-130B, an open bilingual model with 130 billion parameters that was accepted to ICLR 2023.
ReAct: Synergizing Reasoning and Acting in Language Models
ReAct is a prompting method that has a language model interleave written reasoning with calls to outside tools or environments, improving question answering, fact verification and interactive decision-making.
US imposes export controls on advanced computing chips to China
On 7 October 2022 the US Bureau of Industry and Security issued two rules restricting China's access to advanced computing chips, supercomputer development and advanced chip manufacturing, setting thresholds that later shaped how Nvidia sold chips in China.
LAION-5B: An open large-scale dataset for training next generation image-text models
LAION-5B is an openly available dataset of 5.85 billion CLIP-filtered image-text pairs, released so that researchers outside a handful of labs could study and reproduce large vision-language models.
Stability AI raises $101 million
On 17 October 2022 Stability AI announced $101 million in funding, led by Coatue, Lightspeed Venture Partners and O'Shaughnessy Ventures, to accelerate open AI models across several media.
Mobileye returns to public markets on Nasdaq
Mobileye priced its Nasdaq initial public offering at 21 dollars a share and sold 41 million Class A shares, with a large part of the proceeds earmarked to repay a note owed to parent company Intel.
Nvidia ships the A800 to China to stay inside export rules
In late 2022, after the United States restricted exports of its A100 and H100 data-center chips, Nvidia began selling the A800 in China — a part whose only change was slower links between GPUs, keeping it inside the rules.
Midjourney releases V4 in alpha
On 5 November 2022 Midjourney released the alpha version of V4, its fourth numbered image model, which was trained on Google TPUs and was described as a large jump in quality over V3.
Meta launches Galactica and pulls its demo three days later
On 15 November 2022 Meta AI and Papers with Code unveiled Galactica, a language model meant to help scientists, with a public demo. Meta took the demo offline about three days later after users showed it producing fabricated, confident answers.
Holistic Evaluation of Language Models (HELM)
A 2022 paper by Percy Liang, Rishi Bommasani, Tony Lee and many co-authors that ran 30 language models through the same 42 scenarios and seven metrics, turning scattered benchmarks into one standardized evaluation.
Stability AI releases Stable Diffusion 2.0
On 24 November 2022 Stability AI released Stable Diffusion 2.0 as open source, with an OpenCLIP text encoder and a revised training-data filter; version 2.1 followed on 7 December.
OpenAI releases text-davinci-003 in the GPT-3.5 series
On 28 November 2022 OpenAI released text-davinci-003 in its GPT-3.5 series; two days later came ChatGPT, fine-tuned from a model in the same series and free during its research preview[^1][^2][^4].
