2012–2017 · 95 key events · 84 related people, organisations and concepts
Deep learning
Neural networks trained on GPUs take the lead in vision, speech and games.
2012
ImageNet Classification with Deep Convolutional Neural Networks
The 2012 NeurIPS paper that introduced AlexNet, a large convolutional network trained on GPUs. It won the ILSVRC-2012 image classification competition by a wide margin and is widely seen as a turning point for deep learning in computer vision.
Amazon agrees to acquire Kiva Systems
In March 2012 Amazon agreed to buy Kiva Systems, a maker of warehouse robots used by large retailers, for about $775 million in cash — the company's second-largest acquisition at the time.
DARPA announces the DARPA Robotics Challenge
On 10 April 2012 DARPA announced a challenge to build ground robots that can carry out complex tasks in dangerous, degraded, human-built environments, backing it with a $2 million prize and a list of tasks from driving a vehicle to closing a valve.
Nevada issues the first US autonomous vehicle licence
On 7 May 2012 Nevada's DMV issued the first US licence for an autonomous vehicle, a Toyota Prius fitted with Google's self-driving technology, after tests on highways, Carson City streets and the Las Vegas Strip.
Google launches the Knowledge Graph
On 16 May 2012 Google announced the Knowledge Graph, an attempt to move search from matching strings to identifying entities; at launch it held more than 500 million objects.
Building High-level Features Using Large Scale Unsupervised Learning
In 2012, Quoc Le, Andrew Ng and colleagues trained a large unsupervised neural network on 10 million internet images and showed it could detect faces without a single labelled face.
Google announces Google Now at I/O 2012
Google announced Google Now at its I/O conference in June 2012, presenting it as an Android search experience that shows cards like weather, traffic and sports scores without the user searching. It arrived alongside Android 4.1 Jelly Bean and the Nexus 7 tablet.
Merck Molecular Activity Challenge on Kaggle
A Kaggle competition run in August–October 2012 in which competitors predicted molecules' biological activities from numerical descriptors of their chemical structures; a deep-learning team led by Geoffrey Hinton took first place.
Curiosity lands on Mars
NASA's Curiosity rover landed on the floor of Gale Crater on 6 August 2012 Universal Time, lowered on a tether by a rocket-powered descent stage that then cut the cords and flew off to a controlled crash.
Rethink Robotics unveils Baxter
In September 2012 Rethink Robotics unveiled Baxter, a robot with sensors and software to help it see and understand its environment, priced at $22,000 and set to go on sale in October 2012.[^1]
Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups
A 2012 article in IEEE Signal Processing Magazine in which four research groups argued that deep neural networks, replacing Gaussian mixture models, gave better acoustic models for speech recognition.
AlexNet wins the ImageNet ILSVRC 2012 challenge
In October 2012 AlexNet, a deep neural network built by a University of Toronto team, won the ImageNet image-classification challenge by a wide margin. Its result showed that large neural networks trained on GPUs could outperform hand-designed vision methods.
Microsoft demonstrates deep-learning speech translation
On 25 October 2012, Microsoft's Rick Rashid demonstrated a system in Tianjin that turned his spoken English into computer-generated Chinese speech, built on deep neural networks that sharply cut recognition errors.
Large Scale Distributed Deep Networks
A 2012 NIPS paper by Jeffrey Dean, Andrew Ng and colleagues that introduced DistBelief, a framework for training deep networks across clusters of thousands of machines, together with the Downpour SGD and Sandblaster optimisation procedures.
2013
Clarifai wins ILSVRC 2013 classification
In 2013 the ILSVRC classification task was won by Clarifai with an ensemble of convolutional networks at 11.7% top-5 error, while OverFeat won localisation. Zeiler and Fergus's visualization paper, also reporting ImageNet results, appeared the same month.
Efficient Estimation of Word Representations in Vector Space
Submitted to arXiv in January 2013 by Tomas Mikolov and colleagues at Google, this paper introduced CBOW and Skip-gram, two simple architectures that learn word vectors quickly from very large text collections. They are the basis of the word2vec project.
Maxout Networks
A 2013 paper by Goodfellow, Warde-Farley, Mirza, Courville and Bengio introduces maxout units, whose output is the maximum of a set of inputs, designed to work with dropout, and reports state-of-the-art results on four benchmarks.
Google acquires DNNresearch
In March 2013 the University of Toronto announced that Google had acquired DNNresearch, a start-up incorporated in 2012 by Geoffrey Hinton and two of his graduate students for work on deep neural networks.
Speech Recognition with Deep Recurrent Neural Networks
A 2013 paper by Graves, Mohamed and Hinton that examines deep recurrent neural networks for speech recognition, arguing for end-to-end sequence training with methods such as connectionist temporal classification.
First ICLR held in Scottsdale with open peer review
ICLR was founded in 2012 by Yann LeCun and Yoshua Bengio, and first met in Scottsdale, Arizona, in May 2013, with paper decisions posted publicly on OpenReview.
Distributed Representations of Words and Phrases and their Compositionality
A 2013 paper by Mikolov, Sutskever, Chen, Corrado and Dean presents continuous Skip-gram as an efficient method for learning distributed word and phrase representations, with extensions that improve vector quality and training speed.
Rich feature hierarchies for accurate object detection and semantic segmentation
R-CNN, posted to arXiv by Girshick, Donahue, Darrell and Malik in November 2013, applied convolutional neural networks to region proposals for object detection and lifted mean average precision on PASCAL VOC 2012 to 53.3%, more than 30% above the previous best[^1].
Visualizing and Understanding Convolutional Networks
Zeiler and Fergus ask why large convolutional networks perform so well on ImageNet, introduce a visualization technique for intermediate layers, and use ablation to find an architecture that outperforms Krizhevsky et al. The model also transfers to Caltech-101 and Caltech-256.
Google acquires Boston Dynamics
In December 2013 Google confirmed it had completed its acquisition of Boston Dynamics, the eighth robotics company it had bought in six months, as part of a robotics effort led by Andy Rubin.
Amazon unveils Prime Air drone delivery plan
On 1 December 2013 Jeff Bezos announced Prime Air on CBS's 60 Minutes: drones to deliver packages within 30 minutes, carrying up to 2.3 kg within about 16 km of a warehouse, pending FAA rules and years of testing.
Facebook founds FAIR with Yann LeCun as director
In late 2013 Facebook launched Facebook AI Research (FAIR), with Yann LeCun as its first director. The lab's open research model shaped Meta's later AI work, including PyTorch and the Llama models.
DeepMind posts its deep Q-network Atari paper
On 19 December 2013, DeepMind researchers posted a paper showing that one convolutional network, trained with a variant of Q-learning, could learn Atari 2600 control policies directly from raw pixels and outperform prior methods on six of seven games.
Playing Atari with Deep Reinforcement Learning
A 2013 paper by Mnih, Kavukcuoglu, Silver and colleagues showed a convolutional neural network trained with a Q-learning variant learning to play seven Atari 2600 games directly from raw pixels, beating prior methods on six and human experts on three.
Auto-Encoding Variational Bayes
A 2013 paper by Diederik P. Kingma and Max Welling that introduced a stochastic variational inference and learning algorithm for directed latent-variable models with intractable posteriors, making approximate inference practical on large datasets.
DARPA Robotics Challenge Trials
A two-day DARPA competition in December 2013 in which 16 teams put prototype disaster-response robots through eight simulated tasks; SCHAFT of Tokyo scored highest with 27 points out of a possible 32.
2014
GoogLeNet and VGG dominate ILSVRC 2014
At ILSVRC 2014, GoogLeNet won the classification track with 6.67% top-5 error, while VGG's network won localisation and took second in classification; more than 50 institutions took part.
DeepFace: Closing the Gap to Human-Level Performance in Face Verification
DeepFace is a 2014 face verification pipeline that rebuilt the alignment and representation stages of the standard face pipeline, reaching 97.35% accuracy on Labeled Faces in the Wild with a nine-layer network of over 120 million parameters.
Dropout: A Simple Way to Prevent Neural Networks from Overfitting
A 2014 paper by Srivastava, Hinton, Krizhevsky, Sutskever and Salakhutdinov introducing dropout, a training trick that randomly removes units so large neural networks overfit less.
IBM creates the Watson Group business unit
On 9 January 2014 IBM announced the Watson Group, a business unit for cloud-delivered cognitive services, backed by more than $1 billion and about 2,000 professionals — an attempt to turn Watson into a commercial platform rather than a research project.
Google acquires DeepMind
On 26 January 2014 Google confirmed its acquisition of DeepMind, the London AI company founded in 2010. The deal brought a small deep-learning research group known for Atari-playing systems into Google and set up a unit later merged with Google Brain.
Microsoft unveils Cortana with Windows Phone 8.1
On 2 April 2014 at Build, Microsoft unveiled Windows Phone 8.1 and introduced Cortana, a Bing-powered personal assistant with a Halo-inspired persona, a developer API and a rollout starting with a U.S. beta.
Microsoft COCO: Common Objects in Context
A 2014 paper introducing Microsoft COCO, a large dataset of everyday scenes in which common objects are labeled with per-instance segmentations, built to study object recognition as part of scene understanding.
Andrew Ng joins Baidu as chief scientist
On 16 May 2014 US time, Baidu named Andrew Ng, a Stanford professor and co-founder of Coursera, as chief scientist in overall charge of Baidu Research, announced at the opening of its Sunnyvale office and new Silicon Valley AI Lab.
Google unveils its own self-driving prototype without a steering wheel
On 27 May 2014, Google published a design for a self-driving prototype with no steering wheel or pedals and said it planned to build about a hundred vehicles and test them with safety drivers.
The original GAN paper is submitted
Goodfellow, Bengio, and co-authors proposed a generative-model framework with adversarially trained generator and discriminator.
Generative Adversarial Nets
Goodfellow, Bengio, and co-authors proposed a generative-model framework with adversarially trained generator and discriminator.
IBM publishes the TrueNorth neuromorphic chip in Science
On 8 August 2014 Science published the paper describing TrueNorth, an IBM chip holding 1 million spiking neurons and 256 million synapses on 5.4 billion transistors, showing a brain-inspired design running on milliwatts.
Neural Machine Translation by Jointly Learning to Align and Translate
Submitted to arXiv in September 2014 by Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio, this paper let a translation model look back at the whole source sentence while producing each word. This mechanism, called attention, later became a basis for Transformer models.
Very Deep Convolutional Networks for Large-Scale Image Recognition
A 2014 study showing that stacking many small 3×3 convolution filters, up to 16–19 weight layers, improves large-scale image recognition; its models took first place in localisation and second in classification at ImageNet Challenge 2014 and were released publicly.
Sequence to Sequence Learning with Neural Networks
Ilya Sutskever, Oriol Vinyals and Quoc Le's 2014 paper showed that two LSTMs, one reading the input and one writing the output, could translate sentences and outperform a phrase-based statistical baseline. It set the encoder–decoder pattern behind later translation models.
Going Deeper with Convolutions
The 2014 paper that introduced the Inception architecture and its 22-layer GoogLeNet network, which took first place in the ImageNet 2014 classification and detection challenge with a 6.67% top-5 error.
Going Deeper with Convolutions (GoogLeNet / Inception)
A 2014 paper from Google researchers that introduced the Inception architecture, whose GoogLeNet model set a new state of the art in ILSVRC 2014 by widening and deepening a network within a fixed computational budget.
GloVe: Global Vectors for Word Representation
GloVe is a 2014 method that learns word vectors by factorising global word co-occurrence counts. Released together with pre-trained vectors, it became, alongside word2vec, one of the two standard ways of producing static word embeddings.
Neural Turing Machines
A 2014 paper by Graves, Wayne and Danihelka that couples a neural network to an external memory it reads and writes through attention, making the whole machine differentiable and trainable by gradient descent.
Conditional Generative Adversarial Nets
A 2014 paper by Mehdi Mirza and Simon Osindero that adds conditioning data to both the generator and the discriminator, enabling a GAN to generate samples such as MNIST digits conditioned on class labels.
Show and Tell: A Neural Image Caption Generator
A 2014 paper by Vinyals and colleagues that trained a single deep network to turn a photograph into a sentence, reporting a BLEU-1 score of 59 on Pascal against 25 for the previous state of the art.
Deep Speech: Scaling up end-to-end speech recognition
A 2014 paper presenting an end-to-end deep-learning speech recognition system that removes hand-engineered noise modules and the phoneme dictionary, relying instead on recurrent networks trained across multiple GPUs and synthesized data.
Adam: A Method for Stochastic Optimization
Adam is a stochastic optimisation algorithm from a 2014 paper by Diederik P. Kingma and Jimmy Ba that sets per-parameter step sizes from running estimates of the gradient's first and second moments, needing little memory and little hyper-parameter tuning.
2015
Deep learning
A 2015 Nature review by Yann LeCun, Yoshua Bengio and Geoffrey Hinton explaining deep learning: models of multiple processing layers that learn layered representations of data, and the results this produced in speech, vision and other fields.
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
A 2015 arXiv paper by Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho and co-authors that introduced an image-captioning model with visual attention, able to show where it is looking while writing each word.
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
A 2015 paper by Sergey Ioffe and Christian Szegedy that proposed normalising each layer's inputs per training mini-batch, which allowed higher learning rates, looser initialisation and much faster training of deep networks.
Human-level control through deep reinforcement learning
A 2015 Nature paper by Mnih, Kavukcuoglu, Silver and colleagues showed that one deep Q-network agent, given only screen pixels and the game score, learned to play 49 Atari 2600 games at a level comparable to a professional human tester.
U-Net: Convolutional Networks for Biomedical Image Segmentation
A convolutional network for biomedical image segmentation, posted to arXiv on May 18, 2015, that trains end to end from very few annotated images and won the ISBI cell tracking challenge 2015 in its categories.
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
A 2015 paper that introduced a Region Proposal Network sharing full-image convolutional features with the detection network, making proposals nearly cost-free and enabling end-to-end training, 5 fps GPU detection, and state-of-the-art accuracy.
DARPA Robotics Challenge Finals
The DARPA Robotics Challenge Finals, held on 5 and 6 June 2015 at the Fairplex in Pomona, California, put 23 teams and their robots through eight tasks; Team KAIST's DRC-Hubo took first place and the $2 million prize.
You Only Look Once: Unified, Real-Time Object Detection
The 2015 paper introduced YOLO, which treats object detection as a single regression from image pixels to bounding boxes and class probabilities. One neural network makes predictions in one evaluation, reaching real-time speed.
You Only Look Once: Unified, Real-Time Object Detection
The 2015 YOLO paper recast object detection as a single neural network's regression over bounding boxes and class probabilities, achieving real-time detection while reducing false positives on empty background.
Tesla enables Autopilot with Version 7.0
On 14 October 2015 Tesla announced Version 7.0 of its car software, which switched on Autopilot: lane steering, turn-signal lane changes and traffic-aware cruise control, running on sensor hardware the Model S had been carrying since the year before.
Toyota announces the Toyota Research Institute
Toyota announced in November 2015 that it would create the Toyota Research Institute, a separately incorporated artificial intelligence research company, with $1 billion over five years and former DARPA programme manager Gill Pratt as chief executive.
Google open-sources TensorFlow
On 9 November 2015 Google released TensorFlow, its second-generation machine learning system, as open source under the Apache 2.0 licence, letting people outside Google build and train models with the same system Google used in its own products.
OpenAI is founded as a nonprofit AI research lab
In December 2015 a group of entrepreneurs and AI researchers founded OpenAI in San Francisco as a non-profit research company whose stated aim was to advance digital intelligence for humanity's benefit rather than financial return.
Deep Residual Learning for Image Recognition
The 2015 paper introduced residual networks, which add each layer's input back to its output through a shortcut connection. This made networks of over a hundred layers trainable and won first place in several ILSVRC and COCO tasks in 2015.
ResNet wins ILSVRC 2015 with 3.57% error
The 2015 ImageNet classification contest was won by ResNet, a network up to 152 layers deep whose layers learn residual functions relative to their inputs; it reached 3.57% error on the test set and, by one account, exceeded human performance.
2016
The Partnership on AI is established
A multi-stakeholder organisation formally established in late 2016 by researchers from six of the world's largest technology companies to study and share best practices in AI, advance public understanding and provide an open platform for discussion of AI's effects on society.
Mastering the game of Go with deep neural networks and tree search
The 2016 Nature paper describing AlphaGo, which combined deep neural networks with tree search. In October 2015 its distributed version beat the European Go champion Fan Hui 5-0, the first time a program defeated a professional player in full-sized Go without handicap.
AlphaGo paper in Nature reveals win over Fan Hui
On 27 January 2016 DeepMind announced AlphaGo alongside a paper in Nature reporting that the program had beaten European Go champion Fan Hui 5–0 in October 2015 — the first time a computer program had beaten a professional Go player.
AlphaGo defeats Lee Sedol 4–1 in Seoul
In March 2016 DeepMind's AlphaGo beat Lee Sedol, an 18-time world champion Go player, 4–1 in a five-game match in Seoul, a result watched by more than 200 million people worldwide.
Identity Mappings in Deep Residual Networks
A 2016 analysis of residual networks arguing that with identity skip connections and the activation placed after addition, signals travel directly between blocks; a new residual unit built on this eases training, and a 1001-layer network reaches 4.62% error on CIFAR-10.
Microsoft's Tay chatbot is shut down within a day after posting racist tweets
Tay was a Microsoft chatbot launched on Twitter on March 23, 2016 to learn from conversations with young users. Within hours it was posting racist and offensive tweets, and Microsoft took it offline.
Google reveals the Tensor Processing Unit
In May 2016 Google disclosed the Tensor Processing Unit, a custom chip built for machine learning that had already run in its data centers for over a year and powered RankBrain, Street View and AlphaGo.
Google introduces the Assistant and Google Home at I/O 2016
At its I/O conference in May 2016, Google introduced the Google assistant, previewed the Google Home speaker that brings it into the house, and showed the Allo and Duo apps that carry it on phones.
China issues its three-year action plan for "Internet Plus" AI
On 18 May 2016, four Chinese government bodies issued a three-year action plan for "Internet Plus" artificial intelligence, setting targets for 2018, naming priority technologies and products, and putting a figure on the market it aimed at.
ResNet wins the CVPR 2016 Best Paper Award
At CVPR 2016 in Las Vegas, the Best Paper Award went to "Deep Residual Learning for Image Recognition" by Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun. Google Scholar later reported it as the most cited research paper over five years.
SQuAD: 100,000+ Questions for Machine Comprehension of Text
SQuAD is a reading-comprehension dataset of more than 100,000 crowdworker questions on Wikipedia articles, each answered by a span of its passage. Released in 2016, it gave machine comprehension a large, freely available benchmark with a measured human ceiling.
NVIDIA delivers the first DGX-1 to OpenAI
In August 2016 NVIDIA CEO Jensen Huang hand-delivered the first DGX-1, a turnkey eight-GPU deep learning machine, to OpenAI in San Francisco — an early piece of dedicated AI hardware at a young research lab.
DeepMind introduces WaveNet for raw audio generation
On 8 September 2016, DeepMind announced WaveNet, a deep neural network that generates raw audio one sample at a time. In blind listening tests it beat Google's existing text-to-speech systems and narrowed the gap to human speech.
WaveNet: A Generative Model for Raw Audio
WaveNet is a deep neural network that generates raw audio waveforms one sample at a time, predicting each sample from all the ones before it. Used for text-to-speech, it was rated by listeners as more natural than the systems it was compared with.
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
A 2016 paper describing Google's neural machine translation system, GNMT: a deep LSTM encoder-decoder with attention, designed to make end-to-end translation practical by addressing rare words, parallelism and inference cost.
Google launches GNMT in production for Chinese-to-English translation
On 27 September 2016 Google Research announced GNMT, a neural machine translation system it put into production for Chinese-to-English on Google Translate, reporting large cuts in translation errors.
Hangzhou launches City Brain with Alibaba Cloud ET
On 13 October 2016 Alibaba Cloud announced a Hangzhou project, led by the municipal government, that used its ET AI programme and big-data analytics to predict traffic in real time and adjust traffic lights.
Neural Architecture Search with Reinforcement Learning
A 2016 paper by Barret Zoph and Quoc V. Le that uses a recurrent network trained with reinforcement learning to design neural network architectures automatically, matching human-designed models on CIFAR-10 and beating LSTM on Penn Treebank.
Amazon unveils the Amazon Go checkout-free store
On 5 December 2016 Amazon showed a prototype food store in Seattle, Amazon Go, where shoppers would be charged automatically instead of queuing at a checkout.
Alphabet spins its self-driving car project out as Waymo
In December 2016 Alphabet spun its self-driving car project out into a separate company, Waymo, turning a long-running research effort into a business that supplies self-driving technology rather than cars.
2017
The term 'deepfake' appears on Reddit
In late 2017 a Reddit user going by the name "deepfakes" posted face-swapped videos, and the word became the common label for synthetic media made with deep learning.
PyTorch is publicly released
PyTorch was publicly released in January 2017 as a beta, with first-class Python integration and an imperative style. Built by people from the Lua Torch community with Facebook's backing, it later became a widely used deep-learning framework.
Mask R-CNN
Mask R-CNN is a 2017 framework for object instance segmentation. It detects objects and produces a high-quality mask for each instance by adding a parallel mask branch to Faster R-CNN.
Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks
CycleGAN learns image-to-image translation between two domains without aligned pairs, combining an adversarial loss with a cycle-consistency constraint to constrain the mapping.
Baidu launches the Apollo open autonomous driving platform
In April 2017 Baidu launched Apollo, an open platform giving partners vehicle, hardware and software platforms plus cloud data services, which Baidu hoped would become the Android of the automotive world.
AlphaGo beats Ke Jie 3–0 and retires
In May 2017 AlphaGo beat Ke Jie, then the world's top-ranked Go player, in all three games of a match at the Future of Go Summit in Wuzhen, China. DeepMind retired the program from competition afterwards.
NVIDIA unveils Volta and Tesla V100 at GTC 2017
On May 10, 2017, NVIDIA unveiled Volta, its seventh-generation GPU architecture, together with the Tesla V100 data center GPU, presented as the most powerful GPU computing architecture yet for AI and high performance computing.
