Google Brain

What it was

Google Brain was a deep-learning research team at Google that began in 2011 inside X, the company's moonshot incubator. It started as a collaboration involving Jeff Dean, Greg Corrado and Stanford professor Andrew Ng^[1]. The team worked on neural networks — layered stacks of simple numerical units whose connection weights are tuned during training — and on running them at a scale that had until then been mostly a matter of speculation.

The cat experiment

The team's first widely reported result came in June 2012, and Google published it on its blog on 26 June 2012^[2]. The network had more than 1 billion connections spread across 16,000 CPU cores, and it was trained on unlabeled stills taken from YouTube videos^[2]. Unlabeled means no human had tagged the images in advance, so the network had to find structure in the raw pixels on its own. One artificial neuron — a single unit inside the network — learned to detect cats, responding most strongly to images that contained them^[2]. The team reported a 70% relative improvement in accuracy on a standard image-classification test^[2]. What travelled was the detail that a concept had emerged from raw video without anyone supplying its name.

A team assembled

In 2013 Google hired Geoffrey Hinton and acquired DNNResearch, the company he had founded with Alex Krizhevsky and Ilya Sutskever; Sutskever became a research scientist at Google Brain^[6].

In September 2014 Sutskever, Oriol Vinyals and Quoc Le posted "Sequence to Sequence Learning with Neural Networks", the encoder-decoder approach behind later neural machine translation^[7]. In this setup one network reads a whole input sequence into an internal representation, and a second network generates the output from it, so a single model can handle inputs and outputs of different lengths.

Tools and translation

Google open-sourced the TensorFlow machine-learning framework on 9 November 2015^[5]. Making the framework public let outside researchers build on the same machinery the team used internally.

In September 2016 Google launched Google Neural Machine Translation (GNMT), which translates whole sentences in context rather than piece by piece^[1]. Sentence-level translation matters because word order and word meaning depend on the rest of the sentence.

The Transformer

In June 2017 Google researchers, including Ashish Vaswani and Noam Shazeer, published "Attention Is All You Need", which introduced the Transformer architecture^[4]. The paper was submitted to arXiv on 12 June 2017 and proposed an architecture based solely on attention mechanisms^[4]. Attention lets every position in a sequence weigh every other position directly, instead of passing information step by step through a recurrence.

Scale

In April 2022 Google Research announced PaLM, a 540-billion-parameter language model trained with the Pathways system^[8]. Parameters are the adjustable numbers that make up a model; a count in the hundreds of billions describes how much capacity was trained.

Into Google DeepMind

On 20 April 2023 Google combined the Brain team with DeepMind to form Google DeepMind under CEO Demis Hassabis, with Jeff Dean becoming Google's Chief Scientist. Sundar Pichai credited the two teams with AlphaGo, Transformers, word2vec, WaveNet, AlphaFold, sequence-to-sequence models, distillation, deep reinforcement learning, and TensorFlow and JAX^[3].

Why it mattered

Google Brain sat where academic-style deep-learning research turned into production systems inside a large company: a small group started in an incubator, a result showed that training on unlabeled raw data could recover something recognisable, hiring and an acquisition brought researchers in, and a chain of openly published methods and frameworks followed that others could build on^[1][2][5][6]. The 2023 merger placed that work in the same organisation as DeepMind^[3].