Barret Zoph

Who he is

Barret Zoph is an AI researcher whose public profile combines research work with leadership roles. His X profile lists him as CTO and co-founder of Thinking Machines Lab and as VP Research at Google DeepMind ^[6]. His name is attached in particular to two lines of work: neural architecture search, which he led as first author, and the Switch Transformer, which he co-authored.

Starting out in the Google Brain Residency

Zoph has said that he started his AI career in the Google Brain Residency class ^[4]. Residency programs of this kind are a common entry route into industrial AI labs: researchers spend a fixed period working on real projects inside the lab alongside its research staff, rather than following a course of study. For someone entering the field, the residency is where the day-to-day practice of machine learning research — running experiments, reading results, deciding what to try next — gets learned.

Neural architecture search

On 5 November 2016, the paper "Neural Architecture Search with Reinforcement Learning" was submitted to arXiv, with Zoph as first author alongside Quoc V. Le ^[1].

The problem it addressed is a basic one in deep learning. A neural network's architecture — which layers and operations it contains and how they are connected — was normally chosen by hand, with researchers drawing on experience and on trial and error. The paper's idea was to make that choice part of learning itself. A learned controller proposes candidate architectures; each candidate is trained and evaluated on the task; the resulting performance is fed back as a reward signal, and the controller is adjusted so that it proposes better architectures next time. The architecture becomes something searched for rather than something specified in advance. This is the sense in which the method is a reinforcement learning one: a system that takes actions (proposing designs) and improves them using the reward it receives (how well the resulting network performs).

Switch Transformers

On 11 January 2021, "Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity" was submitted to arXiv, co-authored with William Fedus and Noam Shazeer ^[2].

The title points at a trade-off that shapes large models. Raising the number of parameters in a dense network normally raises the computation needed for every input, because every parameter takes part in every forward pass. Sparsity offers a way around that. In a sparse mixture-of-experts model, the network contains many separate "expert" sub-networks together with a routing mechanism that sends each input to only a small subset of them. Most parameters therefore stay unused for any given input, and the model can hold far more parameters than a dense model with the same per-input computational cost.

Moving between labs

After six years at Google Brain, Zoph announced on X that he had joined OpenAI ^[3]. In another post on X he wrote that he was joining Google DeepMind to work on reinforcement learning and post-training ^[4]. The Next Web headlined the move as a return to Google after "six years and two startups" ^[5]. His X profile, which lists both Thinking Machines Lab and Google DeepMind, therefore records a co-founder role at a new company alongside a research leadership role at Google's AI unit ^[6].

What the record shows

The two papers sit at different layers of the same enterprise. One asks how the shape of a network gets decided, and proposes handing that decision to a learning system rather than to intuition. The other asks how a model can be made much larger without making every input much more expensive to process, and answers with sparsity. Between them they cover two recurring questions in the design of large-scale machine learning systems: where the design choices come from, and how capacity is paid for.