Quoc Le

Overview

Quoc Le is a computer scientist working on machine learning. His name appears on papers in several different parts of the field: learning features from unlabelled data, mapping one sequence to another, searching for network architectures automatically, and scaling image models. His doctoral thesis at Stanford was on learning features at scale.^[3]

Doctoral work

Stanford University's library record lists a Ph.D. thesis titled "Scalable feature learning" by Quoc V. Le, submitted in 2013, and names Andrew Y. Ng as the primary advisor.^[3] The subject sits close to the centre of how modern machine learning works. Rather than a person deciding by hand which measurements of the data a model should pay attention to, the model is given raw data and learns a representation — a set of features — for itself. The thesis title points at the version of that question in which both the data and the model are large, so the method has to remain workable as it scales up.

High-level features from unlabelled data

A paper titled "Building high-level features using large scale unsupervised learning" was submitted to arXiv on 29 December 2011; Le is one of its co-authors, alongside Marc'Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg S. Corrado, Jeff Dean and Andrew Y. Ng.^[1] Unsupervised learning means training a model on data that arrives without labels attached — nobody has marked which examples belong to which category. The question the title raises is whether such training, carried out at large scale, can produce "high-level" features: internal units that respond to broad categories rather than to raw pixels or simple edges.

Sequence to sequence

"Sequence to Sequence Learning with Neural Networks", written with Ilya Sutskever and Oriol Vinyals, was submitted to arXiv on 10 September 2014.^[2] The setting it describes is one in which both the input and the output are sequences — sentences, for instance — and the model has to read the input and then generate an output that need not have the same length. The phrase "sequence to sequence" became the name for treating that whole mapping as a single learned problem rather than a chain of separately designed steps. The paper was published at NeurIPS 2014, and NeurIPS later announced its 2024 Test of Time Paper Awards for papers published at that conference in 2014, with the sequence-to-sequence paper among the winners.^[4] Such an award looks back ten years and singles out work from an earlier conference whose significance has lasted.

Neural architecture search

"Neural Architecture Search with Reinforcement Learning", with Barret Zoph, was submitted to arXiv on 5 November 2016.^[5] A neural network's architecture is its layout: how many layers it has, what each layer does, and how the layers are wired together. Designing that layout has traditionally been a matter of human judgement, guided by experiment. The title names the alternative the paper explores — treating the choice of architecture as something a learning algorithm can make for itself. Reinforcement learning is the branch of machine learning in which a system learns by trial, adjusting its behaviour according to the reward it receives; here that machinery is applied to the design decision instead of to the task the network is later trained on.

EfficientNet

"EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks", with Mingxing Tan, was submitted to arXiv on 28 May 2019.^[6] Convolutional networks, the standard architecture for image recognition, offer several separate dials for making a model bigger: more layers, wider layers, or larger input images. The paper's subject is how those dials should be turned — one at a time, or together in some proportion — so that extra computation buys as much accuracy as possible. The word "efficient" in the title refers to exactly that trade-off between the size of a model and the quality of its predictions.

A recurring move

Across these papers a similar move repeats. A decision that would otherwise be made by hand — which features to use, how to carry one sequence over into another, which architecture to build, how far to scale it — is turned into something that can be learned, searched over, or measured. That is the thread connecting a doctoral thesis on scalable feature learning to later work on sequence models, architecture search and image models.