Jacob Devlin

Overview

Jacob Devlin is a research scientist best known as the first author of BERT, a language model that Google introduced in 2018. Before BERT, language systems were usually built separately for each task, which required large labelled datasets for each one. BERT became a widely used starting point for language understanding systems, and the paper later received a best paper award at a major language conference.^[2][4]

Education and early research

Devlin earned a master’s degree in computer science from the University of Maryland in 2009, where he was advised by Bonnie Dorr.^[1] His 2014 paper, “Fast and Robust Neural Network Joint Models for Statistical Machine Translation,” won the Best Long Paper award at ACL 2014.^[1] Statistical machine translation is the approach that translates text by learning patterns from large collections of parallel text, and this paper combined it with neural network models.

Key contributions

On 11 October 2018 Devlin posted “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” to arXiv. He is its first author, with Ming-Wei Chang, Kenton Lee and Kristina Toutanova as co-authors.^[2]

Earlier language models often read text in one direction, from left to right. BERT instead uses the words on both sides of each word to work out its meaning. It is pre-trained on large amounts of unlabelled text by hiding some words and training the model to predict them, and it can then be adapted to a specific task with comparatively little additional training. This two-step pattern of pre-training and then fine-tuning became a common way to build language systems.

With Chang, Devlin wrote Google’s 2 November 2018 post announcing the open-source release of BERT.^[3]

Positions

From 2014 to 2017 Devlin was a research scientist at Microsoft Research. There he helped move Microsoft Translate from phrase-based to neural machine translation, and built on-device models for mobile translation. He was also lead author of RobustFill (2017), Program synthesis asks a system to produce a working program from examples of the desired behaviour, instead of a person writing the code by hand.

In 2018 he was a Senior Research Scientist at Google, working on deep-learning models for information retrieval and language understanding.^[1]

Impact

The BERT paper won the Best Long Paper award at NAACL 2019.^[4] Its pre-training approach has been widely adopted and used as a comparison point in later language research. Devlin’s work moved from machine translation toward language understanding, and BERT is the work most closely associated with his name.^[4] For readers new to the field, the core idea is that a model which has first learned the general structure of language from unlabelled text can be taught a specific task with far fewer labelled examples than a model built from scratch. Pre-training on unlabelled text has since become a common pattern across language models.