Google researchers publish BERT on arXiv

Background

Earlier language models read text in one direction, usually left to right, so each word was predicted from the words before it. Tasks such as answering a question depend on context on both sides of a word, and a one-directional design is a poor fit for that. The Transformer architecture gave researchers a flexible base for models that look at context in every direction.

What happened

On 11 October 2018 Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova, all of Google, submitted the paper to arXiv.^[1] BERT stands for Bidirectional Encoder Representations from Transformers. It pre-trains deep bidirectional representations from unlabeled text, using both left and right context in every layer.^[1]

A pre-trained BERT model can be fine-tuned with just one additional output layer for tasks such as question answering and language inference, without major changes to the task-specific architecture.^[1]

Fine-tuning means taking a model that has already been trained on large amounts of unlabelled text and continuing its training on a smaller labelled dataset for one task. The added output layer is the only new component, which is why the change to the architecture is described as small.^[1]

Key details

The paper reported new state-of-the-art results on eleven natural language processing tasks. They included a GLUE score of 80.5%, a 7.7-point absolute improvement, and a MultiNLI accuracy of 86.7%, 4.6 points higher.^[1] GLUE is a collection of language understanding tasks, and its score combines results across them.

On SQuAD v1.1, a question-answering benchmark, BERT reached a test F1 score of 93.2, 1.5 points better than the previous result. On SQuAD v2.0 it reached 83.1, 5.1 points better.^[1] F1 is a score that balances precision and recall; higher is better.

Google's blog noted that the SQuAD v1.1 F1 of 93.2 exceeded the previous best of 91.6 and the human-level score of 91.2.^[2]

On 2 November 2018 Google open-sourced BERT, releasing TensorFlow source code, pre-training code and several pre-trained English-only models.^[2] Google said that with the release, anyone could train a state-of-the-art question answering system in about 30 minutes on a single Cloud TPU, or in a few hours on a single GPU.^[2]

Impact

BERT made the pattern of pre-training followed by fine-tuning easy for others to reuse. Because the code and pre-trained models were released, researchers and developers could start from a strong model instead of training one from scratch. The released models were English-only, so work in other languages needed other models. The reported gains were measured on public benchmarks, and the human-level reference in Google's blog shows how such scores are read: as comparisons with human performance on the same tests.

Benchmarks such as GLUE and SQuAD became common ways to compare language models, and a high score was often read as progress. A score on a fixed test set does not capture everything about a model, however, so such results are best read as evidence on those specific tasks.