BERT

What BERT is

BERT, short for Bidirectional Encoder Representations from Transformers, is a language model built from an encoder-only Transformer. An encoder reads a piece of text and turns each word into a numerical representation that carries information about the words around it; it does not generate new text by itself. Those representations can then be attached to a task-specific output layer.

The model comes from Google AI Language. The paper by Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova was posted to arXiv on 11 October 2018, and Google open-sourced the code and the pre-trained models on 2 November 2018.^[1]

Masking, and why both directions matter

The design choice that separates BERT from earlier pre-trained models is how much context it uses. The OpenAI GPT model of the same period reads text left to right, so each word is understood only through the words before it. BERT uses context on both sides of every word, starting from the bottom of the network.^[1]

The obstacle is the training objective. Teaching a model by asking it to predict the next word only works in one direction, and a model that can see the answer has nothing to learn. BERT's solution is to hide some of the input words and train the model to recover them from the surrounding words.^[1] Because the missing word is not visible in the input, the model has to use both the left and the right context to guess it. That is what makes deep bidirectionality trainable.

After pre-training, the model is fine-tuned on a labelled dataset for a particular task, such as answering questions or judging whether two sentences follow from each other.

Two sizes were released. BERT-Base has 12 layers, 768 hidden units and 12 attention heads, for 110 million parameters; BERT-Large has 24 layers, 1024 hidden units and 16 heads, for 340 million parameters. Both were pre-trained on Wikipedia and BookCorpus.^[1]

Results

Google reported a score of 93.2% F1 on the SQuAD v1.1 question-answering benchmark, above the previous best of 91.6% and above the human-level score of 91.2%. On GLUE, a set of 9 language-understanding tasks, the improvement was 7.6 percentage points in absolute terms.^[2] The open-sourcing announcement credits BERT with state-of-the-art results on 11 natural language processing tasks.^[2]

Google also stressed how little computing fine-tuning needs: a state-of-the-art question-answering system could be trained in about 30 minutes on a single Cloud TPU, or in a few hours on a single GPU.^[2] Pre-training is expensive and done once; adapting the result to a new task is cheap. That split is what let many research groups and small teams work with the model.

Releases and later versions

All code and models came out under the Apache 2.0 license. A multilingual model covering 104 languages and a Chinese model followed in November 2018; Whole Word Masking models arrived in May 2019, and 24 smaller BERT models in March 2020.^[3] The first release contained English-only pre-trained models, and the source code was built on TensorFlow.^[2] The paper won the Best Long Paper award at NAACL 2019.^[5]

In Google Search and afterwards

On 25 October 2019 Google announced that BERT was being used in Search, where it helped the system understand about one in 10 searches in U.S. English. Google described it as "the biggest leap forward in the past five years".^[4]

Seen from a distance, BERT marks the point where a single pre-trained encoder plus a small amount of task-specific training replaced the practice of building a separate system for every task. Its masking objective and its fine-tuning recipe became standard parts of the language-model toolkit, and the sequence of multilingual, Chinese, whole-word and smaller models shows the direction the work took: wider language coverage and lower cost of use.