BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

The problem

Many language models read text in one direction, so the representation of a word can only draw on the words before it. For many understanding tasks, such as judging the meaning of a sentence or answering a question about a passage, the words on both sides of a position matter. The BERT paper aimed to build representations that use context from both directions at once. ^[1]

The approach

BERT uses only the encoder half of the Transformer, so every word can attend to every other word in the input. Parameters are the adjustable numbers inside a network, and more parameters usually means more capacity. Two sizes were released: BERT-Base, with 110 million parameters, and BERT-Large, with 340 million. ^[1]

It is pre-trained on two tasks. The first is masked language modelling, in which some words are hidden and the model predicts them from the context on both sides. The second is next-sentence prediction, which asks the model to judge whether two sentences follow each other. ^[1]

The paper was submitted to arXiv on 11 October 2018 by Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova of Google AI Language. It was published at NAACL 2019, where it received the best long paper award. ^[1]

A helpful way to picture masked language modelling is as a fill-in-the-blank exercise. The model sees a sentence with some words covered and must reconstruct them from the words around them on both sides. Because the right answer depends on context from both directions, the model is pushed to build representations that capture meaning, rather than only guessing a likely next word. ^[1]

The results

BERT set a new state of the art on eleven natural language processing tasks, including raising the GLUE benchmark score to 80.5%. GLUE is a collection of language understanding tasks used to compare models. Google released the code and pre-trained models, and in 2019 began using BERT in Google Search. ^[1]

The results also show the value of a shared starting point. A team with a limited labelled dataset does not need to design and train a large language model from scratch. It can take a pre-trained model, adjust it for its own task, and reuse the same foundation across many applications. That practical benefit is a large part of why the approach spread. ^[1]

Pre-training is what makes this practical. The large body of raw text available to any team can supply the training signal, so the expensive general stage is done once and then shared across many uses. ^[1]

Why it matters

BERT became a common building block of natural language processing research. Researchers could download the same public pre-trained model and fine-tune it rather than training from scratch. Because it has only an encoder, though, it cannot easily be prompted or generate high-quality text. ^[2]