ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators

The problem

Most text encoders of this kind are pre-trained with masked language modeling: part of the input is hidden and the model is trained to restore the missing tokens. The difficulty is that the training signal comes only from the positions that were hidden, while the rest of the input contributes nothing to the loss. Each pass over the text therefore yields less supervision than it could.

Replaced token detection

The paper, submitted to arXiv on March 23, 2020 by Kevin Clark, Minh-Thang Luong, Quoc V. Le and Christopher D. Manning, proposes a different pre-training task called replaced token detection. Instead of masking the input, it corrupts the input by replacing some tokens with plausible alternatives sampled from a small generator network, and then trains a discriminative model to predict whether each token was replaced^[1].

The difference is one of coverage. The task is defined over all input tokens, not just the small subset that was masked out, which makes it more efficient than masked language modeling^[1]. In effect the model practises a decision at every position rather than at a handful of blanks, and the representations that are kept are learned by a discriminator that judges tokens rather than by a network that generates them.

Results

Given the same model size, data and compute, the learned representations substantially outperform BERT's^[1].

The gain in efficiency is most visible at small scale: a model trained on one GPU for 4 days outperforms GPT, which was trained using 30 times more compute, on the GLUE benchmark^[1]. GLUE is a collection of natural language understanding tasks used to compare encoders. At larger scale, ELECTRA performs comparably to RoBERTa and XLNet using less than a quarter of their compute^[1].

A larger ELECTRA model performed comparably to RoBERTa and XLNet despite having fewer parameters and one quarter of the training compute, and further training took it past ALBERT on GLUE and to a new state of the art on SQuAD 2.0^[1]. SQuAD 2.0 is a reading-comprehension benchmark that includes questions which have no answer in the passage.

Why it matters

The paper's argument is about where the learning signal comes from. A pre-training objective is a design choice, and it determines how much a model learns from each token of text and from each unit of computation. By turning the objective into a judgement about every token instead of a prediction about a few missing ones, the work shows that a smaller budget of data and compute can be spent more effectively, and that a model with fewer parameters can still stand next to much larger ones.

For readers new to the area, the useful lesson is that pre-training is not a single fixed recipe. Masked language modeling was one way to force a network to absorb the structure of language; replaced token detection is another, and the comparison between them is a reminder that the objective function, not only the architecture or the amount of hardware, shapes what a model ends up knowing.