Baidu introduces the 10-billion-parameter ERNIE 3.0

Background

Pre-training followed by fine-tuning on individual tasks had become the standard way to build a strong language model: a network is first trained on a very large body of text, then adjusted for each downstream job. The ERNIE 3.0 paper questions two assumptions in that recipe. It argues that large pre-trained models trained only on plain text lack explicit linguistic or world knowledge^[1]. It also argues that purely auto-regressive pre-training — the objective in which a model learns by predicting the next token in a sequence — makes fine-tuning weak on language understanding tasks^[1].

What Baidu announced

On 5 July 2021, Baidu researchers posted the ERNIE 3.0 paper to arXiv^[1]. The model has 10 billion parameters^[1]. Parameters are the learned numbers inside a network, and their count is a rough measure of how large the model is. Training drew on a 4TB corpus of plain text together with a large-scale knowledge graph^[1]. A knowledge graph stores entities and the typed relations between them in a structured form rather than as free-running sentences, so it can supply knowledge that ordinary text leaves implicit.

How it works

The architecture combines two kinds of network: auto-encoding and auto-regressive^[1]. An auto-encoding network is trained to reconstruct parts of an input that have been masked or corrupted, using the surrounding context; to do that well it has to represent the sentence as a whole, which suits understanding tasks such as classification. An auto-regressive network is trained to produce the next token given the preceding ones, which is what text generation requires. ERNIE 3.0 puts the two together so that a single model covers both understanding and generation tasks^[1]. That design is a direct response to the paper's second complaint: if pre-training is purely auto-regressive, the resulting model is aimed at generation, and the authors argue this leaves it weaker once it is fine-tuned for understanding.

Results

The paper reports that the model beat prior state-of-the-art models on 54 Chinese natural language processing tasks^[1]. Its English version placed first on SuperGLUE as of 3 July 2021 with a score of 90.6, against a human baseline of 89.8^[1]. SuperGLUE is a collection of language-understanding tasks that comes with a published human baseline — an estimate of how people score on the same problems — so a result above that number is measured as beating the human estimate rather than merely beating other models.

Why it matters

The paper makes two arguments at once: that pre-training should carry structured knowledge as well as raw text, and that understanding and generation should not be forced into a single auto-regressive objective. ERNIE 3.0 is the working model built to support both claims. The SuperGLUE placement gave the second claim an external check, since it is a public leaderboard rather than an in-house evaluation, while the 54 Chinese tasks showed the same approach on the language Baidu's products mainly serve. Read together, the results describe a model that was, at the time of publication, competitive with the best systems on both a Chinese task suite and an English understanding benchmark^[1].