ERNIE 3.0

ERNIE 3.0 is a large pre-trained language model whose paper was submitted to arXiv on 5 July 2021^[1]. It has 10 billion parameters and was trained on a 4TB corpus that combines plain text with a large-scale knowledge graph^[1]. The paper lists 22 authors, with Yu Sun and Shuohuan Wang named first^[1].

Why it was built

The authors argued that GPT-3-style models trained on plain text lacked linguistic and world knowledge^[1]. ERNIE 3.0's answer was to keep a very large text corpus but add a large-scale knowledge graph to pre-training, so the model meets structured world knowledge alongside ordinary language^[1]. A knowledge graph stores facts as entities and the relations between them, which is a different shape of information from running text; mixing the two is meant to give the model something to draw on that raw sentences alone do not state explicitly.

How it works

Two kinds of network are fused in one model. An auto-encoding network hides part of its input and is trained to recover it, a setup that suits language understanding; an auto-regressive network predicts the next token in a sequence, a setup that suits generation. ERNIE 3.0 combines the two so that a single model can handle both language understanding and generation^[1]. In practice this means the same set of weights can be asked either to classify, match or extract from a passage, or to continue and produce text, rather than needing separate models for the two jobs.

Results

The model outperformed prior state-of-the-art models on 54 Chinese NLP tasks^[1]. Its English version ranked first on the SuperGLUE benchmark as of 3 July 2021, scoring 90.6 against a human baseline of 89.8^[1]. SuperGLUE is a collection of difficult language-understanding tasks; a score above the human baseline indicates that the model's answers agreed with the reference labels more often than the human comparison did.

Scaling up: ERNIE 3.0 Titan

In December 2021 the line was scaled up to the 260-billion-parameter ERNIE 3.0 Titan^[2]. The Titan paper was submitted to arXiv on 23 December 2021 and lists 29 authors^[2]. Its abstract reports that the model outperforms state-of-the-art models on 68 NLP datasets^[2], describes it as the largest Chinese dense pre-trained model so far^[2], and says training used the PaddlePaddle platform together with an online distillation framework^[2]. Distillation in this setting means a smaller or student model is trained to imitate the behaviour of a much larger one while training proceeds, which is one way to make an extremely large model usable in practice.

Where it sits

The ERNIE 3.0 model powered ERNIE Bot when the product opened to invited testers in March 2023, before an upgrade to ERNIE 3.5 in June 2023^[3]. That trajectory is what makes the release notable: a knowledge-enhanced pre-training recipe first published as a research paper became the engine behind a public-facing chatbot, and then a starting point for the larger Titan scale-up^[1][2][3].