ERNIE 2.0

What it is

ERNIE 2.0 is a pre-trained language model from Baidu, presented as a framework for learning language representations. The paper describing it was posted to arXiv on 29 July 2019 under the identifier 1907.12412 and later appeared at AAAI 2020 in the Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, issue 05, pages 8968–8975^[1][2]. Its authors are Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu and Haifeng Wang^[1].

Like other pre-trained models of its generation, it is trained on large amounts of text first and then adapted to specific downstream tasks. The difference ERNIE 2.0 proposes lies in what the model is asked to learn during that first stage.

How it works

The core idea is a continual pre-training framework: instead of a fixed set of pre-training objectives, the pre-training tasks are built incrementally and learned through continual multi-task learning^[1]. The tasks are designed to cover lexical, syntactic and semantic levels of language, rather than relying only on word and sentence co-occurrence^[1].

Seven types of pre-training tasks were integrated. Their targets include named entities, sentence order, sentence proximity, semantic similarity and discourse relations^[3]. To let one model handle several kinds of task at once, the input embeddings sum token, sentence, position and task embeddings^[3].

MIT Technology Review described the latest version of Baidu's ERNIE as masking strings of characters rather than single characters^[5]. That version was also trained to tell meaningful character strings from random ones, so it could decide which combinations to mask^[5]. It further takes sentence ordering and the distance between sentences into account, and uses continual training to learn new data and tasks without forgetting earlier ones^[5].

Hao Tian, chief architect of Baidu Research, said the team first thought about the characteristics of Chinese but found the approach applicable beyond it^[5].

Results

The paper reported that the model outperformed BERT and XLNet on 16 tasks, including English GLUE tasks and several Chinese tasks^[1]. As single models, the ERNIE 2.0 variants beat BERT and XLNet on seven GLUE tasks, and beat BERT on all nine Chinese NLP tasks tested, including DuReader reading comprehension; the large model set state-of-the-art results on Chinese tasks^[3].

In December 2019 the latest version of Baidu's ERNIE topped the GLUE leaderboard, making Baidu the first team to score above 90 out of 100; the average person scores around 87^[5]. GLUE is a collection of English language-understanding tasks used as a common yardstick for such models, which is why a leaderboard position attracted attention.

Where it sits

The source code and pre-trained models were released on GitHub in the PaddlePaddle/ERNIE repository^[1][4], so others could download the weights and reproduce or extend the work rather than only read the numbers.

ERNIE 2.0 is the successor to ERNIE 1.0, which appeared in March 2019, and it was followed by ERNIE 3.0 in July 2021^[3][6]. Its place in the line is the step where Baidu moved from training on word and sentence co-occurrence to a schedule of incrementally added tasks spanning entities, sentence relationships and discourse, together with a mechanism for absorbing new data without discarding what earlier training had produced^[1][5].