Baidu releases ERNIE 2.0, beating BERT and XLNet on 16 tasks

Background

Pre-trained language models were a fast-moving area in 2019, and BERT and XLNet were the reference models that new systems were compared against. ERNIE 2.0 approached the problem of training by adding training tasks over time rather than training once on a fixed set.

What happened

The ERNIE 2.0 paper was posted to arXiv on 29 July 2019. Baidu also released English pre-trained models and fine-tuning code on GitHub.^[1][2]

Multi-task learning means training one model on several objectives at the same time, so that what it learns for one objective can help with the others. Pre-training tasks are the training objectives applied to unlabelled text, and choosing which tasks to use, and in what order, is one of the main design decisions in such a model. Benchmarks such as GLUE are useful because every model is tested on the same tasks with the same scoring rules, so readers can compare results directly. A single average score makes comparison quick, although it hides differences between individual tasks.

Key details

ERNIE 2.0 is a continual pre-training framework. Continual pre-training means a model keeps learning in stages, and this framework adds pre-training tasks one after another through multi-task learning, in which several objectives are trained together.^[1]

The model outperformed BERT and XLNet on 16 tasks, including seven GLUE tasks and nine Chinese tasks.^[2] GLUE is a standard English benchmark that combines several language-understanding tasks into one score.

The work was published at AAAI 2020, a major artificial intelligence conference.^[3]

In December 2019 Baidu became the first team to score above 90 on the GLUE benchmark with the latest version of ERNIE. MIT Technology Review noted that the average person scores around 87 points on GLUE.^[4]

Impact

The GLUE result, the first score above 90 by any team at that point, was a notable benchmark milestone. The paper's publication at AAAI 2020 put the method before the wider research community.^[2][3][4]