ERNIE 1.0

What it is

ERNIE 1.0 (Enhanced Representation through kNowledge IntEgration) is the first pre-trained language model Baidu released under the ERNIE name. It came out in March 2019, and both the code and the model were open-sourced on GitHub, where they run on Baidu's PaddlePaddle deep learning framework.^[1] The accompanying paper, by Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng and colleagues, was posted to arXiv on 19 April 2019.^[2]

The idea: masking more than single characters

A pre-trained language model is a network that first reads a very large amount of text on its own, without labels, and only afterwards is adapted to particular tasks such as question answering or sentiment analysis. The usual self-supervised exercise is the masked language model: hide part of the input and ask the model to recover it from the surrounding context. BERT popularised this recipe, and like most Chinese models of the period ERNIE works on characters rather than words.^[1]

ERNIE keeps BERT's masked-language-model pre-training but adds what its authors call knowledge masking: entity-level masking and phrase-level masking, instead of masking only single characters.^[2] Entity-level masking hides entities, which are usually made of several words, and phrase-level masking hides whole multi-word phrases.^[2] The bet is that recovering an entire named entity or an entire phrase demands more than local character statistics — the model has to draw on what it has picked up about how things in the world fit together.

Synced's illustration: where BERT, given a masked character in the word "Harbin", learns to recover that character, ERNIE learns that Harbin is the capital of Heilongjiang and a city known for ice and snow.^[1]

Training data

ERNIE was trained on multi-source Chinese data — encyclopedia articles, news and forum dialogues — and the model is character-based.^[1] Mixing encyclopedic text with news and informal dialogue gives the pre-training a range of registers: the encyclopedia supplies the factual pairings that entity-level masking rewards, while news and forum text supply the ordinary sentence patterns the model also has to handle.

Results

The authors reported new state-of-the-art results on five Chinese NLP tasks, among them natural language inference, semantic similarity, named entity recognition, sentiment analysis and question answering, as well as stronger knowledge inference on a cloze test.^[2][1] The paper's abstract states that ERNIE has more powerful knowledge inference capacity on a cloze test.^[2]

The paper

The arXiv listing names ten authors: Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian and Hua Wu.^[2] It is an eight-page preprint in the Computation and Language (cs.CL) category.^[2]

The name

The name is a nod to BERT's Sesame Street namesake.^[1]

Why it mattered

ERNIE 1.0 is a change to an existing recipe rather than a new architecture: the modification sits in the pre-training objective, which decides what the model is rewarded for learning. Masking entities and phrases tilts that objective towards relations between concepts instead of local character co-occurrence. For Chinese, where word boundaries are not marked by spaces and single characters are often ambiguous, the difference carries extra weight: a character-level model that only ever fills in single characters can score well on surface statistics without ever being forced to commit to a fact. The result also illustrates that pre-training data and masking strategy, not model size alone, move benchmark numbers — and it put a Chinese-language model at the top of Chinese benchmarks.^[2][1]

Where it sits in the line

ERNIE 1.0 was superseded in July 2019 by ERNIE 2.0, which replaced the fixed pre-training tasks with a continual multi-task pre-training framework.^[3]