Baidu releases ERNIE, a knowledge-enhanced Chinese language model

Background

By 2019, pre-trained language models, which are networks first trained on large text collections and then applied to specific tasks, had become a central approach in natural language processing. Google's BERT was one of the most widely used of these models.

What happened

In March 2019 Baidu released ERNIE and open-sourced its code and model on GitHub. The model ran on PaddlePaddle, Baidu's deep-learning framework.^[1]

Pre-training is the step in which a model learns general patterns of language from a large amount of unlabelled text, often by predicting words that have been hidden from it. The learned model can then be adapted with a smaller set of labelled examples for a particular task. Several of the Chinese tasks named in this entry are standard NLP tasks. Natural language inference asks whether one sentence follows from another. Semantic similarity asks how close in meaning two sentences are. Named entity recognition finds the names of people, places and organisations in text, and sentiment analysis judges whether a passage is positive or negative. Question-answer matching asks whether a candidate answer fits a question.

Key details

Unlike BERT, which learns from raw character signals, ERNIE directly models prior semantic knowledge units such as entities (names of specific things) and phrases.^[1] Baidu reported that ERNIE outperformed BERT on Chinese natural language inference, semantic similarity, named entity recognition, sentiment analysis and question-answer matching.^[1]

The training data came from encyclopedia articles, news and forum dialogues.^[1] The name ERNIE nods to BERT's Sesame Street namesake.^[1]

The accompanying paper was posted to arXiv on 19 April 2019. It reported new state-of-the-art results on five Chinese NLP tasks.^[2]

Impact

The accompanying paper reported state-of-the-art results on five Chinese NLP tasks, and the code and model were open to anyone on GitHub. The name ERNIE was carried forward to later Baidu models.^[1][2]