Baidu and Peng Cheng Lab describe the 260-billion-parameter ERNIE 3.0 Titan

Background

ERNIE is Baidu's line of knowledge-enhanced pre-trained models. The idea behind the name is that a model should learn from knowledge-type signals as well as from raw text, so that entities, relations and other structured information become part of what it absorbs during pre-training rather than being inferred only from word co-occurrence. The family had already reached a 10-billion-parameter version in July 2021^[2]. Titan takes the same framework and applies it at a much larger size.

Parameter count is a rough dial for pre-trained models: more parameters give a network more room to store patterns, but they also make training and serving more expensive. A structural distinction matters here. A dense model uses all of its parameters on every input, while a sparse model — a mixture of experts, for instance — activates only part of the network at a time. At equal parameter counts, the dense design is the more computationally expensive one.

What the paper describes

On 23 December 2021, authors from Baidu and Peng Cheng Laboratory posted a paper to arXiv^[1]. The title, "ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generation", states the agenda: push knowledge-enhanced pre-training to a larger scale, and cover language understanding and language generation within one model instead of treating them as separate problems^[1].

The model has over 260 billion parameters, and the authors describe it as the largest Chinese dense pre-trained model to that point^[1]. It is a scaled-up version of the ERNIE 3.0 framework: the earlier 10-billion-parameter release supplied the framework and the training approach, and Titan carries them to a larger size^[2].

How it was trained

The paper sets out several pieces of the training recipe.

One is a self-supervised adversarial loss^[1]. Adversarial training generally means putting harder cases in front of the model instead of easy, fixed ones. Two things are worth noting about this version: the training signal is self-supervised, so it does not depend on human labels, and the difficulty is constructed adversarially rather than held fixed.

A second is a controllable language modeling loss^[1]. Ordinary language modeling asks a model to predict what comes next; this loss adds constraints that steer the generation process, so that output can be shaped by conditions beyond plain next-token prediction.

A third is an online distillation framework^[1]. Distillation usually means training a smaller model to reproduce what a larger one has learned. Calling it online means this happens inside the same pipeline as pre-training, rather than as a separate stage afterwards.

On the engineering side, the model was built on PaddlePaddle, Baidu's deep learning framework^[1].

Results

The paper reports that the model outperformed state-of-the-art models on 68 natural language processing datasets^[1]. That figure is an aggregate over many tasks rather than a single score, and it covers the understanding and generation workloads the title refers to.

Where it sits

Set beside the July 2021 ERNIE 3.0 release, the clearest change is scale: the same knowledge-enhanced framework moving from 10 billion parameters to more than 260 billion^[2][1]. Beyond scale, the paper makes two choices worth recording. Understanding and generation share one model, and distillation is placed inside the pre-training pipeline, so the question of how a very large model is put to use is written into the training plan from the start^[1]. Building on Baidu's own framework also makes the paper a record of how far that training stack was pushed at this size^[1].