Baichuan releases the open Baichuan 2 models

Background

Baichuan had already published open models before this release. Its first generation included Baichuan-7B, and the next step was a second generation with larger training data and more released material. ^[1]

What happened

On 6 September 2023 Baichuan released Baichuan 2, its next-generation open model family. It came in 7B and 13B sizes. ^[1] China Daily reported the launch of the two open-source large language models on the same day. ^[4]

The models were trained on 2.6 trillion tokens. ^[2] A token is a small piece of text, often a word or part of a word, so this figure describes how much text the models read during training.

Key details

For the 13B Base model, Baichuan reported 59.17 on MMLU, an English multi-subject knowledge test, and 61.97 on CMMLU, its Chinese counterpart. ^[2] These are the company's own published results.

Baichuan also released intermediate training checkpoints for research. A checkpoint is a saved copy of a model partway through training, which lets researchers see how its abilities develop as training goes on. ^[2]

Commercial use was allowed on application, for services with under 1 million daily active users. ^[2]

A technical report was published on arXiv as arXiv:2309.10305, titled 'Baichuan 2: Open Large-scale Language Models'. ^[3]

Two benchmark names in the figures need explaining. MMLU is a multiple-choice test across many school and university subjects, so a score reflects broad knowledge rather than skill in one area. CMMLU applies the same idea to Chinese-language material. Scores from different test settings, such as the number of worked examples given in the prompt, cannot be compared directly.

Significance

The release continued Baichuan's approach of publishing its models openly, along with the training details, research checkpoints and licence terms that let other people evaluate them. The report on the launch by China Daily gave the release a wider public record outside the company's own channels. ^[4] For readers, it is a dated, documented example of a Chinese company releasing a family of large language models in two sizes, with its evaluation figures and terms published alongside them.