Baichuan 3

What it is

Baichuan 3 is a closed large language model released by the Chinese company Baichuan on 29 January 2024, with more than one hundred billion parameters^[1]. It was published as a closed model and made available for trial on the company's website, baichuan-ai.com^[1]. Parameter count is a rough measure of size: it is the number of weights the model adjusts during training, and a model of this scale needs substantial computing to train and to serve.

How it was trained

Baichuan said the pre-training data included a medical dataset of more than one hundred billion tokens^[1]. After pre-training, the model was tuned with a combination of reinforcement learning from human feedback (RLHF) and reinforcement learning from AI feedback (RLAIF), inside an iterative reinforcement learning process^[1]. RLHF adjusts a model towards answers that human raters prefer; RLAIF replaces part of that human judgement with judgements produced by another model. For data, Baichuan said it drew on search-engine experience to filter out duplicate and low-quality material^[1]. At inference time, the company said it optimised medical prompts^[1].

Reported benchmark results

On English benchmarks such as MMLU, Baichuan said the model reached about 90% of GPT-4's level^[1]. MMLU is a multiple-choice test of knowledge across many academic subjects. On Chinese benchmarks such as CMMLU and GAOKAO, Baichuan's own evaluation put it well ahead of GPT-3.5 and above GPT-4^[1]; CMMLU is a Chinese multi-subject multiple-choice test, and GAOKAO uses the national college entrance examination as its question source. On mathematics and code benchmarks, and on the multi-turn conversation test MT-Bench and the instruction-following test IFEval, it beat GPT-3.5 and Claude and trailed only GPT-4^[1].

Medical capability

Medical ability was the centre of the announcement: Baichuan called Baichuan 3 the strongest Chinese large model for medical capability^[1]. It said the model surpassed GPT-4 on Chinese medical tasks such as MCMLE, MedExam and CMExam, and came close to GPT-4 on English medical tasks such as USMLE and MedMCQA^[1]. These are largely medical examination and question-answering sets, so they measure exam-style medical knowledge rather than day-to-day clinical work.

Writing

Baichuan claimed a more than fivefold improvement in the usability of the model's classical poetry writing, and said its literary ability decisively beats GPT-4^[1].

What the claims amount to

The benchmark figures in the release are Baichuan's own, and they compare Baichuan 3 with GPT-4 and GPT-3.5^[1]. Read together, they describe a model built to be strong in Chinese and in medicine rather than uniformly ahead on every English benchmark: it claims a lead over GPT-4 on Chinese tests and Chinese medical tasks, but only about 90% of GPT-4 on MMLU, and second place behind GPT-4 on maths, code and instruction following^[1]. The dedicated medical dataset, the medical prompt optimisation at inference time and the poetry claim point the same way — towards Chinese-language and vertical capability as the differentiator.