Baichuan-13B

Baichuan-13B is an open-source large language model with 13 billion parameters — 13,264,901,120 of them — published by Baichuan in two versions, a base model and a chat model^[1]. Both were distributed as downloadable weights.

Architecture and training

The model stacks 40 layers with a hidden size of 5,120, 40 attention heads and a vocabulary of 64,000 tokens^[3]. It was trained on 1.4 trillion tokens with a context window of 4,096 tokens^[1]. That token count is larger than the 1 trillion tokens Meta used for LLaMA-13B^[2].

For positional information the model uses ALiBi (attention with linear biases). Instead of encoding positions into the input embeddings, ALiBi tells the model about token order by adding a distance-based term to the attention scores, so the cost of handling position grows more slowly with sequence length. Baichuan said ALiBi carries a smaller computational load than the rotary embedding its previous model, Baichuan-7B, had used^[3].

Benchmark results

In 5-shot evaluation — five worked examples are placed in the prompt before the question is asked — the base model scored 52.4 on C-Eval, 51.6 on MMLU and 55.3 on CMMLU; the chat model scored 51.5, 52.1 and 55.8 respectively^[1]. C-Eval and CMMLU are Chinese-language suites covering many school and professional subjects, while MMLU covers English academic subjects; scores from different suites are only comparable when the evaluation setting is the same.

Baichuan's own head-to-head 5-shot averages against LLaMA-13B put C-Eval at 52.4 versus 28.5, MMLU at 51.6 versus 46.3 and CMMLU at 55.3 versus 31.2^[3]. On generation speed, producing 2,000 tokens averaged 25.4 tokens per second for Baichuan-13B against 19.4 for LLaMA-13B, a 31.6% improvement^[3].

Quantisation and deployment

int8 and int4 quantised versions were offered. Quantisation stores the model's weights at lower numerical precision, which shrinks the memory needed to hold them; the int4 build needs about 9.7GB of GPU memory, enough to run on consumer cards such as the NVIDIA 3090^[1]. Baichuan said the quantised versions show almost no performance loss compared with the unquantised model^[3].

Licence and updates

The weights are released under the Baichuan-13B Model Community License, which permits commercial use after registration and written authorisation; the source code is under Apache 2.0^[1]. The Baichuan-13B-Chat weights were updated on 1 August 2023^[1].

Where it sits

Baichuan-13B is the larger successor to Baichuan-7B^[3], and it was positioned directly against Meta's LLaMA-13B: the two sit in the same parameter class, and Baichuan reported comparisons on Chinese and English benchmarks as well as on generation throughput^[3]. Releasing the weights under a licence that allows commercial use after authorisation, together with int4 support that fits a single consumer graphics card, put a model of this size within reach of teams without large computing budgets^[1][3].