Baichuan-7B

What it is

Baichuan-7B is a 7-billion-parameter language model built for both Chinese and English and released as an open-source model in June 2023 ^[1][2]. It was the first model released by Baichuan ^[1][2]. The "7B" in the name refers to the number of weights in the network, a rough measure of model size that also determines how much computing power is needed to run it.

Design and training

The model was trained on about 1.2 trillion tokens ^[1]. A token is the unit a language model actually reads — roughly a word or a piece of a word — so the training set corresponds to an enormous amount of text. Training used a context length of 4,096 tokens, meaning the model saw text in stretches of that length while learning; Baichuan said its tests showed the model extends well beyond 5,000 tokens, and the model card lists a context length of 4,096 tokens ^[1]. The context window is what limits how much text — a passage, a conversation, a document excerpt — the model can take in at once when answering.

Architecturally, Baichuan-7B uses a standard Transformer design following LLaMA, with rotary position embeddings, SwiGLU activations and RMSNorm pre-normalisation ^[1]. These are the same building blocks used in many open language models of its generation. Rotary position embeddings encode where a word sits in a sequence by rotating its representation, which lets the model reason about order; SwiGLU is a variant of the feed-forward activation function inside each Transformer block; and RMSNorm pre-normalisation normalises the signal before each sub-layer to keep training stable.

Its tokenizer is a SentencePiece BPE tokenizer with a 64,000-token vocabulary ^[1]. Byte-pair encoding splits text into frequently occurring fragments rather than whole words, and a large vocabulary of this kind lets the model cover both Chinese and English in relatively few tokens per sentence.

Results

Baichuan reported benchmark scores under 5-shot testing, meaning each question was preceded by five worked examples in the prompt. On C-Eval the average was 42.8, and on the C-Eval hard subset the average was 31.5; on MMLU the average was 42.3 ^[1]. C-Eval is a Chinese evaluation suite, MMLU a broad multiple-choice test spanning many subjects, and the hard subset is the more difficult portion of C-Eval. The model also averaged 36.24 on Gaokao single-choice questions, drawn from China's university entrance examination, and 34.44 on AGIEval four-choice questions ^[1]. These numbers are the model card's own reported averages across each benchmark; they describe a small open model's performance rather than a state-of-the-art result.

Licence and use

The source code is released under the Apache 2.0 licence, and the model supports commercial use after registering and obtaining written authorisation from Baichuan ^[1]. That split — permissive code, conditional use of the weights — became a common pattern for open Chinese models. Wikipedia notes that the model was used by university researchers ^[2].

Where it sits

Baichuan-7B was followed by Baichuan-13B, a larger sibling, and then by Baichuan 2, released on 6 September 2023 ^[1]. Baichuan 2 was announced in an update dated 6 September 2023 as a new-generation open-source model in 7B and 13B sizes ^[1]. Baichuan-7B therefore marks the starting point of a model line that expanded in size and was refreshed within the same year.