Baichuan open-sources Baichuan-13B

Background

Baichuan is a Chinese AI start-up founded by Wang Xiaochuan, the founder of Sogou. Before Baichuan-13B, the company had released Baichuan-7B, which was trained on 1.2 trillion tokens. ^[1]

What happened

In July 2023 Baichuan released Baichuan-13B. ^[2] Two versions were published as open models: Baichuan-13B-Base and Baichuan-13B-Chat. ^[1] The Base version is the model after pre-training on large amounts of text. The Chat version was trained further so that it can answer questions and carry on a conversation.

An open model in this sense means that the model's weights, the numbers that hold what the model has learned, are published so that others can download and run it. A large language model (LLM) is a program that learns the patterns of language from large amounts of text and then generates text in response to input.

Key details

The model was trained on 1.4 trillion tokens, more than the 1.2 trillion used for Baichuan-7B. A token is a small piece of text, often a word or part of a word, so the token count gives a rough sense of how much text the model read during training. ^[1]

Baichuan-13B used ALiBi positional encoding and a context length of 4,096 tokens. Positional encoding tells the model where each word sits in a sentence, and the context length is the amount of text it can consider at one time. ^[1]

Quantised versions in int8 and int4 formats were also released. Quantisation stores the model's numbers with fewer bits, which reduces the memory needed to run it. These versions allowed the model to run on consumer GPUs. ^[1]

Commercial use was allowed after registration, under a community license. ^[1]

Significance

Baichuan-13B shows the approach the company took at the time: publish the model openly, describe its training size and hardware needs, and allow commercial use after a registration step. For readers, it is one concrete example of a Chinese company making a large language model available for anyone to download, test and run on their own hardware.