Mengzi3

What it is

Mengzi3 is an open-source family of large language models from Langboat, released in two sizes, 8B and 13B parameters^[1][2]. A base model of this kind is the model as it comes out of pretraining on raw text, before any tuning for conversation; chat versions are adapted from it afterwards. The 13B base model's weights were uploaded to Hugging Face on 29 March 2024^[1][2].

How it was built

The models use the Llama architecture^[1]. Langboat did not pretrain from an empty starting point: training continued on trillions of multilingual tokens, building on an existing checkpoint instead of starting from scratch^[1]. The architecture fixes the shape of the network, while the training run determines what the model is exposed to; continuing from a checkpoint lets work already spent on earlier training carry over, and the added tokens decide which data and languages the model moves towards. A token is the unit of text the model reads and predicts, roughly a word or a piece of one. The corpus was drawn from web pages, encyclopedias, social media, news and open datasets^[1] — a mix of long reference text and short, informal writing.

Measured performance

In 5-shot evaluation, Mengzi3-13B-Base scores 0.651 on MMLU, 0.588 on CMMLU, 0.776 on OCNLI, 0.631 on GSM8K and 0.287 on HumanEval^[1]. "5-shot" means the prompt gives the model five worked examples before the question it must answer. The MMLU, CMMLU and OCNLI numbers come from FlagEval, the evaluation platform run by BAAI^[1]. The results are uneven across tasks: the model is strongest on OCNLI, a Chinese natural language inference benchmark, and weakest on HumanEval, which tests code generation, at 0.287^[1]. Scores of this kind are mainly useful for comparison — they record how often a model picks the right option or passes a set of test cases, not how it behaves in an open conversation.

A later version

The repository also lists a Mengzi3.5-13B-Base, with MMLU 0.776, CMMLU 0.813 and HumanEval 0.532^[1].

Licence and distribution

The licence is Apache-2.0: the models are open for academic research and free for commercial use^[1][2]. Apache-2.0 is a permissive licence, meaning users may reuse and redistribute the weights, including commercially, as long as the licence terms are kept. Weights are mirrored on Hugging Face, ModelScope, Wisemodel and MindSpore MindFormers^[1], so the same files can be fetched from several hosting services, including one tied to the MindSpore framework.

Where it sits

The 8B chat model followed the base release, with its weights uploaded on 14 September 2024^[2]. When the 13B base weights shipped, the 13B chat model was still listed as forthcoming in the repository^[1]. Langboat also states that its team has not built any application on top of Mengzi3 itself^[1], which leaves the release as a contribution of model weights rather than a packaged product: what others do with it is up to them.