Alibaba open-sources Qwen-7B, its first open-weight model

Background

When a language model finishes training, what remains is a set of parameter files — the weights — together with the code that runs them. A company can offer such a model only through a chat website or an API, in which case users can send it text but cannot see the model itself, copy it, or run it on their own machines. Publishing the weights instead lets anyone with the necessary hardware download the model, run it offline, fine-tune it on their own data and measure it directly. This is what "open-weight" means, and it is narrower than "open source": the files may be public while the licence still sets conditions on commercial use, redistribution, or both.

In 2023 Alibaba Cloud released the first weights in what became the Qwen family; press coverage described these as small open versions of its Tongyi Qianwen model ^[2].

The first release

On 3 August 2023 the company published Qwen-7B, the first open-weight model in the family ^[1]. China Daily reported that Alibaba Cloud would open-source Qwen-7B and Qwen-7B-Chat, each with 7 billion parameters, as small versions of its Tongyi Qianwen model ^[2]. The pair covered the usual split at the time: a base model trained on raw text, whose job is to continue a passage, and a chat version trained further so that it answers questions and follows instructions.

Growth into a family

Qwen-14B followed on 25 September 2023, and Qwen-1.8B and Qwen-72B on 30 November 2023 ^[1]. The same lineage therefore spanned from 1.8 billion to 72 billion parameters, so that a user could pick a version by the hardware available rather than the other way round.

Training data and specifications

The published pre-training volumes differ by size: 2.2 trillion tokens for Qwen-1.8B, 2.4 trillion for Qwen-7B, and 3.0 trillion tokens for both Qwen-14B and Qwen-72B ^[1]. A token is the unit in which a model reads text — roughly a short word, or a piece of one — so these numbers describe how much text passed through training, not how good the result is. The maximum context, meaning the length of text the model can take in at once, was 32K tokens for Qwen-1.8B, Qwen-7B and Qwen-72B, and 8K for Qwen-14B ^[1].

Qwen-72B scored 77.4 on the 5-shot MMLU benchmark ^[1]. MMLU is a set of multiple-choice questions across many academic subjects, and "5-shot" means five worked examples are placed in the prompt ahead of each question. A score of this kind is a snapshot taken under one evaluation setup; results shift with prompt format and with how answers are extracted, so the number is best read as one published data point rather than a general measure of ability.

Licence terms

Qwen-7B, Qwen-14B and Qwen-72B were released under the Tongyi Qianwen License Agreement, with commercial use granted by application; Qwen-1.8B used a research licence ^[1]. At the time of the first release, China Daily reported that companies with fewer than 100 million monthly active users could use the models free for commercial use, while larger companies had to request a licence from Alibaba Cloud ^[2]. Because the terms were not identical across sizes, anyone deploying one of these models had to check which agreement applied to the version in hand.

Why it matters

Open weights change who can do what with a model. Researchers can run controlled experiments on the exact model rather than on a service someone else operates; small teams can fine-tune on private data without sending that data to an outside API; and anyone can check published claims by re-running the model. A family running from 1.8 billion to 72 billion parameters also gives users a way to trade capability against hardware: the small versions run in modest environments, the largest does not.

The release put a published number next to a downloadable model. Qwen-72B's 77.4 on 5-shot MMLU could be compared, and contested, by anyone who downloaded the weights ^[1]. Having the score and the model it describes available together is what makes an open-weight release useful as a reference point rather than only as a product.