Alibaba releases Qwen2.5 with 13 open models

Background

Qwen2.5 was the next generation of Alibaba's Qwen large language model family. The release focused on a broad set of sizes and on specialised models.^[1]

What happened

Alibaba released Qwen2.5 on 19 September 2024.^[1] The launch included 13 open-weight models. The general models were Qwen2.5 at 0.5B, 1.5B, 3B, 7B, 14B, 32B and 72B. The coding models were Qwen2.5-Coder at 1.5B, 7B and, later, 32B. The mathematics models were Qwen2.5-Math at 1.5B, 7B and 72B.^[1]

Key details

The models were pretrained on up to 18 trillion tokens. Qwen2.5-Coder was trained on 5.5 trillion code-related tokens.^[1] They support up to 128K tokens of context and can generate up to 8K tokens, in more than 29 languages.^[1] A token is the unit of text a model reads and writes, roughly a word or part of one.

All of the open models are under Apache 2.0, except the 3B model, which uses the Qwen Research license, and the 72B model, which uses the Qwen license.^[1][3] Apache 2.0 is a permissive licence that allows commercial use, subject to conditions such as keeping copyright notices.

By September 2024 Alibaba said it had released more than 100 open-weight models, downloaded over 40 million times.^[3]

Specialised models are trained with a narrower focus, such as writing code or solving mathematics problems. For a developer, two questions usually come first: which size fits the hardware available, and whether the licence allows the intended use. Because the licence differed by model in this release, the answer depends on which size is chosen. The 13 models ranged from 0.5 billion to 72 billion parameters, so a small model could run on modest hardware while a large one needed far more capacity.

Impact

The release gave developers a wide range of sizes to choose from, with specialised models for code and mathematics.