Alibaba releases Qwen2, moving most sizes to Apache 2.0

Background

Qwen2 followed Qwen1.5. It was the next generation of Alibaba's Qwen large language model family, and it came with new size options and longer context windows.^[2]

What happened

Alibaba released Qwen2 on Hugging Face and ModelScope. The Qwen repository dates the release to 6 June 2024, and the Qwen team's blog post is dated 7 June 2024.^[2][1] The release included five sizes of pretrained and instruction-tuned models: 0.5B, 1.5B, 7B, 57B-A14B and 72B. The 57B-A14B model uses a mixture-of-experts design, with 14 billion parameters active for each input.^[1]

Key details

The context lengths differ by size. The 0.5B and 1.5B models support 32K tokens, the 57B-A14B supports 64K, and the 7B and 72B support 128K. The instruct models reach 128K with YaRN, a method for extending the length a model can handle.^[1] A token is the unit of text a model reads and writes, roughly a word or part of one.

The training data covered 27 languages beyond English and Chinese.^[1] Grouped Query Attention was used at all sizes, and the two smallest models used tied embeddings.^[1]

The licences also changed. Qwen2-72B kept the Qianwen License. The 0.5B, 1.5B, 7B and 57B-A14B models moved to Apache 2.0, a permissive open-source licence that allows commercial use.^[1]

The Qwen team also compared Qwen2-72B with Llama-3-70B. On the base models, Qwen2-72B scored 84.2 against 79.5 on MMLU, 64.6 against 48.2 on HumanEval, 89.5 against 83.0 on GSM8K, and 51.1 against 42.5 on MATH. These are the team's own reported figures. Smaller models need less memory to run, so the size range let developers choose between quality and the hardware they had.^[1]

A mixture-of-experts model such as the 57B-A14B routes each input to only some of its sub-networks. It has 14 billion active parameters for each input, fewer than its total of 57 billion, which is why the model is described with both numbers.

Impact

Most sizes therefore came under the Apache 2.0 licence, while the largest model kept the Qianwen License.