Qwen1.5

What it is

Qwen1.5 is a generation of the Qwen large language model family developed by Alibaba. It is made up of base models, which are trained to continue text, and chat models, which are tuned to answer questions and follow instructions. The sizes run from 0.5 billion to 72 billion parameters, and the series was later extended with a 110 billion parameter model. A mixture-of-experts model was added to the lineup as well.^[1]

Origins

The Qwen team published the release on its blog, dated 4 February 2024, and the Qwen repository lists 5 February 2024. Alibaba released it just before Chinese New Year.^[1][2] Qwen1.5 succeeded the first Qwen generation, . It was in turn succeeded by Qwen2 in June 2024.^[2]

Main features

The models share a common context length of up to 32,768 tokens. A token is the unit of text a model reads and writes, roughly a word or part of one, and the context length is how much text the model can take into account at once. Every model in the family supported this length.^[1]

The chat models were aligned with DPO and PPO, two methods that adjust a model after its initial training so that its answers match what people prefer.^[1] The team also reported that Qwen1.5-MoE-A2.7B, released on 28 March 2024, was the first mixture-of-experts model in the Qwen series. In a mixture-of-experts model, each input is routed to a subset of specialist sub-networks, so only part of the model runs for each token.^[2]

Benchmarks

The Qwen team published results for its largest base model. Qwen1.5-72B scored 77.5 on MMLU, a test of knowledge across many subjects, 84.1 on C-Eval, a Chinese knowledge benchmark, 79.5 on GSM8K, a set of grade-school math problems, and 41.5 on HumanEval, a coding test. It outperformed Llama2-70B on all of the benchmarks listed.^[1]

For chat models, Qwen1.5-72B-Chat scored 8.61 on MT-Bench, a multi-turn conversation test, and a 27.18 win rate on AlpacaEval 2.0, which compares answers with a reference model. These results were ahead of Claude-2.1 and GPT-3.5-Turbo-0613, but still well behind GPT-4-Turbo.^[1]

The multilingual evaluation covered 12 languages, including Arabic, Spanish, French, Japanese, Korean, Vietnamese and Thai.^[1]

Where it could be used

The models were made available through Hugging Face, ModelScope, DashScope and together.ai.^[1] Integration with Hugging Face transformers 4.37.0 let developers load the models without trust_remote_code, a setting that runs code shipped with a model and is normally a security concern. Support was also available in vLLM, SGLang, Ollama, LM Studio and llama.cpp, along with Int4 and Int8 quantizations in the GPTQ, AWQ and GGUF formats. Quantization stores the model's numbers at lower precision so it needs less memory.^[1]

Its place in development

Qwen1.5 was the step between the first Qwen generation and Qwen2, which was released in June 2024.^[2]