Qwen releases Qwen3-Next with hybrid linear attention

Background

Most large language models at the time used a standard transformer layout, in which every layer applies full attention over the whole text. Attention over long inputs becomes costly as the text grows, and making models both larger and cheaper to run was an active area of work. The Qwen team's Qwen3-Next took a different route, changing the layout itself. ^[1]

What happened

In September 2025, the Qwen team released Qwen3-Next under the Apache 2.0 license. The weights were posted on Hugging Face. ^[2][1]

Key details

The model has 80 billion parameters in total but activates only 3 billion for each token it processes. This is the core of its efficiency claim: the model holds a large store of parameters, yet each token is processed by only a small part of them. ^[1]

Its layers form a hybrid. Groups of three Gated DeltaNet layers, a linear-attention type, alternate with one Gated Attention layer, and every layer is followed by a mixture-of-experts layer. The sparse MoE part uses 512 experts, with 10 routed and 1 shared expert active for each token. ^[1]

Impact

Pre-trained on 15 trillion tokens, the base model was reported to beat Qwen3-32B-Base on downstream tasks at 10% of that model's training cost. For contexts longer than 32K tokens, Alibaba reported 10 times the inference throughput of Qwen3-32B-Base. ^[1]

Linear attention and sparse expert layers are both ways of reducing the work a model does for each token. Linear attention avoids the cost of comparing every token with every other, while sparse experts activate only a small part of the network. Combining them is a bet that both savings can coexist without weakening the model, and the reported results are the developer's evidence for that bet.

The hybrid layout changes how the cost of a long input grows. In a standard transformer, every layer compares tokens across the whole text, so the work rises quickly with length. Keeping a few full attention layers preserves the ability to compare distant parts of a text directly, while the linear layers handle most of the work at lower cost.