Alibaba open-sources the Qwen3-Omni speech-and-vision model

Background

Speech and vision had been handled by separate models in many systems. An omni-modal model accepts several kinds of input in one network and can respond in speech as well as text, which is useful for voice assistants and for interactive applications that involve video.

What happened

In September 2025 Alibaba released Qwen3-Omni under Apache 2.0. The model is a mixture-of-experts design built around a Thinker-Talker arrangement, with one component for understanding and text, and another for speech output. ^[2][1]

Key details

Qwen3-Omni understands text in 119 languages, takes speech input in 19 languages and produces speech in 10. ^[1]

The Qwen team reported state-of-the-art results on 22 of 36 audio and audio-visual benchmarks, and open-source state of the art on 32. It also said the model's speech recognition, audio understanding and voice conversation were comparable to Gemini 2.5 Pro. ^[1]

The release came with three variants: Instruct, a text-only Thinking model with chain-of-thought reasoning, and a Captioner fine-tuned for detailed audio captions with low hallucination. ^[1]

Impact

For readers, the significance is that a model handling speech, audio, video and text was made available with open weights, so researchers and developers could test and adapt it. Benchmark results are reported by the developer and measure particular tasks, so they are best read as evidence of capability rather than as a complete ranking. ^[1]

A speech system built from separate parts, such as one component to transcribe speech, another to reason and a third to produce a voice, adds delay at each handoff. An omni-modal design tries to run these functions together, so that a reply can begin while the user is still speaking.

Some of the terms in the release are specialized. A codebook is a set of discrete units used to represent audio, and a multi-codebook scheme uses several of them together to describe each moment of speech. A Thinker-Talker design separates the part of the model that understands and reasons from the part that produces the spoken reply, so the two can work as a pipeline.