Qwen2-VL

What it is

Qwen2-VL is a vision-language model from Alibaba's Qwen team. A vision-language model takes images, and often video, together with text as input and produces text as output. It was released on 29 August 2024 and succeeded the Qwen-VL family.^[1][2] It is built on the Qwen2 language models.^[1]

Origins

Qwen2-VL is the successor to Qwen-VL, the vision-language family.^[2] It pairs a vision encoder of roughly 600 million parameters, a Vision Transformer, with Qwen2 language models.^[1] The 2B and 7B models were open-sourced under Apache 2.0, while the 72B model was initially available only through an API.^[1]

The weights for the 72B model were opened on 19 September 2024, alongside the Qwen2.5 release. Wikipedia lists this model under the Qwen license.^[3][2]

Main features

Qwen2-VL can understand videos longer than 20 minutes and answer questions about them or hold a dialogue based on them.^[1] It can also act as an agent that operates mobile phones and robots from visual input and text instructions.^[1]

It reads text in images in many languages, including most European languages, Japanese, Korean, Arabic and Vietnamese.^[1] Two technical features support this. Naive Dynamic Resolution maps images of any resolution to a variable number of visual tokens, so a large or detailed image is not forced into one fixed size. Multimodal Rotary Position Embedding, or M-ROPE, splits the rotary position embeddings into temporal, height and width components, so the model can track position in video frames as well as across the image.^[1]

Results

The Qwen team reported state-of-the-art results on MathVista, DocVQA, RealWorldQA and MTVQA, which are benchmarks for mathematical visual reasoning, document understanding, real-world image questions and multilingual text-based visual question answering.^[1] The team also said that the 72B model often surpassed GPT-4o and Claude 3.5 Sonnet in document understanding.^[1] These are the team's own statements about its results.

Its place in development

Qwen2-VL was built on Qwen2 and succeeded Qwen-VL in August 2024.^[1][2] It was followed by Qwen2.5-VL.^[3] The Qwen2-VL line shows how the family moved from a general language model to a vision model that can work with video, text and devices.