Alibaba releases Qwen2-VL with 20-minute video understanding

Background

Qwen2-VL was the next vision-language model after Qwen-VL. It was built on the Qwen2 language model.^[1][2] Vision-language models of this kind take images or video alongside text and answer in text.

What happened

Alibaba released Qwen2-VL on 29 August 2024.^[1] The release came in several sizes with different access terms. The 2B and 7B models were open-sourced under Apache 2.0, a permissive open-source licence. The 72B model was initially API-only, so developers could use it through a service but not download its weights at first.^[1]

Key details

The model can understand videos longer than 20 minutes, and it can answer questions about them and hold a dialogue based on them.^[1] It can also act as an agent that operates mobile phones and robots from visual input and text instructions.^[1] A visual agent of this kind receives what the camera or screen shows, along with a written instruction, and decides what action to take.

Two technical features are described in the release. Naive Dynamic Resolution maps images of any resolution to a variable number of visual tokens. A visual token is a unit that represents part of an image in the input sequence, so the number of tokens grows with the size of the image. Multimodal Rotary Position Embedding, or M-ROPE, splits rotary position embeddings into temporal, height and width components, so that the model can keep track of order in time as well as position in space.^[1]

A vision-language model has to turn each image, or each video frame, into a sequence that the language part of the model can process, which is why the number of visual tokens matters. More tokens keep more detail but cost more computation, so a method that adapts the token count to the image gives a flexible trade-off. A video is a sequence of frames, so its length in time is a separate challenge from its resolution.

Impact

Qwen2-VL extended the Qwen-VL line with support for long video, for text in many languages and for agent-style use on devices.