Alibaba releases the Qwen-VL vision-language models

Background

The Qwen language model was the base on which the Qwen-VL models were built. The release extended it to images, so that the model could take pictures as input alongside text. Vision-language models of this kind combine a visual component with a language model, so a single system can answer questions about what it sees.

What happened

Alibaba released the Qwen-VL vision-language model family in August 2023. The paper describing it was posted to arXiv on 24 August 2023.^[1] The Qwen-VL models were built on the Qwen language model, and the team added a visual receptor, an input-output interface, a three-stage training pipeline and a cleaned multilingual multimodal corpus to make that possible.^[1]

Key details

Beyond captioning and visual question answering, the models can ground objects in an image and read text in it. Grounding means pointing to where a described object is. To do this, the training aligned image, caption and bounding-box data.^[1] A bounding box is a rectangle that marks where an object sits in the picture.

The authors reported results for generalist models of similar size on visual benchmarks, in both zero-shot and few-shot settings. A zero-shot result is one obtained without example tasks, and a few-shot result is one obtained with a handful of examples in the prompt.^[1] The chat-tuned variant, Qwen-VL-Chat, outperformed existing vision-language chatbots on real-world dialogue benchmarks.^[1]

The Base and Chat variants use the Tongyi Qianwen license. The larger Qwen-VL-Plus and Qwen-VL-Max variants are proprietary.^[2] Licensing therefore differed across the family.

Vision-language models of this kind are often judged by how accurately they answer questions about images and how precisely they can point to objects. The zero-shot and few-shot settings mentioned above differ in how much guidance the model receives before it is tested, so results under the two settings should be read separately.

Impact

The authors made the code, demos and models publicly available.^[1] The release extended the Qwen language model to images, with the grounding and text-reading abilities described above. The paper and the open variants are the main records of the release for later readers.^[1]