Qwen-VL

What it is

Qwen-VL is a family of vision-language models: models that take an image as well as text as input and produce text. It was released in August 2023, and the accompanying paper was posted to arXiv on 24 August 2023^[1][2]. Its starting point is the Qwen language model, which the team extended so that it could also work with pictures.

How it was built

The design adds four things to the language model: a visual receptor, an input-output interface, a three-stage training pipeline and a cleaned multilingual multimodal corpus^[1]. In practice the visual receptor is the part that turns an image into something the language model can process, and the interface handles passing information between the two halves. The language model itself was initialized from Qwen-7B^[4]. The models take images at an input resolution of 448×448, a choice the team says promotes fine-grained text recognition, document question answering and bounding-box annotation^[4].

What it can do

Beyond ordinary image captioning and visual question answering, Qwen-VL can ground objects in images — that is, point to where a described thing is — and read text in images. The team obtained these abilities by aligning image, caption and bounding-box data during training, so that a region of a picture, the words describing it and its coordinates are learned together^[1]. The 448×448 input resolution is aimed at exactly the uses where small details matter: reading text, answering questions about documents, and marking out bounding boxes^[4].

Results

The authors reported new results for generalist models of similar size on visual benchmarks, in both zero-shot and few-shot settings^[1]. On three standard benchmarks, Qwen-VL scored 78.8 on VQAv2 (dev), 58.6 on OK-VQA and 59.3 on GQA, while the chat-tuned Qwen-VL-Chat scored 78.2, 56.6 and 57.5 on the same three^[4]. Qwen-VL-Chat also outperformed existing vision-language chatbots on real-world dialogue benchmarks^[1].

Variants, licensing and availability

The family comes in several versions with different terms. The Base and Chat variants are released under the Tongyi Qianwen license, while the larger Qwen-VL-Plus and Qwen-VL-Max variants are proprietary^[2]. Code, demos and models were made publicly available^[1].

Where it sits in the line

Qwen-VL was succeeded by Qwen2-VL on 29 August 2024^[3]. Within the Qwen series it marks the point where a text-only language model line was extended into models that also see, and the open release of the Base and Chat versions, together with code and demos, made that step something outside researchers could build on rather than only read about^[1][2].