Serving Qwen3-VL with vLLM

Deploying a multimodal model adds one layer a text-only stack never faces: inputs are no longer just a token stream — images and videos enter the context through a vision encoder. This page puts Qwen3-VL-4B-Instruct on vLLM and verifies three workload types in sequence — daily Q&A, image description, and video inference — with a VRAM budget of one RTX 5090 ^[1].

Where the model sits

Qwen3-VL-4B-Instruct is the vision-language entry of the Qwen3-VL series, strong on text understanding and generation, visual perception and reasoning, and spatial plus video-dynamic understanding ^[1]. Evaluations put it on par with GPT-5-Mini and Claude-4-Sonnet across STEM, VQA, OCR, video understanding, and agent tasks ^[1]. Note that it is a non-reasoning model — for thinking mode, the same-size Qwen3-VL-30B-A3B-Thinking or the flagship Qwen3-VL-235B-A22B-Thinking are the upgrades ^[1].

Environment and installation

The reference stack: Ubuntu 22.04, Python 3.12, CUDA 12.8, PyTorch 2.8.0 ^[1]. Driver, CUDA, and the PyTorch CUDA build must match — self-check before continuing ^[1]. Four pinned packages cover everything ^[1]:

pip install vllm==0.11.0
pip install openai==2.3.0
pip install modelscope==1.30.0
pip install qwen_vl_utils==0.0.14

qwen_vl_utils is the preprocessing piece for multimodal inputs, used later in the video section. The model downloads from ModelScope; swap in your own cache_dir ^[1]:

# model_download.py
# 注意修改cache_dir为保存的路径
from modelscope import snapshot_download
model_dir = snapshot_download('Qwen/Qwen3-VL-4B-Instruct', cache_dir='请修改我！！！', revision='master')

print(f"模型下载完成，保存路径为：{model_dir}")

Serving and its parameters

A single-GPU launch is one command, with every flag earning its place ^[1]:

CUDA_VISIBLE_DEVICES=0 python -m vllm.entrypoints.openai.api_server \
  --model 请修改我！！！/Qwen/Qwen3-VL-4B-Instruct \
  --served-model-name Qwen3-VL-4B-Instruct \
  --max-model-len 8192 \
  --tensor-parallel-size 1 \
  --port 8085 \
  --trust_remote_code \
  --gpu_memory_utilization 0.9

CUDA_VISIBLE_DEVICES picks the card; --max-model-len caps per-request input plus output, trading VRAM for headroom and OOM risk; --gpu_memory_utilization is the target VRAM fraction and can rise when OOM strikes; --served-model-name defines what clients pass in the model field; --trust_remote_code is mandatory or some models fail to initialize ^[1]. Seeing Application startup complete means the service is live, OpenAI-compatible, and callable straight from the SDK ^[1].

Images through the OpenAI protocol

Daily Q&A and image description share one SDK entry point. First, a test of character-level precision ^[1]:

from openai import OpenAI

openai_api_key = "EMPTY"
openai_api_base = "http://127.0.0.1:8085/v1" # 使用正确的端口
daily_chat_message = "将“I love Qwen3-VL-4B-Instruct”这句话的所有内容反过来写"

# 实例化OpenAI client
client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

# 日常问答
daily_chat_response = client.chat.completions.create(
    model="Qwen3-VL-4B-Instruct",
    messages = [
        {
            "role": "user",
            "content": daily_chat_message
        }
    ]
)
print(f"Qwen3-VL-4B-Instruct日常问答: {daily_chat_response.choices[0].message.content}")
print("-"*100)

The model reasoned step by step and produced a fully correct character reversal: tcurtsnI-B4-LV-3newQ evol I ^[1]. Image description puts an image_url and text in one message ^[1]:

# 图片描述
image_des_response = client.chat.completions.create(
    model="Qwen3-VL-4B-Instruct",
    messages = [
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg"
                    }
                },
                {"type": "text", "text": "Describe this image."},
            ],
        }
    ]
)
print(f"Qwen3-VL-4B-Instruct图片描述: {image_des_response.choices[0].message.content}")

The output ran deep: a yellow Labrador Retriever, a black-and-white plaid flannel shirt, a blue harness with colorful paw prints, golden sunrise light — even the high-five between dog and owner ^[1]. Human review found both replies solid; against Qwen2-VL-2B-Instruct on the same image, the 4B model won across detail richness, emotional atmosphere, structure, and language expressiveness ^[1]. The protocol-level difference for multimodal work is exactly here — content upgrades from a string to an array where image and text parts interleave, identical to the OpenAI wire format ^[1].

Inside the video pipeline, process_vision_info samples frames at the model-required cadence, frame counts flow through video_kwargs, and the returned metadata carries per-frame positions; the <|video_pad|> placeholder in the prompt expands into a visual token sequence matching the frame count. Denser sampling means finer perception, with context and VRAM rising in step — video tuning is ultimately a balance between clarity and budget.

Offline video inference

Video skips the serve mode and runs through the offline LLM entry. Imports and the multiprocessing method come first ^[1]:

import torch
from qwen_vl_utils import process_vision_info
from modelscope import AutoProcessor
from vllm import LLM, SamplingParams

import os
os.environ['VLLM_WORKER_MULTIPROC_METHOD'] = 'spawn'

qwen_vl_utils converts video messages into the multimodal payload the engine consumes ^[1]:

def prepare_inputs_for_vllm(messages, processor):
    text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    # qwen_vl_utils 0.0.14+ reqired
    image_inputs, video_inputs, video_kwargs = process_vision_info(
        messages,
        image_patch_size=processor.image_processor.patch_size,
        return_video_kwargs=True,
        return_video_metadata=True
    )
    print(f"video_kwargs: {video_kwargs}")

    mm_data = {}
    if image_inputs is not None:
        mm_data['image'] = image_inputs
    if video_inputs is not None:
        mm_data['video'] = video_inputs

    return {
        'prompt': text,
        'multi_modal_data': mm_data,
        'mm_processor_kwargs': video_kwargs
    }

Engine initialization is visibly more aggressive than the serving profile — memory utilization pushed to 0.99 ^[1]:

if __name__ == '__main__':
    messages = [
        {
            "role": "user",
            "content": [
                {
                    "type": "video",
                    "video": "./space_woaudio.mp4",
                },
                {"type": "text", "text": "Describe this video. And guess what is the man going to do?"},
            ],
        }
    ]

    checkpoint_path = "请修改我！！！/Qwen/Qwen3-VL-4B-Instruct"
    processor = AutoProcessor.from_pretrained(checkpoint_path)
    inputs = [prepare_inputs_for_vllm(message, processor) for message in [messages]]

    llm = LLM(
        model=checkpoint_path,
        trust_remote_code=True,
        gpu_memory_utilization=0.99,
        enforce_eager=False,
        tensor_parallel_size=torch.cuda.device_count(),
        seed=0
    )

Sampling is equally tight — temperature 0 and a 128-token cap ^[1]:

    sampling_params = SamplingParams(
        temperature=0,
        max_tokens=128,
        top_k=-1,
        stop_token_ids=[],
    )

    for i, input_ in enumerate(inputs):
        print()
        print('=' * 40)
        print(f"Inputs[{i}]: {input_['prompt']=!r}")
    print('\n' + '>' * 40)

    outputs = llm.generate(inputs, sampling_params=sampling_params)
    for i, output in enumerate(outputs):
        generated_text = output.outputs[0].text
        print()
        print('=' * 40)
        print(f"Generated Response: {generated_text!r}")

Video preprocessing ran at roughly 1270 tokens/s with generation near 78 tokens/s ^[1]. The model read the clip as a high-tech Mission Control Center and described a middle-aged man in a NASA-logo polo speaking and gesturing — video understanding and reasoning both hold up, but the output was truncated at max_tokens=128 ^[1]. That is the reality of video workloads: frames devour VRAM, single-card setups need extreme parameters just to complete the task, and truncation is the price the memory budget charges ^[1].

The deployment advice compresses to two lines: serve image workloads behind the OpenAI endpoint with comfortable margins; run video workloads offline in batches with utilization pinned high and max_tokens budgeted carefully, where per-task throughput matters more than any single complete answer.

Adapted from the Qwen3-VL-4B-Instruct vLLM deployment section of Datawhale's open guide *self-llm*^[1], released under the Apache-2.0 License.