Serving Qwen3 with vLLM

Who this is for and what you will be able to do

This is for readers who write some Python and want to serve the Qwen3-8B model locally. After working through it, you will download the model with ModelScope, run a short offline inference with vLLM, start an OpenAI-compatible API server, and send it requests with both curl and Python.

What vLLM is and how it differs from plain Transformers

vLLM is an inference and serving system for large language models. It manages the KV cache with PagedAttention to reduce memory waste. It uses asynchronous processing and continuous batching, so throughput is higher when many requests arrive at once. It is compatible with Hugging Face models, supports distributed inference across multiple GPUs, and includes an OpenAI-compatible API server^[1].

Plain Transformers scripts suit experiments. vLLM is an engine built for running a model as a service. The two parts of this guide follow that split: offline inference first, then a server.

Preparation: environment, dependencies and the model

The test environment in the guide is Ubuntu 22.04, Python 3.12, CUDA 12.4 and PyTorch 2.5.1, and it assumes PyTorch with CUDA is already set up^[1]. Install that first if you do not have it.

Before installing dependencies, switch pip to the Tsinghua mirror, then install ModelScope and vLLM:

python -m pip install --upgrade pip
pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple

pip install modelscope
pip install vllm

The order matters: upgrade pip, configure the pip mirror, then install both packages^[1]. The guide also notes that the AutoDL cloud GPU platform offers a prepared Qwen3 environment image, which saves the setup work^[1].

The model is downloaded with ModelScope. The first argument of snapshot_download is the model name, and cache_dir is the download directory. This script saves Qwen3-8B under /root/autodl-tmp:

from modelscope import snapshot_download

model_dir = snapshot_download('Qwen/Qwen3-8B', cache_dir='/root/autodl-tmp', revision='master')

Change cache_dir to your own path before running it^[1]. The download takes a while.

Step 1: offline inference to check the engine

Offline inference calls the model directly from a script, without a network interface. Import LLM, the main class for offline inference, and SamplingParams, which controls randomness and diversity in generation^[1].

Set an environment variable at the top of the script. It tells vLLM to download from ModelScope; otherwise it fetches from Hugging Face^[1]:

# 自动下载模型时，指定使用modelscope; 否则，会从HuggingFace下载
os.environ['VLLM_USE_MODELSCOPE']='True'

The get_completion helper has these defaults: temperature 0.6, top_p 0.95, top_k 20, min_p 0, max_tokens 4096 and max_model_len 8192^[1].

def get_completion(prompts, model, tokenizer=None, temperature=0.6, top_p=0.95, top_k=20, min_p=0, max_tokens=4096, max_model_len=8192):

Thinking mode and non-thinking mode

Qwen3 has thinking enabled by default, which the guide compares to QwQ-32B. Setting enable_thinking=True in tokenizer.apply_chat_template, or leaving it at the default, puts the model into thinking mode^[1]. The example below formats the prompt with the tokenizer's chat template:

if __name__ == "__main__":
    # 初始化 vLLM 推理引擎
    model='/root/autodl-tmp/Qwen/Qwen3-8B' # 指定模型路径
    tokenizer = AutoTokenizer.from_pretrained(model, use_fast=False) # 加载分词器

    prompt = "给我一个关于大模型的简短介绍。"
    messages = [
        {"role": "user", "content": prompt}
    ]
    text = tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True,
        enable_thinking=True  # 是否开启思考模式，默认为 True
    )

The code comment gives the recommended sampling parameters for thinking mode as temperature 0.6, TopP 0.95, TopK 20 and MinP 0^[1].

To turn thinking off, set enable_thinking to False. The comment recommends temperature 0.7, TopP 0.8, TopK 20 and MinP 0 for non-thinking mode^[1].

In thinking mode, the response has two parts: the reasoning between <think> and </think>, and the final answer after </think>^[1].

Run the script with:

python vllm_model.py

Step 2: start an OpenAI-compatible server

vllm serve starts an OpenAI-protocol server. By default it listens on http://localhost:8000, hosts one model at a time, and provides a model list endpoint plus completions and chat completions endpoints^[1].

• completions does basic text generation: the model continues a given prompt.
• chat completions is for conversation, such as chatbots^[1].

The server options are --host and --port for the address, --model for the model, --chat-template for the chat template, --served-model-name for the name clients see, and --max-model-len for the maximum length^[1].

Add --enable-reasoning to turn on thinking mode. It must be used together with --reasoning-parser. A reasoning model's response then includes an extra reasoning_content field with the reasoning steps, and the deepseek_r1 parser handles the <think> ... </think> format^[1].

The full command for Qwen3-8B is:

VLLM_USE_MODELSCOPE=true vllm serve /root/autodl-tmp/Qwen/Qwen3-8B --served-model-name Qwen3-8B --max_model_len 8192 --enable-reasoning --reasoning-parser deepseek_r1

When loading finishes, the terminal shows that the service has started.

Step 3: test the API with curl and Python

List the models first:

curl http://localhost:8000/v1/models

Next, test the Completions endpoint. This request asks the model to continue an arithmetic question:

curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen3-8B",
        "prompt": "我想问你，5的阶乘是多少？<think>\n",
        "max_tokens": 1024,
        "temperature": 0
    }'

The Chat Completions endpoint takes a list of messages:

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen3-8B",
        "messages": [
            {"role": "user", "content": "我想问你，5的阶乘是多少？<think>\n"}
        ]
    }'

From Python, you can use the OpenAI SDK and point base_url at the local server^[1]:

# vllm_openai_chat_completions.py
from openai import OpenAI
openai_api_key = "sk-xxx" # 随便填写，只是为了通过接口参数校验
openai_api_base = "http://localhost:8000/v1"

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

chat_outputs = client.chat.completions.create(
    model="Qwen3-8B",
    messages=[
        {"role": "user", "content": "什么是深度学习？"},
    ]
)
print(chat_outputs)

python vllm_openai_chat_completions.py

The sk-xxx in the client example is a placeholder. The comment says it only passes the interface parameter check^[1]. If you expose the server to other users, you need access control of your own.

While the server handles requests, it prints logs and statistics for each one to its console. Check there first when debugging^[1].

Common questions

The server cannot find the model at startup. Make sure the model path in the command matches your download directory. The guide's path is /root/autodl-tmp/Qwen/Qwen3-8B, and its download script points cache_dir at /root/autodl-tmp^[1].

**--enable-reasoning is set but no reasoning comes back.** The flag must be paired with --reasoning-parser, and the parser must match the model's output format. The guide's example uses deepseek_r1^[1].

Thinking-mode answers are too long. Lower max_tokens, or turn thinking off with enable_thinking=False. The max_tokens default in the guide's helper is 4096^[1].

Do I need a real API key? No. The sk-xxx in the example only passes the client-side parameter check^[1].

Limits

The guide uses an 8B model with pinned environment versions. A larger model, a newer CUDA or a newer PyTorch may require changes. A larger max_model_len uses more GPU memory, so set it to fit your card. The guide downloads from ModelScope, and if you switch to Hugging Face, the VLLM_USE_MODELSCOPE setting must change with it^[1].

Next steps

For the full meaning of each vllm serve option, read the official vLLM documentation. For more on the model itself, see this site's entry on Qwen3.

Adapted from the "02-Qwen3-8B vLLM deployment and calls" lesson of Datawhale's open-source self-llm project^[1], released under the Apache-2.0 License.