Deploying Qwen2.5-Coder with vLLM

Who this is for

This guide is for developers who want to use Qwen2.5-Coder for code generation and run their own inference service. By the end you should be able to start Qwen2.5-Coder-7B-Instruct with vLLM, send completion and chat requests with curl, and use vLLM's own script to compare throughput with Transformers.

What vLLM is

vLLM is a system for large language model inference and serving. It uses the PagedAttention algorithm to manage the KV cache, which reduces wasted memory^[1]. It also supports asynchronous processing and continuous batching of requests, which raises inference throughput^[1]. vLLM works with Hugging Face models, is compatible with the OpenAI API server, and supports distributed inference across multiple GPUs^[1].

Environment

The test environment was Ubuntu 22.04, Python 3.12, CUDA 12.1 and PyTorch 2.4.0^[1]:

----------------
ubuntu 22.04
python 3.12
cuda 12.1
pytorch 2.4.0
----------------

PyTorch with CUDA support should already be installed; if it is not, install it first^[1]. Next, switch pip to the Tsinghua mirror to speed up downloads, then install the dependencies^[1]:

python -m pip install --upgrade pip
pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple

pip install modelscope==1.20.0
pip install transformers==4.46.2
pip install torch==2.4.0 torchvision==0.19.0 torchaudio==2.4.0 --index-url https://download.pytorch.org/whl/cu121
pip install vllm==0.6.3.post1
pip install accelerate==0.26.0

Install in the order shown. The torch line points at the cu121 index, which matches CUDA 12.1^[1].

Download the model

The model can be downloaded with modelscope. The first argument of snapshot_download is the model name, and local_dir sets where it is saved^[1]. Qwen2.5-Coder-7B-Instruct is about 16 GB, and the download took about 12 minutes in the test^[1]. The script is below; change local_dir to your own path before running it^[1]:

from modelscope import snapshot_download

snapshot_download('Qwen/Qwen2.5-Coder-7B-Instruct', local_dir='/root/autodl-tmp/Qwen2.5-Coder-7B-Instruct')

Offline inference with LLM

The LLM class runs offline inference with the vLLM engine, and SamplingParams sets the sampling options^[1]. You pass a model name or path, and vLLM handles initializing the model and tokenizer for you^[1]. The helper function below wraps this step^[1]:

def prepare_model(model_path,  max_tokens=512, temperature=0.8, top_p=0.95, max_model_len=2048):
    # 初始化 vLLM 推理引擎
    llm = LLM(model=model_path, tokenizer=model_path, max_model_len=max_model_len,trust_remote_code=True)
    return llm

temperature controls how varied the output is, and top_p controls the range of nucleus sampling^[1]. The example also sets stop_token_ids to [151329, 151336, 151338]^[1].

A chat model needs the chat template applied first. The example does this with^[1]:

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

Start an OpenAI-compatible server

vLLM can create a server compatible with the OpenAI API. By default it starts at http://localhost:8000, serves one model at a time, and provides a model list, completions and chat completions^[1]. Use completions to continue a prompt and chat completions for multi-turn dialogue^[1].

Start the server with^[1]:

python -m vllm.entrypoints.openai.api_server --model /root/autodl-tmp/Qwen2.5-Coder-7B-Instruct  --served-model-name Qwen2.5-Coder-7B-Instruct --max-model-len=2048

Among the options, --host and --port set the address, --model sets the model, --served-model-name sets the name clients use, --max-model-len sets the maximum length, and --chat-template sets the chat template^[1]. The name Qwen2.5-Coder-7B-Instruct in the example comes from --served-model-name, so requests must use the same name^[1].

List the models with curl^[1]:

curl http://localhost:8000/v1/models

Test both endpoints with curl

The completions endpoint takes a prompt field, max_tokens caps the output length, and a temperature of 0 makes the output more deterministic^[1]:

curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen2.5-Coder-7B-Instruct",
        "prompt": "帮我写一个用于注册的Servlet",
        "max_tokens": 500,
        "temperature": 0
    }'

The chat endpoint takes a messages array, with a role and content for each message^[1]:

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen2.5-Coder-7B-Instruct",
        "messages": [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "你好，你都会什么编程语言？"}
        ]
    }'

The server logs each request and prints statistics while it handles them^[1].

Compare throughput with Transformers

vLLM ships with a benchmark_throughput.py script. You can use the copy in the repository or download the latest version from GitHub^[1]^[2]. Its main options include --backend, which accepts vllm, hf or mii for vLLM, HuggingFace and Mii; --input-len and --output-len for lengths; --num-prompts for the number of prompts; and --hf-max-batch-size, which applies only to the hf backend and must be set there^[1].

The vLLM backend command and its result:

python benchmark_throughput.py \
	--model /root/autodl-tmp/Qwen2.5-Coder-7B-Instruct \
	--backend vllm \
	--input-len 64 \
	--output-len 128 \
	--num-prompts 25 \
	--seed 2024 \
    --dtype float16 \
    --max-model-len 512

The Transformers backend command and its result:

python benchmark_throughput.py \
	--model /root/autodl-tmp/Qwen2.5-Coder-7B-Instruct \
	--backend hf \
	--input-len 64 \
	--output-len 128 \
	--num-prompts 25 \
	--seed 2024 \
	--dtype float16 \
    --hf-max-batch-size 25

In these two runs, vLLM produced 1024.10 output tokens/s and the HuggingFace backend 722.39 output tokens/s, about 1.4 times as many^[1]. The runs were made on a single RTX 3090 24G. They are not a rigorous benchmark, and results depend on each machine and environment, so the numbers are only a reference. A reliable comparison needs several runs on your own hardware, averaged^[1].