Serving DeepSeek-R1-Distill-Qwen with vLLM

A 7B reasoning model is one of the few LLMs you can realistically run on a single GPU, and vLLM is the shortest path to serving it — one command turns the model into an OpenAI-compatible API. This page walks the full path for DeepSeek-R1-Distill-Qwen-7B: environment, model download, offline inference, API server, and a throughput benchmark.

What vLLM is

vLLM is an efficient inference and deployment service system for large language models ^[1]. Its features map directly onto deployment pain points: the PagedAttention algorithm manages the KV cache efficiently to cut memory waste; asynchronous processing and continuous batching raise throughput; HuggingFace models integrate seamlessly, and the built-in API server speaks the OpenAI protocol; multi-GPU distributed inference is supported for larger models; and the open-source community keeps it moving ^[1].

Environment and dependencies

The reference environment: Ubuntu 22.04, Python 3.12, CUDA 12.1, PyTorch 2.3.0 — the CUDA-enabled PyTorch environment is assumed to be set up already ^[1]. Install dependencies through a Tsinghua PyPI mirror, with every version pinned ^[1]:

python -m pip install --upgrade pip
pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple

pip install modelscope==1.22.3
pip install openai==1.61.0
pip install tqdm==4.67.1
pip install transformers==4.48.2
pip install vllm==0.7.1

The pins matter: inference stacks are version-sensitive, and vllm 0.7.1 pairs with transformers 4.48.2.

Download the model

ModelScope's snapshot_download takes the model name and a cache_dir for the download path ^[1]:

from modelscope import snapshot_download

model_dir = snapshot_download('deepseek-ai/DeepSeek-R1-Distill-Qwen-7B', cache_dir='/root/autodl-tmp', revision='master')

Save as model_download.py, run python model_download.py, and wait — a 7B model is a multi-gigabyte download ^[1].

Offline inference: the first script

Create vllm_model.py. Three things happen: the VLLM_USE_MODELSCOPE environment variable routes automatic downloads through ModelScope; a get_completion helper wraps the sampling parameters and the inference engine; the main block sets the model path and input ^[1]:

# vllm_model.py
from vllm import LLM, SamplingParams
from transformers import AutoTokenizer
import os
import json

# 自动下载模型时，指定使用modelscope; 否则，会从HuggingFace下载
os.environ['VLLM_USE_MODELSCOPE']='True'

def get_completion(prompts, model, tokenizer=None, max_tokens=8192, temperature=0.6, top_p=0.95, max_model_len=2048):
    stop_token_ids = [151329, 151336, 151338]
    # 创建采样参数。temperature 控制生成文本的多样性，top_p 控制核心采样的概率
    sampling_params = SamplingParams(temperature=temperature, top_p=top_p, max_tokens=max_tokens, stop_token_ids=stop_token_ids)
    # 初始化 vLLM 推理引擎
    llm = LLM(model=model, tokenizer=tokenizer, max_model_len=max_model_len,trust_remote_code=True)
    outputs = llm.generate(prompts, sampling_params)
    return outputs

The sampling values follow DeepSeek's official guidance: temperature should sit between 0.5 and 0.7 with 0.6 preferred, and max_tokens is set to 8192 because the thinking process itself consumes a large token budget ^[1].

if __name__ == "__main__":    
    # 初始化 vLLM 推理引擎
    model='/root/autodl-temp/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B' # 指定模型路径
    # model="deepseek-ai/DeepSeek-R1-Distill-Qwen-7B" # 指定模型名称，自动下载模型
    tokenizer = None
    # 加载分词器后传入vLLM 模型，但不是必要的。
    # tokenizer = AutoTokenizer.from_pretrained(model, use_fast=False) 
    
    text = ["请帮我制定个简短的初学者Python学习计划<think>\n", ] # 可用 List 同时传入多个 prompt，根据 DeepSeek 官方的建议，每个 prompt 都需要以 <think>\n 结尾，如果是数学推理内容，建议包含（中英文皆可）：Please reason step by step, and put your final answer within \boxed{}.

    # messages = [
    #     {"role": "user", "content": prompt+"<think>\n"}
    # ]
    # 作为聊天模板的消息，不是必要的。
    # text = tokenizer.apply_chat_template(
    #     messages,
    #     tokenize=False,
    #     add_generation_prompt=True
    # )

    outputs = get_completion(text, model, tokenizer=tokenizer, max_tokens=8192, temperature=0.6, top_p=0.95, max_model_len=2048) # 思考需要输出更多的 Token 数，max_tokens 设为 8K，根据 DeepSeek 官方的建议，temperature应在 0.5-0.7，推荐 0.6

Two input conventions matter: every prompt ends with <think>\n, and for math reasoning the prompt should include — in Chinese or English — "Please reason step by step, and put your final answer within \boxed{}." ^[1]

    # 输出是一个包含 prompt、生成文本和其他信息的 RequestOutput 对象列表。
    # 打印输出。
    for output in outputs:
        prompt = output.prompt
        generated_text = output.outputs[0].text
        if r"</think>" in generated_text:
            think_content, answer_content = generated_text.split(r"</think>")
        else:
            think_content = ""
            answer_content = generated_text
        print(f"Prompt: {prompt!r}, Think: {think_content!r}, Answer: {answer_content!r}")

The output is a list of RequestOutput objects; the script splits each generation on </think> and prints the thinking content separately from the final answer ^[1]. Run it with python vllm_model.py ^[1].

Launch an OpenAI-compatible server

More useful than a script is serving the model. DeepSeek-R1-Distill-Qwen speaks the OpenAI API protocol, so vLLM starts a compatible server in one command — default address http://localhost:8000, one model hosted at a time, with list-models, completions, and chat completions endpoints ^[1]:

python -m vllm.entrypoints.openai.api_server \
  --model /root/autodl-tmp/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B \
  --served-model-name DeepSeek-R1-Distill-Qwen-7B \
  --max-model-len=2048

--model is the local model path, --served-model-name is the name clients see, --max-model-len caps the context length ^[1]. Verify with the model list endpoint ^[1]:

curl http://localhost:8000/v1/models

Then test completions — again with the <think>\n suffix and temperature 0 ^[1]:

curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "DeepSeek-R1-Distill-Qwen-7B",
        "prompt": "我想问你，5的阶乘是多少？<think>\n",
        "max_tokens": 1024,
        "temperature": 0
    }'

The response text reasons step by step and closes with \boxed{120}; usage reported 13 prompt tokens and 164 completion tokens ^[1]. From Python, the api_key value is arbitrary — it only passes the interface parameter check ^[1]:

# vllm_openai_completions.py
from openai import OpenAI
client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="sk-xxx", # 随便填写，只是为了通过接口参数校验
)

completion = client.chat.completions.create(
  model="DeepSeek-R1-Distill-Qwen-7B",
  messages=[
    {"role": "user", "content": "我想问你，5的阶乘是多少？<think>\n"}
  ]
)

print(completion.choices[0].message)

Chat completions works the same way with a messages array ^[1]:

# vllm_openai_chat_completions.py
from openai import OpenAI
openai_api_key = "sk-xxx" # 随便填写，只是为了通过接口参数校验
openai_api_base = "http://localhost:8000/v1"

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

chat_outputs = client.chat.completions.create(
    model="DeepSeek-R1-Distill-Qwen-7B",
    messages=[
        {"role": "user", "content": "什么是深度学习？"},
    ]
)
print(chat_outputs)

Benchmark the throughput

vLLM ships a benchmark_throughput.py script. Useful parameters: --backend selects the vllm, hf, or mii inference backend; --input-len and --output-len set lengths; --num-prompts sets the request count; without --dataset, data is generated automatically ^[1]:

python benchmark_throughput.py \
	--model /root/autodl-tmp/qwen/DeepSeek-R1-Distill-Qwen-7B \
	--backend vllm \
	--input-len 64 \
	--output-len 128 \
	--num-prompts 25 \
	--seed 2025 \
  --dtype float16 \
  --max-model-len 512

At 25 prompts with 64-token inputs and 128-token outputs, the measured result was Throughput: 29.34 requests/s, 5632.68 total tokens/s, 3755.12 output tokens/s ^[1]. Rerun it after any hardware or parameter change to see what the change bought you.

Common failures

Three problems account for most failed deployments of reasoning models. Out of memory: the 7B weights are modest, but the KV cache grows with concurrency and context length — lower --max-model-len first (the example's 2048 is a conservative value), then reduce concurrency. An answer that never arrives: max_tokens is too small or the prompt is missing the <think>\n suffix, so generation is cut off mid-thought. A model-not-found API error: the model field in the request does not match --served-model-name, so the server cannot route the call.

Adapted from the DeepSeek-R1-Distill-Qwen-7B vLLM deployment section of Datawhale's open-source guide *self-llm*^[1], released under the Apache-2.0 License.