Serving Kimi K2.5 with vLLM

Kimi K2.5 is Moonshot's flagship-scale open model — the kind whose deployment is defined less by installation than by hardware: the reference setup spans eight GPUs. This page walks through the full path with vLLM: offline inference, an OpenAI-compatible server with reasoning and tool-call parsers, and a function-calling test built around questions reasoning models typically fail.

Setup at flagship scale

vLLM provides the machinery this deployment leans on: PagedAttention for KV cache management, continuous batching for throughput, an OpenAI-protocol server, and tensor-parallel inference across GPUs ^[1]. The reference environment is Ubuntu 22.04, Python 3.12, CUDA 12.8, PyTorch 2.8.0 ^[1]. Dependencies are just modelscope and vllm via a Tsinghua mirror ^[1]:

python -m pip install --upgrade pip
pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple

pip install modelscope
pip install vllm

Download from ModelScope ^[1]:

from modelscope import snapshot_download

model_dir = snapshot_download('moonshotai/Kimi-K2.5', cache_dir='/root/autodl-tmp', revision='master')

Why eight GPUs: at flagship scale the weights alone occupy hundreds of gigabytes of VRAM, and KV cache, activations, and runtime overhead come on top. Tensor parallelism also carries a cost — inter-GPU communication bandwidth shapes decode speed, and more cards means more synchronization per generated token. When planning capacity, budget KV cache headroom for the context length too: at a max_model_len of 8192, cache grows quickly as concurrency climbs.

Offline inference

The script follows the standard vLLM pattern with one consequential number: tensor_parallel_size=8 ^[1].

from vllm import LLM, SamplingParams
from transformers import AutoTokenizer
import os
import json

# 自动下载模型时，指定使用modelscope; 否则，会从HuggingFace下载
os.environ['VLLM_USE_MODELSCOPE']='True'

def get_completion(prompts, model, tokenizer=None, temperature=0.6, top_p=0.95, top_k=20, min_p=0, max_tokens=4096, max_model_len=8192):
    stop_token_ids = [151645, 151643]
    # 创建采样参数。temperature 控制生成文本的多样性，top_p 控制核心采样的概率，top_k 通过限制候选词的数量来控制生成文本的质量和多样性, min_p 通过设置概率阈值来筛选候选词，从而在保证文本质量的同时增加多样性
    sampling_params = SamplingParams(temperature=temperature, top_p=top_p, top_k=top_k, min_p=min_p, max_tokens=max_tokens, stop_token_ids=stop_token_ids)  # max_tokens 用于限制模型在推理过程中生成的最大输出长度
    # 初始化 vLLM 推理引擎
    llm = LLM(model=model, tokenizer=tokenizer, max_model_len=max_model_len,trust_remote_code=True, tensor_parallel_size=8)  # max_model_len 用于限制模型在推理过程中可以处理的最大输入和输出长度之和。
    outputs = llm.generate(prompts, sampling_params)
    return outputs

The prompt formatting uses the chat template with the thinking toggle ^[1]:

if __name__ == "__main__":
    # 初始化 vLLM 推理引擎
    model='/root/autodl-fs/moonshotai/Kimi-K2.5' # 指定模型路径
    tokenizer = AutoTokenizer.from_pretrained(model) # 加载分词器

    prompt = "给我一个关于大模型的简短介绍。"
    messages = [
        {"role": "user", "content": prompt}
    ]
    text = tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True,
        enable_thinking=True  # 是否开启思考模式，默认为 True
    )

    outputs = get_completion(text, model, tokenizer=None, temperature=0.6, top_p = 0.95, top_k=20, min_p=0)  # 对于思考模式，官方建议使用以下参数：temperature = 0.6，TopP = 0.95，TopK = 20，MinP = 0。

Official thinking-mode parameters: temperature 0.6, TopP 0.95, TopK 20, MinP 0 ^[1]. The generation loop prints raw output ^[1]:

    # 输出是一个包含 prompt、生成文本和其他信息的 RequestOutput 对象列表。
    # 打印输出。
    for output in outputs:
        prompt = output.prompt
        generated_text = output.outputs[0].text
        print(f"Prompt: {prompt!r}, \nResponse: {generated_text!r}")

Responses split into a thinking process between <think> and </think> and the final answer after the closing marker ^[1].

The thinking process consumes token budget as well. With thinking enabled, a large share of completion tokens goes to reasoning steps rather than the final answer. If max_tokens is set too low, generation can stop mid-thought and leave no answer at all — the tell-tale sign is a response ending in reasoning text; the fix is either a larger max_tokens or asking the model to keep its reasoning brief.

The server: parsers and a path-as-model-name quirk

Kimi-2.5 speaks the OpenAI protocol; vllm serve exposes list-models, completions, and chat completions at http://localhost:8000, one model at a time ^[1]. The launch line carries the model-parallel setting, the context cap, and two parsers ^[1]:

VLLM_USE_MODELSCOPE=true vllm serve /root/autodl-fs/moonshotai/Kimi-K2.5 -tp 8 --max_model_len 8192 --trust-remote-code --tool-call-parser kimi_k2 --reasoning-parser kimi_k2

With --enable-reasoning set, --reasoning-parser is mandatory: the model emits a reasoning_content field with the steps behind its conclusion, and the parser extracts and formats them ^[1]. One quirk worth knowing before writing clients: no --served-model-name is set here, so the model list reports the model id as the full filesystem path, and every request must use that path as the model name ^[1]:

curl http://localhost:8000/v1/models

A completions test with the path-as-name and the <think>\n suffix ^[1]:

curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "/root/autodl-fs/moonshotai/Kimi-K2.5",
        "prompt": "我想问你，什么是深度学习算法？<think>\n",
        "max_tokens": 1024,
        "temperature": 0
    }'

That call consumed 9 prompt tokens and 343 completion tokens ^[1]. The chat client looks the same, with the path in the model field ^[1]:

from openai import OpenAI
openai_api_key = "sk-xxx" # 随便填写，只是为了通过接口参数校验
openai_api_base = "http://localhost:8000/v1"

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

chat_outputs = client.chat.completions.create(
    model="/root/autodl-fs/moonshotai/Kimi-K2.5",
    messages=[
        {"role": "user", "content": "什么是大模型？"},
    ]
)
print(chat_outputs)

With the parser configured, the chat response separates cleanly: the final answer in content, the thinking steps in a distinct reasoning field ^[1]. The sample call generated 847 completion tokens from a 12-token prompt — a visible demonstration of how much of a reasoning model's budget goes to thinking ^[1].

The path-as-model-name detail is the easiest place for client code to break: a hard-coded string that embeds the server directory layout fails the moment the deployment location changes. A sturdier habit is to fetch the model list once at startup and read the id from the response. Note also that the <think>\n suffix belongs to the completions path; on chat completions the reasoning parser handles it automatically.

The function-calling test

The demo defines four functions — add, mul, compare, count_letter_in_string — each with a typed tool description and required fields ^[1]. The client sends the prompt with tools attached and reads the model's choice ^[1]:

def function_call_playground(prompt):
    messages = [{'role': 'user', 'content': prompt}]
    response = client.chat.completions.create(
        model="/root/autodl-fs/moonshotai/Kimi-K2.5",
        messages = messages,
        temperature=0.01,
        top_p=0.95,
        stream=False,
        tools=tools)

    # print(response)
    func1_name = response.choices[0].message.tool_calls[0].function.name
    func1_args = response.choices[0].message.tool_calls[0].function.arguments
    func1_out = eval(f'{func1_name}(**{func1_args})')

Temperature sits at 0.01 — tool selection should be deterministic ^[1]. The function result returns to the conversation as a tool message, and a second call produces the natural-language answer ^[1]:

    messages.append(response.choices[0].message)
    messages.append({
        'role': 'tool',
        'content': f'{func1_out}',
        'tool_call_id': response.choices[0].message.tool_calls[0].id
    })
    # print(messages)
    response = client.chat.completions.create(
        model="/root/autodl-fs/moonshotai/Kimi-K2.5",
        messages=messages,
        temperature=0.01,
        top_p=0.95,
        stream=False,
        tools=tools)
    return response.choices[0].message.content

The two test prompts target known reasoning-model blind spots — counting r's in "strawberry" and comparing 9.11 with 9.9 — and both came back correct through the tools: three r's, and 9.11 smaller than 9.9 ^[1]. The API backend prints logs and statistics for every request, which is where to look when a call misbehaves ^[1].

Adapted from the Kimi-2.5 vLLM deployment section of Datawhale's open-source guide *self-llm*^[1], released under the Apache-2.0 License.