Deploying Kimi K2.5 with SGLang

Who this is for

This is for developers with an eight-GPU workstation or server who want to run Kimi K2.5 on their own hardware. After working through it you should have a serving endpoint and have tested plain chat, streaming output and a tool call that makes two model requests.

Why run it locally

There are three reasons. Conversations and data stay inside the local network, which helps with privacy and compliance. Latency is more predictable, which matters for frequent interactive use or automated agent tasks. For high request volumes, a local server can also cost less than paying for API calls^[1].

Environment

The tested environment is eight NVIDIA RTX PRO 6000 cards with 96GB each, Ubuntu 22.04, CUDA 12.8, Python 3.12 and PyTorch 2.8.0^[1]. Create a Python 3.12 conda environment and upgrade pip first^[1].

conda create -n sglang python=3.12 pip -y
conda activate sglang
python -m pip install --upgrade pip

On AutoDL, source /etc/network_turbo turns on network acceleration for slow downloads, and unset http_proxy; unset https_proxy turns it off again^[1].

A plain pip install sglang can run into compatibility problems with Kimi 2.5. The official deployment guidance recommends installing SGLang from source and pinning a specific cuDNN build^[1][2].

pip install "sglang @ git+https://github.com/sgl-project/sglang.git#subdirectory=python"
pip install nvidia-cudnn-cu12==9.16.0.29

Download the model

Download moonshotai/Kimi-K2.5 from ModelScope with snapshot_download. Set cache_dir to a path with enough space^[1].

from modelscope import snapshot_download
# 下载 Kimi-K2.5 模型
model_dir = snapshot_download(
    'moonshotai/Kimi-K2.5', 
    cache_dir='./kimi_models', 
    revision='master'
)
print(f"模型下载完成，存储路径为: {model_dir}")

Start the server

Run sglang serve with --model-path pointing to the folder that holds config.json and the weight files, in .bin or .safetensors format^[1]. --tp 8 splits the model across eight GPUs with tensor parallelism^[1]. --trust-remote-code runs the model's custom code, and the two parser flags, --tool-call-parser kimi_k2 and --reasoning-parser kimi_k2, handle Kimi's tool calls and reasoning output^[1].

sglang serve --model-path /your/path/to/Kimi-K2.5 --tp 8 --trust-remote-code --tool-call-parser kimi_k2 --reasoning-parser kimi_k2

Keep this terminal open. Run the client scripts from a second terminal^[1].

Startup time and memory

Startup is slow. In the measurement, reading the weights from disk took about 1 hour 45 minutes. After that came a long initialization stage with multi-GPU P2P communication, tensor splitting and MoE expert arrangement, with no log output for tens of minutes. The total was 8714 seconds, about 2 hours 25 minutes^[1].

Memory is reported two ways. SGLang showed about 72.33G per card, which covers the weights and required buffers. nvidia-smi showed about 81.4G per card, because SGLang preallocates static graphs and a large KV cache pool^[1]. Success looks like HTTP 200 responses and successful API calls in the log^[1].

Chat and streaming

Startup can take a long time, so check the log for HTTP 200 responses and successful API calls before you start testing^[1]. The chat client uses base URL http://127.0.0.1:30000/v1, the placeholder key empty and the model name default. A short instruction returns the full result at once^[1].

import openai
client = openai.Client(
    base_url="http://127.0.0.1:30000/v1", 
    api_key="empty"
)
response = client.chat.completions.create(
    model="default",
    messages=[
        {"role": "user", "content": "你好 Kimi，请介绍一下你自己，并确认一下你现在的版本。"}
    ],
    temperature=0.7
)
print(f"Kimi 的回答：\n{response.choices[0].message.content}")

For long answers, set stream=True and print each chunk as it arrives. Streaming suits long text and interactive chat because it feels faster to the user^[1].

for chunk in response:
    content = chunk.choices[0].delta.content
    if content is not None:
        print(content, end="", flush=True)

Tool calling

Tool calling takes two model requests. The first returns a structured tool call, and the second turns the tool result into a normal answer^[1].

The example tool is get_weather, with location and unit parameters. unit accepts only celsius or fahrenheit^[1].

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "获取一个给定地点的当前实时天气情况，如查询不到则返回错误信息。",
        "parameters": {
            "type": "object",
            "properties": {
                "location": {"type": "string", "description": "城市名，例如：北京, Tokyo, London"},
                "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
            },
            "required": ["location", "unit"]
        }
    }
}]

The first request sets tools and tool_choice="auto"^[1].

response = client.chat.completions.create(
    model="default",
    messages=[{"role": "user", "content": user_query}],
    tools=tools,
    tool_choice="auto"
)

The example weather tool calls the OpenWeatherMap API. The API key must stay private and must not be shared publicly^[1]. In the test run, the model planned a call for Baoji in celsius, and the final answer reported about -0.7°C with overcast clouds^[1].

Adapted from the Kimi K2.5 SGLang deployment lesson in Datawhale's open-source course *self-llm*^[1], released under the Apache-2.0 License.