Serving Tencent Hunyuan-A13B-Instruct with vLLM

Who this is for

This is for developers with a multi-GPU server who want to serve Hunyuan-A13B-Instruct behind an OpenAI-compatible endpoint. After working through it you should have a running server, know how to switch thinking on and off, and have tested a tool call end to end.

What vLLM provides

vLLM is a high-performance inference and serving framework for large language models. Its PagedAttention approach reduces wasted GPU memory, which helps with long texts and many concurrent requests. It exposes completions and chat completions through an OpenAI-compatible API, so existing OpenAI client code can use it^[1].

Environment and installation

The reference environment is Ubuntu 22.04, Python 3.12, CUDA 12.8 and PyTorch 2.8.0^[1]. Point pip at the Tsinghua mirror, then install vllm==0.10.0, openai==1.90.0 and modelscope==1.25.0^[1]. Before going further, check that the NVIDIA driver, the CUDA version and the CUDA build of PyTorch agree, using nvidia-smi and a short PyTorch check^[1].

pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
pip install vllm==0.10.0
pip install openai==1.90.0
pip install modelscope==1.25.0

Download the weights

Download from ModelScope with snapshot_download. Set cache_dir to a real path before running it^[1][2].

from modelscope import snapshot_download
model_dir = snapshot_download('Tencent-Hunyuan/Hunyuan-A13B-Instruct', cache_dir='请修改我！！！', revision='master')

Thinking mode

Hunyuan-A13B-Instruct thinks by default. Two ways turn it off for a request: pass {"chat_template_kwargs": {"enable_thinking": false}} in extra_body, or add a /no_think prefix to the prompt^[1]. To get the reasoning as a separate reasoning_content field from the server, start the server with --enable-reasoning and the correct reasoning parser^[1].

Start the server

The serve command runs vLLM's OpenAI API server on port 8085 across four GPUs. It sets --max-model-len 8192 and --gpu_memory_utilization 0.9, and it names the model Hunyuan-A13B-Instruct^[1]. The official memory recommendation is four H20 GPUs^[1]. When the server is ready, the log prints Application startup complete^[1].

CUDA_VISIBLE_DEVICES=0,1,2,3 python -m vllm.entrypoints.openai.api_server \
  --model /请修改我！！！/Tencent-Hunyuan/Hunyuan-A13B-Instruct \
  --served-model-name Hunyuan-A13B-Instruct \
  --max-model-len 8192 \
  --tensor-parallel-size 4 \
  --port 8085 \
  --trust_remote_code \
  --gpu_memory_utilization 0.9 \
  --enable-reasoning \
  --reasoning-parser hunyuan_a13b

A few parameters matter most. --tensor-parallel-size sets the number of tensor-parallel shards, and it is commonly set to the GPU count^[1]. --max-model-len caps input plus output for one request, and larger values use more memory and can trigger out-of-memory errors^[1]. If that happens, lower --gpu_memory_utilization to 0.8 or 0.7^[1]. --trust_remote_code loads custom code from the model repository, which some models need^[1].

Check that the server answers with curl http://127.0.0.1:8085/v1/models^[1].

curl http://127.0.0.1:8085/v1/models

Call it from Python

The thinking-mode test sends three prompts: a greeting, the AIME 2025 problem and a quicksort request. All three answers were judged good, with clear self-identification, a correct math answer, and commented code with test cases^[1]. Keep the loop short when you reuse it, because each request with thinking on produces a long reasoning trace.

Create the OpenAI client with base URL http://127.0.0.1:8085/v1 and the placeholder key EMPTY^[1].

from openai import OpenAI

openai_api_key = "EMPTY"
openai_api_base = "http://127.0.0.1:8085/v1" # 使用正确的端口

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

The chat request sets temperature=0 and max_tokens=8000, and it turns thinking on through chat_template_kwargs^[1].

response = client.chat.completions.create(
    model="Hunyuan-A13B-Instruct",
    messages = [{"role": "user", "content": prompts[i]}],
    temperature=0,
    max_tokens=8000,
    extra_body={
        "chat_template_kwargs": {"enable_thinking": True}, # 默认开启思考，设置为False则关闭
    }
)

The response message keeps reasoning_content and content apart^[1]. In the math test, the prompt was an AIME 2025 problem with the answer 70, and the final answer was 70^[1].

Tool calling

Tool calling needs two more flags. The serve command adds --tool-call-parser hunyuan_a13b and --enable-auto-tool-choice^[1][2].

CUDA_VISIBLE_DEVICES=0,1,2,3 python -m vllm.entrypoints.openai.api_server \
  --model /请修改我！！！/Tencent-Hunyuan/Hunyuan-A13B-Instruct \
  --served-model-name Hunyuan-A13B-Instruct \
  --max-model-len 8192 \
  --tensor-parallel-size 4 \
  --port 8085 \
  --trust_remote_code \
  --gpu_memory_utilization 0.9 \
  --tool-call-parser hunyuan_a13b \
  --enable-auto-tool-choice \
  --enable-reasoning \
  --reasoning-parser hunyuan_a13b

The official README has two outdated items. In vLLM, the tool-call parser value is hunyuan_a13b, and the reasoning parser is already integrated^[1][2].

The example weather function is simulated. It takes location, date and unit and returns a fixed description. Real use needs an external weather API^[1].

normalized_unit = "°C" if unit == "celsius" else "°F"
# 这里返回一个模拟结果，真实场景可替换为外部 API 调用
return f"{location}{date}多云，气温 28{normalized_unit}，湿度 70%，东北风 3 级。"

The flow has three steps. Append the assistant message that contains tool_calls. Append a tool message with the matching tool_call_id. Send a second request so the model can write the final answer^[1]. In the Shenzhen test, the final answer reported cloudy weather, 28°C, 70% humidity and a level-3 northeast wind, based on the simulated result^[1].

Adapted from the Hunyuan-A13B-Instruct vLLM deployment lesson in Datawhale's open-source course *self-llm*^[1], released under the Apache-2.0 License.