Serving GLM-4.7-Flash with vLLM

GLM-4.7-Flash is a compact reasoning model that ships with two features many small models lack: a toggleable thinking mode and native tool calling. Paired with vLLM, it can serve as a local chat endpoint with properly separated reasoning output. This page walks through deployment, the thinking-mode parameters, the API server, and a working function-calling test.

The model and the stack

vLLM is an efficient inference and deployment service system: PagedAttention manages the KV cache to cut memory waste, continuous batching raises throughput, the built-in server speaks the OpenAI protocol, and multi-GPU distributed inference handles larger models ^[1]. The reference environment for GLM-4.7-Flash is Ubuntu 22.04, Python 3.12.3, CUDA 12.8, PyTorch 2.8.0 ^[1]. Dependencies are minimal — vllm 0.10.0 or newer plus modelscope, through a Tsinghua PyPI mirror ^[1]:

python -m pip install --upgrade pip
pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple

pip install modelscope
pip install vllm>=0.10.0

Download the model from ModelScope ^[1]:

from modelscope import snapshot_download

model_dir = snapshot_download('ZhipuAI/GLM-4.7-Flash', cache_dir='your_model_dir', revision='master')

Offline inference with thinking mode

The offline script follows the usual vLLM shape — LLM engine plus SamplingParams — with a few GLM-specific twists ^[1]:

from vllm import LLM, SamplingParams
from transformers import AutoTokenizer
import os
import json

# 自动下载模型时，指定使用modelscope; 否则，会从HuggingFace下载
os.environ['VLLM_USE_MODELSCOPE']='True'

def get_completion(prompts, model, tokenizer=None, temperature=0.6, top_p=0.95, top_k=20, min_p=0, max_tokens=4096, max_model_len=8192):
    stop_token_ids = [151645, 151643]
    # 创建采样参数。temperature 控制生成文本的多样性，top_p 控制核心采样的概率，top_k 通过限制候选词的数量来控制生成文本的质量和多样性, min_p 通过设置概率阈值来筛选候选词，从而在保证文本质量的同时增加多样性
    sampling_params = SamplingParams(temperature=temperature, top_p=top_p, top_k=top_k, min_p=min_p, max_tokens=max_tokens, stop_token_ids=stop_token_ids)  # max_tokens 用于限制模型在推理过程中生成的最大输出长度
    # 初始化 vLLM 推理引擎
    llm = LLM(model=model, tokenizer=tokenizer, max_model_len=max_model_len,trust_remote_code=True, tensor_parallel_size=4)  # max_model_len 用于限制模型在推理过程中可以处理的最大输入和输出长度之和。
    outputs = llm.generate(prompts, sampling_params)
    return outputs

Two details deserve attention. The sampling stack is wider than the usual temperature-plus-top_p pair: top_k limits how many candidate words compete at each step, and min_p filters candidates by a probability threshold — the combination is how the model keeps quality and diversity in balance ^[1]. And tensor_parallel_size=4 shows the engine was configured for four GPUs; on a single-card setup, drop it to 1 ^[1].

The prompt is formatted through the tokenizer's chat template, with the thinking toggle sitting right there in the call ^[1]:

if __name__ == "__main__":
    # 初始化 vLLM 推理引擎
    model='/root/autodl-fs/ZhipuAI/GLM-4.7-Flash' # 指定模型路径
    tokenizer = AutoTokenizer.from_pretrained(model) # 加载分词器

    prompt = "给我一个关于大模型的简短介绍。"
    messages = [
        {"role": "user", "content": prompt}
    ]
    text = tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True,
        enable_thinking=True  # 是否开启思考模式，默认为 True
    )

    outputs = get_completion(text, model, tokenizer=None, temperature=0.6, top_p = 0.95, top_k=20, min_p=0)  # 对于思考模式，官方建议使用以下参数：temperature = 0.6，TopP = 0.95，TopK = 20，MinP = 0。

For thinking mode, the officially recommended parameters are temperature 0.6, TopP 0.95, TopK 20, MinP 0 ^[1]. Turning thinking off is a one-word change — enable_thinking=False — which is handy for simple requests where the extra tokens buy nothing ^[1]. The cost of leaving it on is not just tokens but latency: thousands of thinking tokens stream before the answer starts, which is very visible in interactive settings. A simple rule: keep it off for format-driven tasks like rewriting, translating, and classifying, and on for anything requiring derivation, calculation, or multi-step planning.

The output loop prints raw generations ^[1]:

    # 输出是一个包含 prompt、生成文本和其他信息的 RequestOutput 对象列表。
    # 打印输出。
    for output in outputs:
        prompt = output.prompt
        generated_text = output.outputs[0].text
        print(f"Prompt: {prompt!r}, \nResponse: {generated_text!r}")

The response arrives in two parts: the thinking process wrapped between <think> and </think>, and the final answer after the closing marker ^[1]. The sample run shows the model planning, drafting, and revising an LLM introduction through seven explicit steps before committing to the answer — that visible iteration is the main reason to keep thinking mode on for hard prompts and off for easy ones.

The API server: reasoning and tool parsers

GLM-4.7-Flash speaks the OpenAI protocol, so vllm serve turns it into a compatible endpoint — default address http://localhost:8000, one model at a time, with list-models, completions, and chat completions ^[1]. The interesting flags are the parsers ^[1]:

vllm serve /root/autodl-fs/ZhipuAI/GLM-4.7-Flash \
     --tensor-parallel-size 4 \
     --speculative-config.method mtp \
     --speculative-config.num_speculative_tokens 1 \
     --tool-call-parser glm47 \
     --reasoning-parser glm45 \
     --enable-auto-tool-choice \
     --served-model-name glm-4.7-flash

When --enable-reasoning is set, --reasoning-parser becomes mandatory: the model emits an extra reasoning_content field holding the steps that led to the final conclusion, and the parser extracts and formats them ^[1]. The speculative-config lines enable multi-token prediction to speed up decoding ^[1].

The curl test uses the now-familiar reasoning-model conventions — a <think>\n prompt suffix and temperature 0 ^[1]:

curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "glm-4.7-flash",
        "prompt": "我想问你，5的阶乘是多少？<think>\n",
        "max_tokens": 1024,
        "temperature": 0
    }'

The response returns 120 with its reasoning inline, spending 11 prompt tokens and 421 completion tokens ^[1]. From Python, the chat response separates the two: the final answer sits in content while the thinking steps arrive in a distinct reasoning field — no regex required when the parser is configured ^[1]:

from openai import OpenAI
openai_api_key = "sk-xxx" # 随便填写，只是为了通过接口参数校验
openai_api_base = "http://localhost:8000/v1"

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

chat_outputs = client.chat.completions.create(
    model="glm-4.7-flash",
    messages=[
        {"role": "user", "content": "什么是算法？"},
    ]
)
print(chat_outputs)

A working function-calling test

Tool calling is where small models often embarrass themselves, so it is worth verifying explicitly. The demo defines four Python functions — add, mul, compare, and count_letter_in_string — each mirrored by a tool description with typed parameters and required fields ^[1]. The client loop sends the user prompt with the tools attached and reads back the model's choice ^[1]:

def function_call_playground(prompt):
    messages = [{'role': 'user', 'content': prompt}]
    response = client.chat.completions.create(
        model="glm-4.7-flash",
        messages = messages,
        temperature=0.01,
        top_p=0.95,
        stream=False,
        tools=tools)

    # print(response)
    func1_name = response.choices[0].message.tool_calls[0].function.name
    func1_args = response.choices[0].message.tool_calls[0].function.arguments
    func1_out = eval(f'{func1_name}(**{func1_args})')

Note the near-zero temperature of 0.01 — tool selection wants determinism, not creativity ^[1]. After the function runs, the assistant message and the tool result go back into the conversation, and a second call turns the raw result into prose ^[1]:

    messages.append(response.choices[0].message)
    messages.append({
        'role': 'tool',
        'content': f'{func1_out}',
        'tool_call_id': response.choices[0].message.tool_calls[0].id
    })
    # print(messages)
    response = client.chat.completions.create(
        model="glm-4.7-flash",
        messages=messages,
        temperature=0.01,
        top_p=0.95,
        stream=False,
        tools=tools)
    return response.choices[0].message.content

Two deliberately tricky prompts — counting the letter r in "strawberry" and comparing 9.11 with 9.9 — both came back right through the tools: three occurrences of r, and 9.11 correctly identified as the smaller number ^[1]. These are exactly the questions reasoning models get wrong when answering from instinct, which is why routing them through tools is the meaningful test. When running everything, the API backend prints logs and statistics for each request ^[1].

Two operational notes: single-GPU users must drop tensor_parallel_size to 1 or the engine will fail or exhaust memory, and the demo's use of eval to run the model-nominated function is fine for a playground but should become a name-to-function dictionary mapping in production, since eval on model output is an injection risk.

Adapted from the GLM-4.7-Flash vLLM deployment section of Datawhale's open-source guide *self-llm*^[1], released under the Apache-2.0 License.