Serving GLM-4.5-Air with vLLM

Who this is for

This is for developers with a Linux server and several NVIDIA GPUs who want to run the GLM-4.5-Air model locally. By the end you should be able to download the weights, run one offline generation, start an OpenAI-compatible server, and call it with curl and the OpenAI Python client.

What vLLM does

vLLM is an inference and serving system for large language models. It manages the KV cache with the PagedAttention algorithm, which reduces wasted memory. It processes requests asynchronously and batches them continuously, which raises throughput. It works with Hugging Face models and provides an OpenAI-compatible API server. It can also run inference across several GPUs using model parallelism^[1].

GLM-4.5-Air is compatible with the OpenAI API protocol, so a standard OpenAI client can talk to a server started with vLLM^[1].

Environment and installation

The reference environment is Ubuntu 22.04, Python 3.10, CUDA 12.4 and PyTorch 2.7.1. PyTorch with CUDA must already be set up. Then upgrade pip, switch the index to a mirror in mainland China, and install modelscope and vllm>=0.10.0^[1].

python -m pip install --upgrade pip
pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple

pip install modelscope
pip install vllm>=0.10.0

Download the model

Keep the cache directory on a disk with enough space for the weights. Set cache_dir before the first run, because the same path is later used for loading.

Use ModelScope's snapshot_download. The first argument is the model name ZhipuAI/GLM-4.5-Air, and cache_dir sets the directory where the weights are stored^[1].

from modelscope import snapshot_download

model_dir = snapshot_download('ZhipuAI/GLM-4.5-Air', cache_dir='your_model_dir', revision='master')

Run one offline generation

The offline path uses two classes. LLM runs the vLLM engine for offline inference, and SamplingParams controls sampling such as temperature and top-p^[1]. Setting os.environ['VLLM_USE_MODELSCOPE']='True' makes vLLM fetch from ModelScope. Without it, vLLM downloads from Hugging Face^[1].

The LLM object in the script uses trust_remote_code=True and tensor_parallel_size=4. The max_model_len argument caps the total of input and output tokens for a request^[1].

llm = LLM(model=model, tokenizer=tokenizer, max_model_len=max_model_len,trust_remote_code=True, tensor_parallel_size=4)

Format the prompt with the tokenizer's apply_chat_template. Thinking mode is on by default through enable_thinking=True^[1].

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True  # 是否开启思考模式，默认为 True
)

For thinking mode, the recommended sampling values are temperature 0.6, TopP 0.95, TopK 20 and MinP 0^[1].

The response has two parts. The reasoning sits between <think> and </think>, and the final answer follows the </think> marker^[1].

Start an OpenAI-compatible server

vllm serve starts a server on http://localhost:8000 by default. One server hosts one model, and it implements a model list endpoint plus completions and chat completions^[1]. Completions continue a prompt, and chat completions handle conversations^[1].

vllm serve /model/ModelScope/ZhipuAI/GLM-4.5-Air \
    --tensor-parallel-size 4 \
    --tool-call-parser glm4_moe \
    --reasoning-parser glm4_moe \
    --enable-auto-tool-choice \
    --served-model-name glm-4.5-air

The --reasoning-parser option is required when --enable-reasoning is set. Reasoning models then return an extra reasoning_content field. The deepseek_r1 parser handles <think> ... </think> content^[1]. The server also logs and reports statistics for each request as it processes them^[1].

Call the server

The two endpoints differ in input format. Completions take a raw prompt and continue it. Chat completions take a list of messages, and the server applies the chat template and returns both content and reasoning_content. Chat applications normally use the second endpoint^[1].

Test the completions endpoint with curl. The example sends model glm-4.5-air, max_tokens 1024 and temperature 0^[1].

curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "glm-4.5-air",
        "prompt": "我想问你，5的阶乘是多少？<think>\n",
        "max_tokens": 1024,
        "temperature": 0
    }'

In the example response, the prompt used 11 tokens and the completion used 449^[1].

For the chat endpoint, use the OpenAI Python client with base_url set to http://localhost:8000/v1. The API key can be any placeholder, because the server only checks that the parameter is present^[1]. The chat response returns the answer in content and the reasoning in reasoning_content^[1].

from openai import OpenAI
openai_api_key = "sk-xxx" # 随便填写，只是为了通过接口参数校验
openai_api_base = "http://localhost:8000/v1"

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

chat_outputs = client.chat.completions.create(
    model="glm-4.5-air",
    messages=[
        {"role": "user", "content": "什么是深度学习？"},
    ]
)
print(chat_outputs)

Adapted from the GLM-4.5-Air vLLM deployment lesson in Datawhale's open-source course *self-llm*^[1], released under the Apache-2.0 License.