TellWorldAI World Archive
Ask中文

Home›Practice›Practice

Practice

Deploying Tencent Hy4 flash with Transformers, vLLM and SGLang

Tencent's Hy4 flash can be loaded with Transformers or served with vLLM or SGLang. This guide covers the install steps, the no_think switch, the Docker commands and an OpenAI-compatible client.

Who this is for and what you need

This guide is for developers who want to run Tencent's Hy4 flash on their own machines or servers. It covers three routes: loading the model with Transformers for local inference, serving it with vLLM or SGLang, and calling the server through an OpenAI-compatible client. By the end you should know which variant to pick, what the reasoning switch does, and where a deployment is most likely to stall.

What Hy4 flash is

Tencent describes Hy4 flash as its most cost-effective Hy model to date.[1] The model has 220B total parameters, including 64B Engram parameters, and it activates 12B parameters per token.[1] It accepts text and image input, generates text, and supports a context window of up to 1M tokens, or 1,048,576 tokens.[1]

The language backbone has 42 layers: one dense feed-forward layer and 41 mixture-of-experts layers. Each token activates 16 of the 512 routed experts, plus one shared expert.[1] Attention interleaves sparse attention with sliding-window attention at a 1:3 ratio.[1] For images, a vision encoder built on SigLIP2-So400m-Patch16-Res512 connects to the backbone through an MLP projector.[1]

A built-in drafter called DFly2, with 5 layers and 0.85B parameters, supports speculative decoding.[1]

Pick a variant

Three variants are published on Hugging Face, ModelScope and GitCode.[1] Hy4 flash is the instruct model. Hy4 flash-FP8 is an FP8-quantized instruct model, and the vLLM and SGLang examples below both load it. Hy4 flash-base is a pre-trained base model and is not a chat model, so the examples in this guide do not apply to it.[1]

The model is released under the Tencent Hy License. Read the license file before you use the model in a product.[1]

Reasoning mode and sampling

For the instruct models, reasoning_effort accepts two values. "high" is the default and turns thinking on. "no_think" returns direct answers.[1] Tencent recommends temperature 0.9 and top_p 1.0.[1] Every example below uses no_think, so the replies come back without a thinking step.[1]

Option 1: load it with Transformers

Support for Hy4 flash is not yet in the Transformers main branch. Install from the pull request commit until it is merged:[1]

pip install "transformers[torch,vision] @ git+https://github.com/huggingface/transformers.git@refs/pull/50903/head"

Load the processor and model with the model ID tencent/Hy4-flash. The example uses device_map="auto" for the model:[1]

from transformers import AutoModelForImageTextToText, AutoProcessor

model_id = "tencent/Hy4-flash"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(model_id, device_map="auto")

After that, you pass reasoning_effort="no_think" to apply_chat_template, and call model.generate with max_new_tokens=256.[1]

Option 2: serve with vLLM

For production serving, Tencent recommends vLLM or SGLang.[1] The vLLM route uses the prebuilt image vllm/vllm-openai:hy4-flash.[1] The command below loads Hy4-flash-FP8 with tensor parallel size 8, so check your GPU count before you copy it. It also turns on speculative decoding with the DFly method and 5 speculative tokens, selects the FLASHMLA_SPARSE attention backend, and serves on port 8000 under the name hy4-flash.[1]

docker run --gpus all \
  -p 8000:8000 \
  --ipc=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:hy4-flash tencent/Hy4-flash-FP8 \
    --tensor-parallel-size 8 \
    --speculative-config '{"num_speculative_tokens":5,"method":"dfly"}' \
    --attention-backend FLASHMLA_SPARSE \
    --tool-call-parser hy_v4 \
    --reasoning-parser hy_v4 \
    --enable-auto-tool-choice \
    --port 8000 \
    --served-model-name hy4-flash

Option 3: serve with SGLang

The SGLang image is lmsysorg/sglang:hy4-flash, and it is multi-arch, so it runs on both x86 and Arm.[1] Pull it first, then start the server with sglang serve. The command uses --trust-remote-code, loads tencent/Hy4-flash-FP8 with --tp 8, enables the DFLY speculative algorithm with block size 6, and listens on 127.0.0.1 port 8000.[1]

docker pull lmsysorg/sglang:hy4-flash

docker run --gpus all \
  -p 8000:8000 \
  --ipc=host \
  lmsysorg/sglang:hy4-flash \
  sglang serve \
    --trust-remote-code \
    --model-path tencent/Hy4-flash-FP8 \
    --tp 8 \
    --cuda-graph-backend-prefill disabled \
    --speculative-algorithm DFLY \
    --speculative-dfly-block-size 6 \
    --host 127.0.0.1 \
    --port 8000

Call the server with the OpenAI client

Once the vLLM server is up, point the OpenAI client at http://127.0.0.1:8000/v1 and pass EMPTY as the API key.[1] Set the model name to hy4-flash. To turn thinking off for a request, pass reasoning_effort as no_think in chat_template_kwargs.[1]

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="hy4-flash",
    messages=[
        {"role": "user", "content": "Hello! Can you briefly introduce yourself?"},
    ],
    temperature=0.9,
    top_p=1.0,
    extra_body={"chat_template_kwargs": {"reasoning_effort": "no_think"}},
)
print(response.choices[0].message.content)

Speculative decoding and speed

DFly2 reuses the target model's KV cache and the positions its indexer selects. Tencent says each draft token attends to at most about 2,000 keys regardless of context length, adds no separate KV memory, and does not take part in prefill.[1] Tencent also says that megakernels, 4-bit quantization and DFly2 together push decoding past 150 tokens per second.[1]

Quantization and testing

AngelSlim is Tencent's compression toolkit, and it covers quantization, low-bit quantization and speculative sampling.[1] For end-to-end testing without a large GPU setup, Tencent also provides a GGUF compressed version of the model.[1]

Fine-tuning

The model repository includes a finetuning pipeline, documented in its finetune directory.[1]

Common problems

  • If the Transformers install fails, check that you installed the pull request commit, not the main branch, since support has not landed there yet.[1]
  • If replies are slow or long, check the reasoning setting. The default is high, which enables thinking, and no_think gives direct answers.[1]
  • The vLLM and SGLang commands both set tensor parallel size to 8. Fewer GPUs means changing that value.[1]

Next steps

Tencent points to the vLLM recipe and the SGLang cookbook for each engine's deployment details.[1]

Sources

  1. 1
    tencent/Hy4-flashTencent Hy (Hugging Face model card)

Cite this page: Deploying Tencent Hy4 flash with Transformers, vLLM and SGLang. TellWorld, https://www.tellworld.net/practice/practice-hy4-flash-deploy. Updated 2026-11-18

Related news

View all