Who this is for and what you need
This guide is for developers who want to run Tencent's Hy4 flash on their own machines or servers. It covers three routes: loading the model with Transformers for local inference, serving it with vLLM or SGLang, and calling the server through an OpenAI-compatible client. By the end you should know which variant to pick, what the reasoning switch does, and where a deployment is most likely to stall.
What Hy4 flash is
Tencent describes Hy4 flash as its most cost-effective Hy model to date.[1] The model has 220B total parameters, including 64B Engram parameters, and it activates 12B parameters per token.[1] It accepts text and image input, generates text, and supports a context window of up to 1M tokens, or 1,048,576 tokens.[1]
The language backbone has 42 layers: one dense feed-forward layer and 41 mixture-of-experts layers. Each token activates 16 of the 512 routed experts, plus one shared expert.[1] Attention interleaves sparse attention with sliding-window attention at a 1:3 ratio.[1] For images, a vision encoder built on SigLIP2-So400m-Patch16-Res512 connects to the backbone through an MLP projector.[1]
A built-in drafter called DFly2, with 5 layers and 0.85B parameters, supports speculative decoding.[1]
Pick a variant
Three variants are published on Hugging Face, ModelScope and GitCode.[1] Hy4 flash is the instruct model. Hy4 flash-FP8 is an FP8-quantized instruct model, and the vLLM and SGLang examples below both load it. Hy4 flash-base is a pre-trained base model and is not a chat model, so the examples in this guide do not apply to it.[1]
The model is released under the Tencent Hy License. Read the license file before you use the model in a product.[1]
Reasoning mode and sampling
For the instruct models, reasoning_effort accepts two values. "high" is the default and turns thinking on. "no_think" returns direct answers.[1] Tencent recommends temperature 0.9 and top_p 1.0.[1] Every example below uses no_think, so the replies come back without a thinking step.[1]
Option 1: load it with Transformers
Support for Hy4 flash is not yet in the Transformers main branch. Install from the pull request commit until it is merged:[1]
pip install "transformers[torch,vision] @ git+https://github.com/huggingface/transformers.git@refs/pull/50903/head"Load the processor and model with the model ID tencent/Hy4-flash. The example uses device_map="auto" for the model:[1]
from transformers import AutoModelForImageTextToText, AutoProcessor
model_id = "tencent/Hy4-flash"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(model_id, device_map="auto")After that, you pass reasoning_effort="no_think" to apply_chat_template, and call model.generate with max_new_tokens=256.[1]
Option 2: serve with vLLM
For production serving, Tencent recommends vLLM or SGLang.[1] The vLLM route uses the prebuilt image vllm/vllm-openai:hy4-flash.[1] The command below loads Hy4-flash-FP8 with tensor parallel size 8, so check your GPU count before you copy it. It also turns on speculative decoding with the DFly method and 5 speculative tokens, selects the FLASHMLA_SPARSE attention backend, and serves on port 8000 under the name hy4-flash.[1]
docker run --gpus all \
-p 8000:8000 \
--ipc=host \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:hy4-flash tencent/Hy4-flash-FP8 \
--tensor-parallel-size 8 \
--speculative-config '{"num_speculative_tokens":5,"method":"dfly"}' \
--attention-backend FLASHMLA_SPARSE \
--tool-call-parser hy_v4 \
--reasoning-parser hy_v4 \
--enable-auto-tool-choice \
--port 8000 \
--served-model-name hy4-flashOption 3: serve with SGLang
The SGLang image is lmsysorg/sglang:hy4-flash, and it is multi-arch, so it runs on both x86 and Arm.[1] Pull it first, then start the server with sglang serve. The command uses --trust-remote-code, loads tencent/Hy4-flash-FP8 with --tp 8, enables the DFLY speculative algorithm with block size 6, and listens on 127.0.0.1 port 8000.[1]
docker pull lmsysorg/sglang:hy4-flash
docker run --gpus all \
-p 8000:8000 \
--ipc=host \
lmsysorg/sglang:hy4-flash \
sglang serve \
--trust-remote-code \
--model-path tencent/Hy4-flash-FP8 \
--tp 8 \
--cuda-graph-backend-prefill disabled \
--speculative-algorithm DFLY \
--speculative-dfly-block-size 6 \
--host 127.0.0.1 \
--port 8000Call the server with the OpenAI client
Once the vLLM server is up, point the OpenAI client at http://127.0.0.1:8000/v1 and pass EMPTY as the API key.[1] Set the model name to hy4-flash. To turn thinking off for a request, pass reasoning_effort as no_think in chat_template_kwargs.[1]
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="hy4-flash",
messages=[
{"role": "user", "content": "Hello! Can you briefly introduce yourself?"},
],
temperature=0.9,
top_p=1.0,
extra_body={"chat_template_kwargs": {"reasoning_effort": "no_think"}},
)
print(response.choices[0].message.content)Speculative decoding and speed
DFly2 reuses the target model's KV cache and the positions its indexer selects. Tencent says each draft token attends to at most about 2,000 keys regardless of context length, adds no separate KV memory, and does not take part in prefill.[1] Tencent also says that megakernels, 4-bit quantization and DFly2 together push decoding past 150 tokens per second.[1]
Quantization and testing
AngelSlim is Tencent's compression toolkit, and it covers quantization, low-bit quantization and speculative sampling.[1] For end-to-end testing without a large GPU setup, Tencent also provides a GGUF compressed version of the model.[1]
Fine-tuning
The model repository includes a finetuning pipeline, documented in its finetune directory.[1]
Common problems
- If the Transformers install fails, check that you installed the pull request commit, not the main branch, since support has not landed there yet.[1]
- If replies are slow or long, check the reasoning setting. The default is
high, which enables thinking, andno_thinkgives direct answers.[1] - The vLLM and SGLang commands both set tensor parallel size to 8. Fewer GPUs means changing that value.[1]
Next steps
Tencent points to the vLLM recipe and the SGLang cookbook for each engine's deployment details.[1]
Sources
- 1tencent/Hy4-flashTencent Hy (Hugging Face model card)
Cite this page: Deploying Tencent Hy4 flash with Transformers, vLLM and SGLang. TellWorld, https://www.tellworld.net/practice/practice-hy4-flash-deploy. Updated 2026-11-18