Deploying DeepSeek-V4-Flash with SGLang

Who this is for

This is for readers who have a multi-GPU server and want to host DeepSeek-V4-Flash with SGLang. After following it, you should have a running endpoint and have tested chat, streaming, thinking mode and tool calls. It also covers the points where a first deployment usually stalls.

What SGLang adds

SGLang is an open-source serving framework for large language models. It exposes an OpenAI-compatible API, which lets existing client code talk to the model with only a change of base URL^[1]. DeepSeek-V4-Flash is distributed as the official DeepSeek Instruct release on Hugging Face. The weights are about 149GB in 46 shards, with MoE experts in FP4 and attention and dense layers in FP8^[2].

Pick a route by hardware

There are two routes. The Docker route uses the official SGLang image on RTX PRO 6000 Blackwell cards. The Python route was tested on four 141GB H200 cards^[1]. The Cookbook's verified RTX PRO 6000 setup uses two 96GB cards with TP=2 and the FlashInfer MXFP4 MoE backend. That configuration supports only the Flash model, and HiCache and MegaMoE should stay off^[1][3]. On most cloud servers Docker is not available, so use the Python route there^[1].

Download and check the model

Download once and keep the directory on a disk with room for about 149GB of weights, because the model is too large for a single 141GB H200 card and must be split across GPUs^[1][2].

Download the weights once with the Hugging Face CLI, or with ModelScope if the network in mainland China is slow^[1][2].

hf download deepseek-ai/DeepSeek-V4-Flash \
    --local-dir /root/autodl-fs/models/DeepSeek-V4-Flash

modelscope download \
    --model deepseek-ai/DeepSeek-V4-Flash \
    --local_dir /root/autodl-fs/models/DeepSeek-V4-Flash

After the download, there should be 46 shards, no .incomplete files, a total of about 149GB, and an index file^[1].

ls "$MODEL_PATH"/model-*.safetensors | wc -l
ls "$MODEL_PATH"/*.incomplete 2>/dev/null || echo "no incomplete"
du -sh "$MODEL_PATH"
test -f "$MODEL_PATH/model.safetensors.index.json" && echo "index OK"

Check the GPUs with nvidia-smi and nvidia-smi topo -m^[1].

nvidia-smi
nvidia-smi topo -m

Route 1: RTX PRO 6000 with the Docker image

Pull the image with docker pull lmsysorg/sglang:latest^[1].

docker pull lmsysorg/sglang:latest

The run command passes --gpus '"device=0,1"', so it uses only GPU 0 and GPU 1, even when a server has four RTX PRO 6000 cards. To use other cards, change the device list. The container exposes port 30000 and mounts the model read-only. It sets --reasoning-parser deepseek-v4 and --tool-call-parser deepseekv4^[1].

docker run --rm \
  --gpus '"device=0,1"' \
  --shm-size 32g \
  --ipc=host \
  -p 30000:30000 \
  -v /data/models/DeepSeek-V4-Flash:/models/DeepSeek-V4-Flash:ro \
  lmsysorg/sglang:latest \
  sglang serve \
    --trust-remote-code \
    --model-path /models/DeepSeek-V4-Flash \
    --served-model-name deepseek-v4-flash \
    --tp 2 \
    --moe-runner-backend flashinfer_mxfp4 \
    --mem-fraction-static 0.92 \
    --cuda-graph-max-bs-decode 32 \
    --reasoning-parser deepseek-v4 \
    --tool-call-parser deepseekv4 \
    --host 0.0.0.0 \
    --port 30000

Route 2: four H200 cards with a Python environment

Create a Python 3.12 environment. The tested stack used PyTorch 2.11.0+cu130. The commands below set the uv cache, use the Tsinghua PyPI mirror, install SGLang and then reinstall sglang-kernel from the cu130 index^[1].

conda create -n sglang python=3.12 -y
conda activate sglang

export UV_CACHE_DIR=/dev/shm/uv-cache-sglang
export UV_DEFAULT_INDEX=https://pypi.tuna.tsinghua.edu.cn/simple
export UV_LINK_MODE=copy
mkdir -p "$UV_CACHE_DIR"

python -m pip install --upgrade pip
python -m pip install uv
uv pip install --prerelease=allow sglang
uv pip install --force-reinstall sglang-kernel \
  --index-url https://docs.sglang.ai/whl/cu130/

Verify that CUDA is visible and that the DeepSeek-V4 parser is present^[1].

python -c "import torch; print(torch.__version__, torch.cuda.is_available(), torch.cuda.device_count())"
python -m sglang.launch_server --help | grep -o "deepseek-v4" | head

On H200 the official FP4 and FP8 mixed weights load directly. The tested command uses --tp 4, --mem-fraction-static 0.85, a first context length of --context-length 131072, and the Marlin W4A16 MoE backend (--moe-runner-backend marlin)^[1]. It also turns on EAGLE speculative decoding with three steps, a top-k of 1 and four draft tokens^[1].

conda activate sglang

export CUDA_VISIBLE_DEVICES=0,1,2,3
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

sglang serve \
  --model-path /root/autodl-fs/models/DeepSeek-V4-Flash \
  --served-model-name deepseek-v4-flash \
  --trust-remote-code \
  --tp 4 \
  --host 0.0.0.0 \
  --port 30000 \
  --mem-fraction-static 0.85 \
  --context-length 131072 \
  --moe-runner-backend marlin \
  --reasoning-parser deepseek-v4 \
  --tool-call-parser deepseekv4 \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4

The first start is slow because it loads the model, warms up kernels and captures CUDA Graphs. The service is ready when the log shows these lines^[1].

Application startup complete.
Uvicorn running on http://0.0.0.0:30000
The server is fired up and ready to roll!

Call the API

The endpoint is http://127.0.0.1:30000/v1, and any placeholder key works. Create the client like this^[1].

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:30000/v1", api_key="EMPTY")

Thinking mode is turned on through extra_body={"chat_template_kwargs": {"thinking": True}}. The reasoning text arrives in the reasoning_content field of the streamed delta^[1]. The tool-call test sends a get_weather function definition and prints the returned tool_calls^[1].

Limits and troubleshooting

The --tp flag sets tensor parallelism, the number of GPUs the model is split across. Use 2 for the Docker route and 4 for the H200 route, and make it match the cards you give the process^[1].

Every request's model field must be deepseek-v4-flash, the name set by --served-model-name^[1]. The first start is slow and should not be interrupted, because the model loads, kernels warm up and CUDA Graphs are captured^[1]. If memory is tight, lower --mem-fraction-static or shorten --context-length; the tested H200 configuration uses 0.85 and 131072^[1].

The Docker route depends on the device list. A four-card RTX PRO 6000 host still uses GPU 0 and GPU 1 unless you edit the command. The first start can take several minutes. After loading, use nvidia-smi to check memory on each card and watch utilization during requests^[1].

For the official configuration notes, see the SGLang DeepSeek-V4 Cookbook^[3].

Adapted from the DeepSeek-V4-Flash SGLang deployment lesson in Datawhale's open-source course *self-llm*^[1], released under the Apache-2.0 License.