Deploying DeepSeek-V4-Flash with vLLM

Who this is for

Readers with a multi-GPU NVIDIA server who want to host DeepSeek-V4-Flash themselves and give applications an OpenAI-compatible endpoint. By the end you should have a running service, have tested plain chat, streaming, thinking mode and tool calls, and know where deployments usually get stuck.

What DeepSeek-V4-Flash and vLLM are

DeepSeek-V4-Flash is a DeepSeek model whose weights are published on Hugging Face under deepseek-ai/DeepSeek-V4-Flash^[2]. vLLM is an inference engine that loads a model and exposes it over HTTP. The reference machine has four NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs with 96GB each^[1].

Choose a route before you start

There are two routes. If the machine is a local workstation or bare-metal server with Docker, the NVIDIA driver and the NVIDIA Container Toolkit installed, use the official Docker image. Cloud platforms that do not offer a Docker runtime use the Python source build instead^[1].

Plan disk space first. The weights are about 149GB, the Python environment, source and build output need at least 100GB, and the download cache needs at least 20GB in /dev/shm. Keep the Python environment off a 30GB system disk^[1].

Download the model

Pick one of the two download methods and run it once. The Hugging Face CLI is the default. Users on networks in mainland China can use ModelScope instead^[1][2].

python -m pip install -U huggingface_hub
hf download deepseek-ai/DeepSeek-V4-Flash \
  --local-dir /root/autodl-fs/models/DeepSeek-V4-Flash

python -m pip install -U modelscope
modelscope download \
  --model deepseek-ai/DeepSeek-V4-Flash \
  --local_dir /root/autodl-fs/models/DeepSeek-V4-Flash

Check the GPUs and the model

Run nvidia-smi, nvcc --version and nvidia-smi topo -m. You should see four RTX PRO 6000 Blackwell cards, nvcc reporting CUDA 13.0, 46 weight shards and no .incomplete files^[1].

nvidia-smi
nvcc --version
nvidia-smi topo -m

Route 1: the official Docker image

The Docker route pins the vLLM nightly image vllm/vllm-openai:nightly-c71a583aa9f81400528e67e3d818f66b804e8340. Do not swap in the stable v0.25.1 tag: it pins FlashInfer 0.6.13, and this configuration needs 0.6.14^[1].

export VLLM_IMAGE=vllm/vllm-openai:nightly-c71a583aa9f81400528e67e3d818f66b804e8340
docker pull "$VLLM_IMAGE"

The run command maps GPUs 0 to 3 and publishes port 8000. It mounts the model read-only and sets --tensor-parallel-size 4 and --enable-expert-parallel. The context limit is --max-model-len 131072, which is 128K tokens. The flag --kernel-config '{"moe_backend":"marlin"}' keeps SM120 from choosing the DeepGEMM MegaMoE backend, which supports only SM100^[1]. The first run uses Eager mode, and MTP and DSpark speculative decoding stay off^[1]. The service is ready when the log prints Application startup complete^[1].

Route 2: build the SM120 preview branch

The Python route uses a conda environment with Python 3.12 and uv from the Tsinghua PyPI mirror^[1].

conda create -p "$VLLM_ENV" python=3.12 pip -y
conda activate "$VLLM_ENV"

Clone the fixed branch sm120-pr-41834-stable-preview-20260717 from jasl/vllm, and check that the commit is f63bfd3d7b425b10e0b5e0e2c130fe113a85d009^[1]. This is a community preview branch, not a stable vLLM release. When you reproduce the setup, keep the pinned tag and commit rather than following the moving main branch^[1].

git -c http.version=HTTP/1.1 clone --depth 1 \
  --branch sm120-pr-41834-stable-preview-20260717 \
  https://github.com/jasl/vllm.git "$VLLM_SRC"

If the server cannot reach GitHub, clone on a local computer, pack the directory and upload it^[1].

Install the build dependencies from requirements/build/cuda.txt with --torch-backend=cu130^[1].

uv pip install --verbose --no-progress \
  -r requirements/build/cuda.txt \
  --torch-backend=cu130

FlashInfer must be 0.6.14: flashinfer-cubin is about 436.7MiB and flashinfer-python is about 13.9MiB. Install both from local wheel files with --no-deps. The repository's requirements/cuda.txt carries an extra FlashInfer index, so the build first filters that line out into a separate file before installing the rest^[1]. Seeing compiled: 0 in flashinfer show-config on the first run is normal JIT behavior^[1].

Set the architecture variables before compiling. The build targets 120a and 12.0a, uses the CUDA device target, and disables precompiled wheels^[1].

export CUDA_ARCH_LIST=120a
export TORCH_CUDA_ARCH_LIST=12.0a
export VLLM_TARGET_DEVICE=cuda
export VLLM_USE_PRECOMPILED=0

The build took about two hours, and the time depends on the CPU core count. A successful build ends with Built vllm and Installed vllm^[1]. Then vllm --version should print 20260717, and four GPUs should each report compute capability (12, 0) with the scaled_fp4_quant operator registered^[1].

Start the service and make calls

Start with Eager mode. The first serve command passes --enforce-eager, --disable-custom-all-reduce and --gpu-memory-utilization 0.85, and it names the model deepseek-v4-flash^[1]. In the reference run each GPU held about 84GB once the model loaded^[1].

Check the model list with curl -sS http://127.0.0.1:8000/v1/models^[1]. Then point the OpenAI SDK at the local endpoint:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")

Thinking mode is enabled with extra_body={"thinking": {"type": "enabled"}}. The server's --reasoning-parser deepseek_v4 separates the reasoning from the final answer^[1]. For tool calls, pass tools with tool_choice="auto". The server needs --tool-call-parser deepseek_v4 and --enable-auto-tool-choice^[1].

Optional tuning after Eager works

Only try the tuning flags after the Eager configuration passes its tests. The tuned configuration adds --enable-prefix-caching, a CUDA Graph compilation setting and MTP2 speculative decoding^[1]. In one test, 444 draft tokens were produced and 264 were accepted, about 59.5%. With a shared long prefix, time to first token fell from 18.686 seconds to 0.470 seconds on the second request^[1]. If the tuned configuration fails to start, keep the Eager service, which is enough for the base deployment^[1].

Limits and common problems

The reference setup is specific: four RTX PRO 6000 cards, CUDA 13.0 and a pinned vLLM commit. Other hardware may need different flags. Keep --gpu-memory-utilization 0.85 to leave headroom for the first compile. The engine readiness timeout is set to 3600 seconds with VLLM_ENGINE_READY_TIMEOUT_S=3600, because the first start can be slow^[1].

For the model's own documentation, read the Hugging Face model page^[2].

Adapted from the DeepSeek-V4-Flash vLLM deployment lesson in Datawhale's open-source course *self-llm*^[1], released under the Apache-2.0 License.