Serving gpt-oss-20b with vLLM

GPT-OSS is OpenAI's return to open-weight models, shipped as two sizes at once: gpt-oss-120b and gpt-oss-20b. Both use a MoE Transformer architecture, support 128K context length, and arrive under the Apache 2.0 license with free use and commercial application permitted ^[1]. The 20B's headline trait is its low floor: it runs on 16GB of memory, fitting edge devices and consumer-grade GPUs alike ^[1]. This page walks the full vLLM deployment path — where the real work is not the model itself but matching dependency versions.

What each size is for

The two sizes split cleanly ^[1]. The 120B nearly matches OpenAI o4-mini on core reasoning benchmarks while running efficiently on a single 80GB GPU; the 20B lands close to o3-mini on common benchmarks and suits on-device applications, local inference, and fast iteration without expensive infrastructure ^[1]. Both are strong on tool use, few-shot function calling, and CoT reasoning (see their Tau-Bench results), and on HealthBench they surpass proprietary models like OpenAI o1 and GPT-4o ^[1].

The VRAM math is simple ^[1]: gpt-oss-20b needs about 16GB and runs on cards such as the A100 or H20; gpt-oss-120b does best with at least 60GB on one H100 or a multi-GPU setup.

Environment: racing the latest versions

The environment is the hard part. The vLLM 0.10.1 build that supports gpt-oss was not officially released as of August 6, 2025, so installation comes from pre-release sources, and a fresh Python 3.12 environment is recommended to avoid polluting existing setups ^[1]. Worse, many dependencies sit on the bleeding edge — an older configuration breaks, so upgrade conda and CUDA when needed ^[1]:

conda search cuda-toolkit --channel nvidia
conda install cuda-toolkit=<新版本号> --channel nvidia
nvcc --version #查看是否升级成功

export CUDA_HOME=$CONDA_PREFIX
export LD_LIBRARY_PATH=$CONDA_PREFIX/lib:$LD_LIBRARY_PATH #在conda升级后需设置一下路径

After a conda-level CUDA upgrade, resetting CUDA_HOME and the library search path is easy to skip — and skipping it leaves later builds unable to find CUDA headers. The tested stack: PyTorch 2.9.0, Python 3.12 (Ubuntu 22.04), CUDA 12.8, one NVIDIA H20-96GB ^[1].

Next, clone the official recipe repository and set up an isolated environment with mirrors for speed ^[1]:

git clone https://github.com/huggingface/gpt-oss-recipes.git
cd gpt-oss-recipes

conda create -n gpt_oss_vllm python=3.12
conda activate gpt_oss_vllm

python -m pip install --upgrade pip
pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple

export HF_ENDPOINT="https://hf-mirror.com"
pip install -U huggingface_hub

With HF_ENDPOINT pointed at the mirror, every later Hugging Face download rides it — a large time saver on constrained networks. One command covers the core model dependencies ^[1]:

pip install -U transformers kernels torch accelerate

Model download

Weights live in both the Hugging Face and ModelScope repos, either fetched explicitly or auto-downloaded on first use ^[1]. The explicit route is clearest ^[1]:

huggingface-cli download openai/gpt-oss-20b --local-dir gpt-oss-20b

Adjust local_dir to your disk layout. Downloading up front means later serve commands can point straight at the local directory instead of checking the remote on every start.

transformers first: a small, fast sanity check

Before standing up a service, verify the model runs with a minimal transformers pipeline ^[1]:

from transformers import pipeline
import torch

model_id = "openai/gpt-oss-20b"

pipe = pipeline(
    "text-generation",
    model=model_id,
    torch_dtype="auto",
    device_map="auto",
)

messages = [
    {"role": "user", "content": "Explain quantum mechanics clearly and concisely."},
]

outputs = pipe(
    messages,
    max_new_tokens=256,
)
print(outputs[0]["generated_text"][-1])

Both torch_dtype and device_map are left to auto, letting the framework pick precision and placement. Printing the last element of generated_text yields exactly the model's reply message.

transformers can also expose the model over HTTP with two commands — one for the server, one for the chat client ^[1]:

transformers serve
transformers chat localhost:8000 --model-name-or-path openai/gpt-oss-20b

This family has a distinctive control point: how much the model reasons can be steered with a system message. For deeper reasoning, set the level explicitly ^[1]:

messages = [
    {"role": "system", "content": "Reasoning: high"},
    {"role": "user", "content": "解释量子计算的基本原理"}
]

A high reasoning level spends more thinking tokens for sturdier answers; simple tasks can trade level down for speed. Making reasoning effort a tunable parameter at the prompt layer is a genuinely unusual interface decision for an open model.

The vLLM deployment: pre-release wheels and FlashInfer

One caution before this stage: vLLM releases have poor cross-version compatibility, so pin and verify the exact version ^[1]. Install the pre-release wheel, then FlashInfer separately ^[1]:

pip install --pre vllm==0.10.1+gptoss \
    --extra-index-url https://wheels.vllm.ai/gpt-oss/ \
    --extra-index-url https://download.pytorch.org/whl/nightly/cu128

# !!!安装 FlashInfer!!!
pip install flashinfer-python==0.2.10

The two extra indexes are both required: one serves the gpt-oss-specific vLLM wheel, the other the PyTorch CUDA 12.8 nightly build. FlashInfer is pinned to 0.2.10 as the critical attention kernel.

Starting the service is a single command — vLLM auto-downloads from Hugging Face and launches an OpenAI-compatible server, and the model name can be swapped for a locally downloaded directory ^[1]:

# For 20B 可以替换成本地下载的目录
vllm serve openai/gpt-oss-20b

# For 120B 可以替换成本地下载的目录
vllm serve openai/gpt-oss-120b

The tested configuration adds three things: an environment variable forcing the Triton attention backend, an explicit --served-model-name for a clean external name, and a move to port 8801 ^[1]:

VLLM_ATTENTION_BACKEND=TRITON_ATTN_VLLM_V1 vllm serve openai/gpt-oss-20b --served-model-name gpt-oss-20b --trust_remote_code --port 8801

The served-model-name flag deserves its own note: with it set, client requests use the short name instead of a filesystem path, which keeps downstream integrations clean. Forcing the attention backend is a pre-release stability measure that should fade away once the official build lands.

The whole flow compresses to three habits: keep the environment current, pin dependency versions, and validate with a small pipeline or chat call before opening the floodgates. With open weights and a 16GB VRAM floor, gpt-oss-20b is one of the cheapest paths to running an OpenAI-class model locally.

Adapted from the GPT-OSS-20B vLLM deployment section of Datawhale's open guide *self-llm*^[1], released under the Apache 2.0 License.