vLLM quickstart

vLLM's on-ramp has exactly two lanes: offline batched inference and online serving. The first fits one-shot jobs like data generation and evaluation; the second fits resident APIs behind applications. The official quickstart walks both, and this page arranges it as a followable flow ^[1].

Installation

Prerequisites are Linux with Python 3.11-3.14; macOS users can take vLLM-Metal for Apple Silicon GPU acceleration, which uses MLX rather than PyTorch as the compute backend and expects MLX-optimized models from the mlx-community repo ^[1].

On NVIDIA GPUs, install with uv, a very fast Python environment manager ^[1]:

uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto
vllm download-kernels

The last line deserves its own note: download-kernels fetches precompiled kernels up front; without them vLLM downloads and compiles at startup, which can add several minutes on Hopper and newer GPUs ^[1]. The --torch-backend=auto flag makes uv inspect the installed CUDA driver and pick the matching PyTorch index automatically; pin a specific backend such as cu126 explicitly when needed ^[1]. A lighter option also exists — uv run --with vllm executes commands without any permanent environment, and conda works too by installing uv into the environment ^[1].

Other hardware has its own channels: AMD ROCm goes through uv with a dedicated extra index (currently Python 3.12, ROCm 7.0, glibc 2.35+, plus a nightly Docker image); Intel GPUs use the XPU backend with official images shipping since v0.26.0; TPUs need the vllm-tpu package; Ascend NPUs run through the community-maintained vLLM Ascend plugin ^[1].

Offline batched inference

The offline mode rests on two classes: LLM runs the engine, SamplingParams defines the sampling ^[1]:

from vllm import LLM, SamplingParams

The example defines four prompts with temperature 0.8 and top_p 0.95 ^[1]:

prompts = [
    "Hello, my name is",
    "The president of the United States is",
    "The capital of France is",
    "The future of AI is",
]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

One default behavior is easy to miss: vLLM prefers the sampling parameters recommended by the model creator via generation_config.json in the Hugging Face repository — unspecified SamplingParams usually land on the author-tuned values; pass generation_config="vllm" at engine creation to force vLLM's own defaults ^[1].

The example engine loads facebook/opt-125m, small enough for any machine ^[1]:

llm = LLM(model="facebook/opt-125m")

Models come from Hugging Face by default; setting VLLM_USE_MODELSCOPE=True routes through ModelScope instead ^[1]. generate enqueues the prompts and executes at high throughput, returning RequestOutput objects ^[1]:

outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

A key pitfall: generate does not auto-apply the chat template. Feeding an Instruct or Chat model through generate means running prompts through the tokenizer's template first, or switching to llm.chat — whose message format matches OpenAI's client.chat.completions exactly ^[1]:

# Using tokenizer to apply chat template
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("/path/to/chat_model")
messages_list = [
    [{"role": "user", "content": prompt}]
    for prompt in prompts
]
texts = tokenizer.apply_chat_template(
    messages_list,
    tokenize=False,
    add_generation_prompt=True,
)

# Generate outputs
outputs = llm.generate(texts, sampling_params)

# Print the outputs.
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

# Using chat interface.
outputs = llm.chat(messages_list, sampling_params)
for idx, output in enumerate(outputs):
    prompt = prompts[idx]
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

The selection logic for batch work follows directly: raw completion tasks go through generate with plain prompts, conversational tasks go through chat and skip template handling — both lanes share the same engine and sampling machinery.

Online serving

Serving mode turns vLLM into a server implementing the OpenAI protocol — a drop-in replacement for applications already pointed at the OpenAI API ^[1]. It defaults to http://localhost:8000 with --host and --port overrides; it hosts one model at a time and implements list-models, chat-completion, and completion endpoints ^[1]. The launch command ^[1]:

vllm serve Qwen/Qwen2.5-1.5B-Instruct

The server carries two defaults of its own: the chat template comes from the tokenizer, and sampling defaults also prefer the repository's generation_config.json — --generation-config vllm disables the latter ^[1]. Listing models confirms the service is alive ^[1]:

curl http://localhost:8000/v1/models

For authentication, --api-key or the VLLM_API_KEY environment variable turns on header checks; multiple keys after --api-key are all accepted, enabling rotation without downtime ^[1].

The completions endpoint ^[1]:

curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen/Qwen2.5-1.5B-Instruct",
        "prompt": "San Francisco is a",
        "max_tokens": 7,
        "temperature": 0
    }'

from openai import OpenAI

# Modify OpenAI's API key and API base to use vLLM's API server.
openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8000/v1"
client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)
completion = client.completions.create(
    model="Qwen/Qwen2.5-1.5B-Instruct",
    prompt="San Francisco is a",
)
print("Completion result:", completion)

The chat endpoint supports back-and-forth exchanges stored in history — the right fit for context-dependent tasks ^[1]:

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen/Qwen2.5-1.5B-Instruct",
        "messages": [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "Who won the world series in 2020?"}
        ]
    }'

from openai import OpenAI
# Set OpenAI's API key and API base to use vLLM's API server.
openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8000/v1"

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

chat_response = client.chat.completions.create(
    model="Qwen/Qwen2.5-1.5B-Instruct",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Tell me a joke."},
    ],
)
print("Chat response:", chat_response)

Automatic backend selection is right for most setups, with two exceptions worth manual control: reproducing papers or aligning benchmarks, where pinning a backend removes result jitter from environment changes; and diagnosing a performance regression after an upgrade, where checking whether auto-selection switched backends should come before any other hypothesis. Backend differences concentrate in long-context and special architectures like MLA — short-context chat sees little spread.

Attention backends

vLLM ships several attention backends across hardware and picks the fastest compatible one automatically; --attention-backend overrides manually ^[1]. On NVIDIA CUDA: FLASH_ATTN and FLASHINFER. On AMD ROCm: TRITON_ATTN, ROCM_ATTN, ROCM_AITER_FA, ROCM_AITER_UNIFIED_ATTN, TRITON_MLA, ROCM_AITER_MLA, or ROCM_AITER_TRITON_MLA. On Intel XPU: FLASH_ATTN, TRITON_ATTN, TRITON_MLA, XPU_MLA_SPARSE, TORCH_SDPA, or TURBOQUANT ^[1]. FlashInfer installs alongside vLLM but its precompiled kernels do not — run vllm download-kernels after installing or upgrading, and startup skips the on-the-fly compilation ^[1].

The division of labor comes down to this: offline mode buys throughput — generation ends and nothing stays resident; online mode buys concurrency and protocol compatibility at nearly zero integration cost. Most projects use both — evaluation and data pipelines offline, product features online.

Adapted from the Quickstart section of the vLLM documentation^[1], released under the Apache-2.0 License.