#Serving

Turn the single-process demo from Quick Start into something several users can hit: bind address, VRAM knobs, multi-GPU, and honest auth limits. Flag names: vllm serve --help.

#A reasonable launch skeleton

vllm serve Qwen/Qwen2.5-0.5B-Instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --dtype auto \
  --gpu-memory-utilization 0.90 \
  --max-model-len 4096 \
  --max-num-seqs 64 \
  --api-key "$VLLM_API_KEY"

Run under systemd / Kubernetes in production—not a leftover SSH session. Containers: Docker.

#Tensor parallel: --tensor-parallel-size

When one GPU cannot hold the model, shard matrices inside a layer across GPUs (Megatron-style tensor parallel):

vllm serve Qwen/Qwen2.5-0.5B-Instruct --tensor-parallel-size 2

-tp is the short form if your CLI still exposes it. Rules of thumb:

• Single node: tp often equals the number of GPUs you give this replica.
• GPUs should be homogeneous; NCCL needs a working NVLink/PCIe topology. Select devices with CUDA_VISIBLE_DEVICES=0,1.
• Pipeline parallel (--pipeline-parallel-size) and data parallel exist too. Multi-node setups often use Ray or --distributed-executor-backend. Master tp first.

A 0.5B demo does not need tp=2; that line is syntax. Count GPUs for 70B full precision from VRAM, not habit.

#Prefix cache and throughput knobs

Repeated long system prompts benefit from prefix caching (often --enable-prefix-caching—check current docs) so identical prefixes reuse KV.

Watch:

• Log lines for KV usage and running / waiting queues
• nvidia-smi utilization and memory
• Client P50/P99 latency and tokens/s

Continuous batching means per-request latency and system throughput are not the same optimization. Longer queues fill the GPU and can also grow queueing delay. Use --max-num-seqs as a limiter.

#Auth: --api-key is not a firewall

--api-key / VLLM_API_KEY covers the path prefixes listed in the docs (typically including /v1). Other endpoints on the same HTTP server may be unauthenticated or authenticated differently. Official docs warn against relying on this flag alone.

Do this instead:

1. Bind to localhost or a private network; put a reverse proxy (TLS, allowlists, separate auth) in front of the public internet.
2. Keep Hub tokens and --api-key out of image layers and Git.
3. --host 0.0.0.0 only means “listen on all interfaces,” not “this is secure.”

#Health checks and clients

Load balancers can probe GET /v1/models or the version’s health path (check current docs). Client timeouts must cover time to first token (queue + prefill), not a blindly copied 30s cloud default.

Multiple replicas: one process per GPU (or per tp group); nginx or a cloud LB in front. Do not start two uncoordinated engines on the same card.

#Failure modes

#Next steps

• Docker
• vLLM vs Ollama
• Practical Examples