#Docker

vLLM publishes an official image for the OpenAI-compatible server. On Docker Hub the name is still vllm/vllm-openai (tags such as :latest or a pinned version). The image already contains the CUDA build and entrypoint; you supply GPU access, ports, and a Hugging Face cache. Flags drift slightly—check Using Docker.

The host needs the NVIDIA Container Toolkit and a working nvidia-smi. This is not a CPU image.

#Minimal run

Recent official examples use a tiny Qwen repo. This course stays on small ungated models (you may swap in whatever the docs currently show, e.g. Qwen/Qwen3-0.6B):

docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env "HF_TOKEN=$HF_TOKEN" \
  -p 8000:8000 \
  --ipc=host \
  vllm/vllm-openai:latest \
  --model Qwen/Qwen2.5-0.5B-Instruct

Arguments after the image name are engine args, the same family as vllm serve (--tensor-parallel-size, --quantization, …).

Check:

curl http://localhost:8000/v1/models

#Pick GPUs and tensor parallel

Two GPUs with tp=2:

docker run --runtime nvidia --gpus '"device=0,1"' \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -p 8000:8000 \
  --ipc=host \
  vllm/vllm-openai:latest \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --tensor-parallel-size 2

--gpus syntax varies by Docker / NVIDIA plugin version—see docker run --help on your machine.

#Compose sketch

services:
  vllm:
    image: vllm/vllm-openai:latest
    ipc: host
    ports:
      - "8000:8000"
    volumes:
      - huggingface:/root/.cache/huggingface
    environment:
      HF_TOKEN: ${HF_TOKEN:-}
    command: ["--model", "Qwen/Qwen2.5-0.5B-Instruct", "--dtype", "auto"]
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

Compose GPU syntax is either deploy.resources or a gpus: key depending on file version. Check current Docker docs. Sibling services should call http://vllm:8000/v1, not localhost.

#Security and operations

• Pin a version tag (v0.x.y) in production; do not float on latest forever.
• Do not bake HF_TOKEN into a public Dockerfile ENV.
• Put a reverse proxy in front of any public port; a Docker network is not authentication.
• Logs tell you whether weights finished downloading, NCCL failed, or you OOM’d.
• AMD / Intel variants exist (vllm/vllm-openai-rocm, vllm/vllm-openai-xpu, …—check current docs). Out of scope here.

Root vs non-root UIDs (OpenShift, volume permissions) are documented on the official Docker page; unwritable cache mounts fail Hub downloads.

#Troubleshooting

Get pip serving working on the host first; then you can tell CUDA problems from Docker problems.

#Next steps

• Serving
• Practical Examples
• Docker tutorial on this site