Kenhuang Academy
  • English

      • Installation

      vLLM’s happy path is Linux + an NVIDIA CUDA GPU. Official wheels target Linux; Windows is not natively supported—use a Linux distro inside WSL2. CPU inference, if present at all, is not what this course is for. Without a suitable GPU, use Ollama for local chat.

      Python, CUDA minors, and the compute-capability floor change between releases. Treat the table below as the usual story, then check the current GPU install page.

      Requirements (happy path)

      ItemTypical requirement (check current docs)
      OSLinux (Ubuntu, …); Windows → WSL2
      Python3.10–3.13, 3.12 recommended
      GPUNVIDIA, often compute capability 7.5+ (T4, RTX 20-series+, A100, L4, H100, B200, …)
      Driver / CUDANew enough for the matching PyTorch CUDA wheel; docs often mention CUDA 12.8 / 12.9 builds
      DiskHub cache (default ~/.cache/huggingface) needs several GB free

      Confirm the GPU is visible:

      nvidia-smi

      You should see a name and driver version. The same command must work inside WSL2 before you install vLLM.

      Virtual environment

      The project also recommends uv. pip is enough to start:

      python3 -m venv .venv
source .venv/bin/activate
python -m pip install -U pip

      uv example (check current docs):

      uv venv --python 3.12 --seed
source .venv/bin/activate

      Use a fresh environment. vLLM ships compiled CUDA kernels; mixing them with a random conda PyTorch often fails ABI checks.

      Install vLLM

      Simplest command:

      pip install vllm

      The official GPU page often suggests letting the installer pick a PyTorch CUDA index:

      uv pip install vllm --torch-backend=auto

      or (example—the CUDA tag will change):

      pip install vllm --extra-index-url https://download.pytorch.org/whl/cu129

      Do not hard-code cu129 in production without re-reading install docs and PyTorch Get Started.

      Verify:

      python -c "import vllm; print('vllm ok')"
vllm --help
# or
vllm serve --help

      If you see serve (some newer builds may say server—check the current CLI), continue to Quick Start.

      Windows: WSL2

      1. Install WSL2 and Ubuntu (or another supported Linux).
      2. Install a current NVIDIA driver on Windows so nvidia-smi works inside WSL.
      3. Create the venv and pip install vllm in the WSL terminal, not Win32 Python.
      4. Keep the Hub cache on the Linux filesystem (your WSL home), not a /mnt/c/... path, or downloads crawl.

      Unofficial native Windows builds exist; this course does not cover them. Teach and deploy as Linux / WSL2.

      Hugging Face cache and tokens

      Weights download from the Hub by default. Ungated demos in this course (Qwen 0.5B, OPT-125M) usually need no login. Gated models need:

      export HF_TOKEN="hf_..."   # or HUGGING_FACE_HUB_TOKEN — follow Hub docs

      Never commit tokens. If the Hub is unreachable, vendor a local directory and pass that path as --model.

      Other accelerators (awareness only)

      Docs also mention AMD ROCm, Intel GPUs, TPUs, and community Apple Silicon (vLLM-Metal). They are not this course’s happy path. Read the matching install page; do not assume pip install vllm on macOS equals the CUDA experience.

      Troubleshooting

      nvidia-smi sees no device? Fix drivers / WSL GPU first, then Python packages.

      CUDA / PyTorch mismatch on import? New venv; do not pip install torch from one CUDA tag and vLLM from another.

      CPU only? You can still read the API chapters; do not expect production throughput. Daily chat: Ollama.

      Shared memory in containers? See Docker; --ipc=host is common.

      Next steps

      评论