#vLLM Tutorial

Welcome to the vLLM tutorial. It follows the official vLLM docs and vllm-project/vllm, and shows how to turn Hugging Face checkpoints into a high-throughput, OpenAI-compatible GPU inference service.

vLLM is built for “many requests at once”: PagedAttention manages the KV cache, and continuous batching keeps the GPU busy. It is not a desktop chat app—use Ollama for local UX and one-command model swaps; use Hugging Face for weights, tokenizers, and model cards.

CLI names, CUDA wheels, and quantization enums move quickly. Where this guide says check current docs, confirm against docs.vllm.ai and vllm serve --help before you ship anything.

---

#Table of Contents

#Basics

1. Introduction — Engine vs desktop runtime, PagedAttention, continuous batching
2. Installation — Linux + NVIDIA CUDA, pip install vllm, Windows via WSL2
3. Quick Start — vllm serve, a small model, first /v1 call

#APIs and generation

4. OpenAI-Compatible API — http://localhost:8000/v1, chat/completions, SDKs
5. Sampling — SamplingParams, temperature / top_p, greedy vs sampling

#Performance and deploy

6. Quantization — AWQ / GPTQ / FP8, --quantization, when you can omit the flag
7. Serving — --tensor-parallel-size, VRAM, auth limits
8. vLLM vs Ollama — Desktop DX vs production throughput, plus the Hub
9. Docker — Official vllm/vllm-openai image

#Practice

10. Practical Examples — Offline batch, compatible client, concurrent smoke test
11. Resources — Official docs and related courses on this site

---

#Learning path

#Conventions in this course

• Happy path: Linux + NVIDIA CUDA. CPU backends are not the point; without an NVIDIA GPU, start with Ollama.
• Windows: not natively supported—use WSL2 (Ubuntu) with GPU passthrough.
• Demo models (small, ungated): Qwen/Qwen2.5-0.5B-Instruct (chat) or facebook/opt-125m (completions). Do not treat gated Llama-2 as the only example.
• Server base URL: http://localhost:8000/v1; chat uses /v1/chat/completions.
• CLI: newer installs use vllm serve <model>. Older posts use python -m vllm.entrypoints.openai.api_server --model ... (deprecated in current source). Check current docs and --help (serve vs server).

#Prerequisites

• Python 3.10+ (docs often list 3.10–3.13; check current docs)
• Comfort with a terminal, venv, pip or uv
• Familiarity with Chat Completions (messages, role)
• An NVIDIA GPU in the range the docs currently require (often compute capability 7.5+: T4, RTX 20-series+, A100, L4, H100, B200—check current docs)
• Disk for the Hub cache (gigabytes, sometimes tens of GB)

#What you will be able to do

• Explain how vLLM differs from Ollama and from Transformers generate()
• Install on Linux (or WSL2) and vllm serve a small Instruct model
• Call localhost:8000/v1 with curl or the OpenAI Python SDK
• Use SamplingParams for greedy vs sampled decoding
• Choose among AWQ / GPTQ / FP8 at a high level and know when --quantization matters
• Start a shareable service with --tensor-parallel-size and the official Docker image

#Related tutorials

---

#Next steps

• Introduction
• Installation
• Quick Start