#Introduction

#What is vLLM?

vLLM is an open-source LLM inference and serving engine, maintained in vllm-project/vllm with docs at docs.vllm.ai. It loads Hugging Face-style weights onto the GPU, uses PagedAttention for the KV cache, and continuous batching to pack many requests into the same forward pass—raising throughput (tokens/s) and GPU utilization.

Two typical entry points:

1. Online serving: vllm serve <model> exposes an OpenAI-compatible HTTP API (default http://localhost:8000/v1).
2. Offline inference: from vllm import LLM in Python, then generate / chat over a list of prompts.

vLLM does not search the Hub, write your agent, or ship a desktop tray icon. Checkpoints live on Hugging Face; lightweight local chat is a better fit for Ollama.

#Where it sits in the stack

┌─────────────────────────────────────────────────────────────┐
│  Apps: curl · OpenAI SDK · LangChain · Dify · your gateway  │
└───────────────────────────┬─────────────────────────────────┘
                            │  HTTP  /v1/chat/completions
              ┌─────────────┴─────────────┐
              │  vLLM OpenAI-compatible     │
              │  server · localhost:8000    │
              └─────────────┬─────────────┘
                            │
                    ┌───────┴────────┐
                    │  Scheduler +     │
                    │  continuous batch│
                    │  PagedAttention  │
                    └───────┬────────┘
                            │
                    ┌───────┴────────┐
                    │  NVIDIA GPU      │
                    │  HF weights / KV │
                    └────────────────┘

#PagedAttention (intuition)

Autoregressive generation reads past K/V for every new token. A naive allocator reserves a huge contiguous buffer for max sequence length × concurrency. Most sequences never reach that length, so you waste VRAM and cannot run enough concurrent jobs.

PagedAttention splits KV into fixed-size pages (blocks), like virtual memory: the logical context is contiguous; physical pages need not be. That means:

• You pay for what you use
• New requests can fill free pages
• Prefix caching, preemption, and variable lengths get easier

You do not implement paging yourself—tune max-model-len and gpu-memory-utilization. See Serving.

#Continuous batching (intuition)

Static batching waits until every sequence in the batch finishes. Short replies idle the GPU while the longest sample crawls to EOS.

Continuous batching (iteration-level scheduling) works per decode step: when one sequence ends, its slot is given to a waiting request; new work can join on the next step. The GPU spends more time on real tokens. That is a large part of vLLM’s throughput story.

From the client’s view, one HTTP call is still one completion. The engine multiplexes many calls internally.

#A good fit / think twice

A good fit:

• Multi-user chat or completion APIs
• Existing OpenAI SDK or LangChain code—you only change base_url
• Datacenter GPUs (A100 / L40S / H100, …) where utilization is the KPI
• Offline bulk generation (synthetic data, evals)

Think twice:

• Laptop / no NVIDIA GPU / “just chat once” → Ollama
• Jumping to 70B full precision on 8 GB VRAM
• Publishing port 8000 without a proxy (--api-key is not a firewall—see Serving)
• Treating CPU inference as vLLM’s happy path—other backends exist in the docs; this course is NVIDIA GPU first

#vs Transformers and Ollama

#Next steps

• Installation
• Quick Start
• vLLM vs Ollama