#Quick Start

This chapter gets two things working on an ungated small model: offline generation, and OpenAI-compatible HTTP. Finish Installation first and confirm nvidia-smi sees a GPU.

Default chat model:

Qwen/Qwen2.5-0.5B-Instruct

Even smaller completion model (no chat template; use /v1/completions or LLM.generate):

facebook/opt-125m

Neither requires a Llama license. If VRAM is still tight, pick a smaller Instruct checkpoint or add --max-model-len 2048.

#Start the OpenAI-compatible server

Preferred (newer CLI):

vllm serve Qwen/Qwen2.5-0.5B-Instruct

The first run pulls weights from Hugging Face, then listens on port 8000. Wait for the Uvicorn / ready log line before sending traffic.

Legacy module (still in many blog posts; current source marks it deprecated—check current docs):

python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen2.5-0.5B-Instruct

Some newer messages may say vllm server instead of vllm serve. When unsure, run vllm --help.

Useful flags (full list: --help):

vllm serve Qwen/Qwen2.5-0.5B-Instruct \
  --host 127.0.0.1 \
  --port 8000 \
  --dtype auto \
  --max-model-len 4096

#Chat Completions with curl

Base URL: http://localhost:8000/v1. Instruct models use Chat Completions:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-0.5B-Instruct",
    "messages": [
      {"role": "user", "content": "Explain PagedAttention in one sentence."}
    ],
    "max_tokens": 128
  }'

From Windows PowerShell against a WSL server, prefer curl.exe, or:

Invoke-RestMethod -Method Post -Uri "http://localhost:8000/v1/chat/completions" `
  -ContentType "application/json" `
  -Body '{"model":"Qwen/Qwen2.5-0.5B-Instruct","messages":[{"role":"user","content":"Say hello in one sentence."}],"max_tokens":64}'

The model field is usually the ID you passed to serve (or the name from /v1/models). They must match.

List models:

curl http://localhost:8000/v1/models

#OpenAI Python SDK

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

resp = client.chat.completions.create(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    messages=[{"role": "user", "content": "Give two bullets on continuous batching."}],
    max_tokens=128,
)
print(resp.choices[0].message.content)

Local servers often do not check the key unless you passed --api-key token-abc123. Auth scope is documented in Serving.

#Offline: LLM + SamplingParams

No HTTP process—good for batch scripts:

from vllm import LLM, SamplingParams

llm = LLM(model="facebook/opt-125m")
params = SamplingParams(temperature=0.8, top_p=0.95, max_tokens=64)
outputs = llm.generate(
    ["Hello, my name is", "The capital of France is"],
    params,
)
for o in outputs:
    print(o.prompt, "→", o.outputs[0].text)

Chat models may support llm.chat(...) (check your installed API). Details: Sampling.

#Day-one checklist

1. nvidia-smi lists a GPU
2. import vllm works after pip install vllm
3. vllm serve Qwen/Qwen2.5-0.5B-Instruct stays up
4. curl or the SDK returns non-empty choices[0].message.content
5. Optional: offline generate with facebook/opt-125m

If it fails: lower --max-model-len, wait for the Hub download, or free port 8000.

#Next steps

• OpenAI-Compatible API
• Sampling
• Practical Examples