Practical Examples
Three standalone exercises: offline batch generation, an OpenAI-compatible client, and a tiny concurrency smoke test. Finish Installation and have an NVIDIA GPU. Demos stay ungated: facebook/opt-125m and Qwen/Qwen2.5-0.5B-Instruct.
Example 1: Offline batch inference
Goal: Generate short completions for a list of prompts and feel continuous batching (the engine runs the list together).
from vllm import LLM, SamplingParams
PROMPTS = [
"PagedAttention is",
"Continuous batching helps GPU utilization because",
"An OpenAI-compatible base URL looks like",
]
llm = LLM(model="facebook/opt-125m")
params = SamplingParams(temperature=0.2, max_tokens=48, stop=["\n\n"])
outputs = llm.generate(PROMPTS, params)
for item in outputs:
text = item.outputs[0].text.replace("\n", " ").strip()
print(f"- {item.prompt!r}\n {text}\n")
Check: All three outputs are non-empty. A second run should start faster from cache. Grow the list to dozens of prompts and compare GPU utilization to one-at-a-time Transformers generate().
Example 2: Mini chat client aimed at vLLM
Goal: With vllm serve Qwen/Qwen2.5-0.5B-Instruct already running, product code depends only on the openai package.
from openai import OpenAI
MODEL = "Qwen/Qwen2.5-0.5B-Instruct"
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
messages = [
{"role": "system", "content": "You are a serving assistant. At most three short sentences."},
]
print("Empty line exits. The server must already listen on port 8000.")
while True:
user = input("you: ").strip()
if not user:
break
messages.append({"role": "user", "content": user})
resp = client.chat.completions.create(
model=MODEL,
messages=messages,
temperature=0.6,
max_tokens=128,
)
text = resp.choices[0].message.content or ""
messages.append({"role": "assistant", "content": text})
print("assistant:", text, "\n")
PowerShell smoke test (if Windows talks to a WSL-published port):
Invoke-RestMethod -Method Post -Uri "http://localhost:8000/v1/chat/completions" `
-ContentType "application/json" `
-Body '{"model":"Qwen/Qwen2.5-0.5B-Instruct","messages":[{"role":"user","content":"ping"}],"max_tokens":16}'
Check: Turn two refers to a proper noun from turn one. A wrong port fails immediately—you did not hit public OpenAI by mistake. LangChain uses the same compatible endpoint pattern.
Example 3: Concurrent request smoke test
Goal: Fire several chat calls at once so the server actually multiplexes (the point of continuous batching). Do not treat 0.5B as a production SLA—only verify responses do not get crossed.
import concurrent.futures
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
MODEL = "Qwen/Qwen2.5-0.5B-Instruct"
def ask(i: int) -> str:
r = client.chat.completions.create(
model=MODEL,
messages=[{"role": "user", "content": f"Reply with the number {i} only."}],
temperature=0,
max_tokens=8,
)
return (r.choices[0].message.content or "").strip()
with concurrent.futures.ThreadPoolExecutor(max_workers=8) as pool:
futs = [pool.submit(ask, i) for i in range(8)]
for f in concurrent.futures.as_completed(futs):
print(f.result())
Check: All eight calls return. Tiny models may not print a bare digit; that is OK. If everything times out, inspect --max-num-seqs, VRAM, and whether work is actually queued on CPU.
For real load tests use official vllm bench (subcommand names: check current CLI) or a dedicated load tool—not this 0.5B script for capacity planning.
Anti-patterns
Next steps