#OpenAI-Compatible API

vLLM’s online server speaks OpenAI-style HTTP so existing SDKs can change the base URL (and model name). Defaults:

Chat requires a chat template (e.g. Qwen/Qwen2.5-0.5B-Instruct). Base models like facebook/opt-125m belong on /v1/completions. For the full matrix, see OpenAI-Compatible Server (newer trees may use serving/online_serving/openai_compatible_server/—check current docs).

#Optional API key at launch

vllm serve Qwen/Qwen2.5-0.5B-Instruct --api-key token-abc123

VLLM_API_KEY is the usual environment equivalent (check current docs). Client:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="token-abc123",
)

Important: --api-key mainly protects /v1 (and documented /v2 / /inference prefixes). The same process can still expose unauthenticated paths (official docs have called out /invocations). Do not treat the API key as your only security boundary. Put a reverse proxy and network policy in front of anything public. See Serving.

#Chat Completions

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer token-abc123" \
  -d '{
    "model": "Qwen/Qwen2.5-0.5B-Instruct",
    "messages": [
      {"role": "system", "content": "Answer in concise English."},
      {"role": "user", "content": "How does continuous batching differ from static batching?"}
    ],
    "temperature": 0.7,
    "max_tokens": 200
  }'

Streaming: "stream": true (SSE). Python:

stream = client.chat.completions.create(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    messages=[{"role": "user", "content": "Count from 1 to 5."}],
    stream=True,
)
for chunk in stream:
    delta = chunk.choices[0].delta.content or ""
    print(delta, end="", flush=True)

The compatibility layer may also expose embeddings, transcriptions, Responses, and more—only when the loaded model type matches. This course focuses on text chat/completions.

#Completions (non-chat)

comp = client.completions.create(
    model="facebook/opt-125m",
    prompt="The meaning of life is",
    max_tokens=32,
)
print(comp.choices[0].text)

OpenAI treats Completions as legacy; vLLM still implements it for base LMs and old scripts. New apps should prefer Chat.

#Non-OpenAI fields: extra_body

top_k, structured outputs, and similar extras go through extra_body (names drift—check current docs):

completion = client.chat.completions.create(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    messages=[{"role": "user", "content": "Reply with a single word: yes or no."}],
    extra_body={"top_k": 20},
)

On raw HTTP, merge the same keys into the JSON body. Semantics: Sampling.

#Where defaults come from

If the Hub repo ships generation_config.json, the server may apply the author’s recommended sampling defaults. That is intentional. To force vLLM’s own defaults, try:

vllm serve Qwen/Qwen2.5-0.5B-Instruct --generation-config vllm

Confirm the flag name with --help.

#LangChain and other frameworks

Point any OpenAI-compatible client base_url at vLLM. In LangChain this is usually a custom OpenAI-compatible endpoint (package names shift in 1.x—see LangChain). model= must match the ID loaded by vllm serve.

If the app runs in Docker and vLLM on the host, do not use localhost inside the app container—use host.docker.internal or a Compose service name.

#Common failures

#Next steps

• Sampling
• Serving
• Practical Examples