#Sampling

Generation is not “the one true next token.” Sampling parameters draw from the vocabulary distribution. Offline, that is SamplingParams. Online, the same knobs map to OpenAI fields plus extra_body.

Defaults may come from the repo’s generation_config.json—see OpenAI-Compatible API. Always cap max_tokens so a runaway sample cannot hog scheduler slots.

#Offline: SamplingParams

from vllm import LLM, SamplingParams

llm = LLM(model="Qwen/Qwen2.5-0.5B-Instruct")

greedy = SamplingParams(temperature=0, max_tokens=64)
sampled = SamplingParams(
    temperature=0.8,
    top_p=0.95,
    top_k=50,
    max_tokens=64,
)

prompt = "Write one short slogan for a GPU inference engine."
print("greedy:", llm.generate(prompt, greedy)[0].outputs[0].text)
print("sampled:", llm.generate(prompt, sampled)[0].outputs[0].text)

Reuse one LLM instance; swap SamplingParams per batch. Instruct models may also take llm.chat(messages, sampling_params=...) (confirm the method on your version).

#Parameter intuition

Very high temperature plus a tiny top_p fight each other. Change one or two knobs and compare the same prompt.

#Greedy vs sampling

• Greedy / near-deterministic: temperature=0 (or very low), no nucleus lottery. Good for extraction, classification, rigid formats, regression tests.
• Sampling: temperature around 0.6–1.0 with top_p. Good for chat and writing. Tiny models (0.5B) fall apart if temperature is too high—start near 0.7.

If you need JSON-shaped output, prefer official structured outputs / guided decoding (names change) over begging in the prompt.

#Online: Chat Completions fields

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

resp = client.chat.completions.create(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    messages=[{"role": "user", "content": "List three GPU serving tips."}],
    temperature=0.4,
    top_p=0.9,
    max_tokens=128,
    extra_body={"top_k": 40},
)
print(resp.choices[0].message.content)

Some OpenAI fields (e.g. user) are ignored. Extra vLLM sampling keys go in extra_body; see “Extra parameters” in the official server docs.

#Mapping from Transformers generate()

vLLM tokenizes and detokenizes for you. Do not mix raw token-id lists with string prompts accidentally. Recommended sampling on the model card is often already in generation_config.json.

#Debug checklist

1. Start with temperature=0 and max_tokens=32 to prove the model generates (not an instant EOS from the template).
2. Turn sampling on and confirm the two outputs differ.
3. If it repeats a sentence forever, lower temperature, add stop, or check you did not feed a chat transcript into the completions API.
4. Latency rises with max_tokens and concurrency—that is continuous batching working, not a hang.

#Next steps

• Quantization
• Serving
• Practical Examples