Kenhuang Academy
  • English

      • Text Generation

      The official LLM inference entry is PreTrainedModel.generate() (Pipeline text-generation calls it). Settings live on GenerationConfig. Always set max_new_tokens explicitly—the default is often too small.

      Minimal generate()

      from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "distilbert/distilgpt2"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, device_map="auto", dtype="auto"
)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

inputs = tokenizer("Once upon a time,", return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(out[0], skip_special_tokens=True))

      device_map="auto" and dtype="auto" are the official low-friction load tips.

      Greedy vs sampling

      StrategyKey argsBehavior
      Greedydo_sample=False (often the default)Highest-probability token each step; reproducible, can loop
      Samplingdo_sample=TrueDraw from the distribution; more variety
      Temperaturetemperature (needs sampling)Higher = more random; 0.7–1.0 is common
      Nucleustop_pSample from the smallest set whose mass ≥ p
      Top-ktop_kKeep only the k highest-scoring tokens
      greedy = model.generate(**inputs, max_new_tokens=30, do_sample=False)
sample = model.generate(
    **inputs,
    max_new_tokens=30,
    do_sample=True,
    temperature=0.8,
    top_p=0.9,
)
print(tokenizer.decode(greedy[0], skip_special_tokens=True))
print(tokenizer.decode(sample[0], skip_special_tokens=True))

      Chat assistants usually sample; exams or strict JSON prefer greedy or low temperature.

      Instruct models (Qwen)

      For Chinese, prefer Qwen / DeepSeek Instruct weights over English DistilGPT2:

      from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Qwen/Qwen2.5-0.5B-Instruct"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, device_map="auto", dtype="auto"
)
messages = [
    {"role": "user", "content": "Describe the Hugging Face Hub in three sentences."},
]
prompt = tok.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)
out = model.generate(prompt, max_new_tokens=128, do_sample=True, temperature=0.7)
print(tok.decode(out[0][prompt.shape[-1]:], skip_special_tokens=True))

      Slicing out[0][prompt.shape[-1]:] prints new tokens only, not the whole prompt.

      0.5B usually fits 4–8 GB VRAM; CPU is slow. Larger Qwen / DeepSeek cards list memory on the model card.

      GenerationConfig and common knobs

      from transformers import GenerationConfig

cfg = GenerationConfig(
    max_new_tokens=64,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
    repetition_penalty=1.1,
)
out = model.generate(**inputs, generation_config=cfg)
      ParameterMeaning
      max_new_tokensHow many new tokens at most
      max_lengthTotal length including the prompt—easy to mix up; prefer the former
      eos_token_idStop when this id appears
      repetition_penalty>1 penalizes repeated n-grams

      You can model.generation_config.save_pretrained("./gen-cfg") and upload it with the model.

      vs Ollama

      This site’s Ollama tutorial bundles quantization, chat templates, and an HTTP API—best for “chat on my laptop.” Transformers generate() is for paper-faithful decoding, custom logits processors, and later LoRA. Keep both: daily chat on Ollama, experiments here.

      Next steps

      评论