Kenhuang Academy
  • English

      • Tokenizers and Models

      After Pipeline, split the Preprocessor from the PreTrainedModel. Almost every checkpoint loads through AutoClass + from_pretrained.

      Which AutoClass?

      ClassWhen
      AutoConfigArchitecture hyperparameters only—no VRAM
      AutoTokenizerText ↔ token ids
      AutoImageProcessor / AutoProcessorImages, multimodal
      AutoModelBare backbone (no task head); rarely used directly
      AutoModelForSequenceClassificationSentence classification
      AutoModelForCausalLMAutoregressive GPT / Qwen-style generation
      AutoModelForSeq2SeqLMT5 / BART encoder–decoder

      model_type in config.json picks the concrete class (e.g. DistilBertForSequenceClassification). You do not memorize class names.

      from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "distilbert/distilgpt2"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, device_map="auto", dtype="auto"
)

      Encode and decode

      from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("distilbert/distilbert-base-uncased")
enc = tok("Transformers run on the Hub.", return_tensors="pt")
print(enc.keys())          # input_ids, attention_mask
print(enc["input_ids"])
print(tok.convert_ids_to_tokens(enc["input_ids"][0]))
print(tok.decode(enc["input_ids"][0], skip_special_tokens=True))
      • input_ids: vocabulary indices—the integers the model consumes.
      • attention_mask: 1 for real tokens, 0 for pad, so pad does not attend.
      • return_tensors="pt": PyTorch tensors (see the PyTorch tutorial).
      • DistilBERT adds [CLS] / [SEP]; GPT-style models may only have BOS/EOS—or neither.

      Batching needs padding, and a defined pad token:

      texts = ["short", "a much longer example sentence"]
if tok.pad_token is None:
    tok.pad_token = tok.eos_token
batch = tok(texts, padding=True, truncation=True, return_tensors="pt")

      Forward vs generate

      A classification head consumes tokenizer output and exposes logits:

      from transformers import AutoModelForSequenceClassification

clf_tok = AutoTokenizer.from_pretrained(
    "distilbert/distilbert-base-uncased-finetuned-sst-2-english"
)
clf = AutoModelForSequenceClassification.from_pretrained(
    "distilbert/distilbert-base-uncased-finetuned-sst-2-english"
)
out = clf(**clf_tok("Great acting.", return_tensors="pt"))
print(out.logits.argmax(-1))

      Causal LMs use generate() (next chapter). A single forward is not “the finished article.”

      Instruct models and chat templates

      Models such as Qwen/Qwen2.5-0.5B-Instruct must go through apply_chat_template. Do not treat the user sentence as raw continuation:

      from transformers import AutoModelForCausalLM, AutoTokenizer

mid = "Qwen/Qwen2.5-0.5B-Instruct"
tok = AutoTokenizer.from_pretrained(mid)
model = AutoModelForCausalLM.from_pretrained(
    mid, device_map="auto", dtype="auto"
)
messages = [
    {"role": "system", "content": "You are a concise tutor."},
    {"role": "user", "content": "What is a tokenizer?"},
]
ids = tok.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)

      The template lives in the tokenizer config. Do not paste Qwen markers onto a DeepSeek model.

      Saving and the three artifacts

      save_dir = "./my-distilgpt2"
model.save_pretrained(save_dir)
tokenizer.save_pretrained(save_dir)
# Expect config.json, tokenizer.json, *.safetensors

      That is the official trio: PreTrainedConfig, PreTrainedModel, Preprocessor. Later, from_pretrained(save_dir) loads offline.

      Common pitfalls

      SymptomFix
      pad_token is Nonetokenizer.pad_token = tokenizer.eos_token and sync model.config
      Poor Chinese segmentationUse a Chinese vocab (Qwen), not English DistilBERT
      OOMdevice_map="auto", dtype="auto", a smaller model, or quantization
      Gated 401Switch to this course’s ungated IDs, or accept the license and hf auth login

      Next steps

      评论