Kenhuang Academy
  • English

      • Quantization

      Quantization stores weights (and sometimes activations) in a narrower format than BF16/FP16: less VRAM and bandwidth, some accuracy cost. On vLLM the usual families are AWQ, GPTQ, and FP8, plus newer layouts such as compressed-tensors. That is not the same pipeline as Ollama GGUF Q4/Q8—do not expect to pass a .gguf to vllm serve as the default path.

      Method enums and kernel names (Marlin, …) move between releases. This page is about choosing a family; flags come from vllm serve --help and the quantization docs.

      Three formats (high level)

      MethodIntuitionTypical sourceWhen
      AWQ4-bit weights; protects “salient” channelsHub *-AWQ reposFit 7B–70B on mid-range GPUs
      GPTQ4-bit weights with calibration dataAutoGPTQ / Hub GPTQ reposLarge catalog of existing checkpoints
      FP88-bit float; new GPUs can do W8A8Pre-quantized repos, or convert at loadHopper / Ada-class cards (full FP8 compute often capability ≥ 8.9; older GPUs may fall back to weight-only—check current docs)

      Roughly: 7B at 16-bit is tens of GB; 4-bit often fits an 8–12 GB card plus KV. Do not skip nvidia-smi. Accuracy hits instruction following, math, and long context first—compare against full precision on your evals.

      Do you still need --quantization?

      For pre-quantized checkpoints (quantize_config.json or quantization_config in config.json), vLLM often auto-detects the method. A wrong --quantization value can conflict with the checkpoint and crash.

      Online quantization (convert BF16/FP16 at load, commonly to FP8) is when you set the flag explicitly. Official docs still document:

      vllm serve facebook/opt-125m --quantization fp8
      from vllm import LLM

llm = LLM(model="facebook/opt-125m", quantization="fp8")

      Pre-quantized AWQ / GPTQ (swap in a repo you have checked for license and existence):

      vllm serve <org>/<model>-AWQ
# Only if you must declare it (check current docs):
# vllm serve <org>/<model>-AWQ --quantization awq

      --quantization / -q accepts a long list (awq, gptq, fp8, Marlin variants, bitsandbytes, …). Run --help before copying an old blog enum.

      Practical advice

      1. Prove the stack on an unquantized tiny model (Qwen 0.5B / OPT-125M in this course).
      2. For a larger model, prefer a ready-made Hub repo; read the card (license, tool, group size).
      3. On H100 / L40S-class hardware, evaluate FP8 (throughput vs quality).
      4. “4-bit is always faster” is not a theorem: kernels, batch size, and KV dtype matter. Measure tokens/s.
      5. KV cache can have its own dtype flag (often --kv-cache-dtype—check current docs); that is a separate VRAM lever.

      When searching Hugging Face for AWQ / GPTQ, do not make gated Llama your only option; Qwen and some community OPT/GPT repos are easier to download on day one.

      vs Ollama quantization

      OllamavLLM
      Common formatGGUF (Q4, Q8, …)AWQ / GPTQ / FP8 / compressed tensors
      GoalSingle-machine chatServer-side batches
      SwitchingLibrary tagsAnother Hub repo or --quantization

      Desktop “pull a q4 and chat” → Ollama. OpenAI-compatible gateway + continuous batching → vLLM and the matching checkpoint.

      Troubleshooting

      Quantization mismatch on load? Drop the handwritten --quantization, or align it with quantize_config.json.

      FP8 unsupported architecture? The GPU is too old; use AWQ/GPTQ or a small full-precision model.

      Quality collapsed? Higher-bit checkpoint, or BF16 with a smaller --max-model-len / concurrency.

      bitsandbytes / GGUF? Support varies widely by version—not the default here. Read the current quantization chapter.

      Next steps

      评论