Kenhuang Academy
  • English

      • Models and quantization

      Ollama models come from ollama.com/library. Each page lists the full name, tags, size, license, and capabilities (chat, code, vision, reasoning, embeddings). This tutorial’s gemma4 matches the official quickstart; the catalog moves—trust the Library page.

      Names and tags

      The full reference is name:tag. Omitting the tag usually means :latest.

      FormMeaning
      gemma4Official chat example; the model’s default tag
      gemma4:cloudCloud variant of the same family; compute on Ollama Cloud
      qwen2.5A common alternative for Chinese and coding (pick a size tag)
      llama3.2 / other LlamaFrequent general-purpose stand-ins in ecosystem samples

      Typical tag kinds:

      • Size / parameters: 7b, 8b, 31b (numbers change per family)
      • Quantization: q4_K_M, q8_0 (not every model publishes every quant)
      • Capability: instruct, vision, dedicated embed models
      • cloud: xxx:cloud or a -cloud suffix in the name
      ollama pull gemma4
ollama pull qwen2.5
ollama show gemma4
ollama show --modelfile gemma4

      show prints family, quant, context limits, and more. When unsure, copy the ollama run ... line from the Library page.

      Quantization: Q4 vs Q8

      Weights are rarely shipped as full FP16. GGUF in Ollama stores each parameter in fewer bits:

      Band (simplified)IntuitionWhen to use
      Q4 (e.g. Q4_K_M)Roughly half the bytes, quality fine for daily chatLaptops with 8–16 GB VRAM / unified memory
      Q8Closer to original; size near “params × 1 byte”Spare VRAM, less quant noise
      Lower (Q2/Q3, …)Very small; more hallucinated junkSmoke tests only

      Memory arithmetic:

      1. Weights: 7B params × 0.5 bytes (~Q4) ≈ 3.5 GB; × 1 byte (~Q8) ≈ 7 GB.
      2. KV cache: grows with context length and concurrency. Official defaults scale with VRAM (about 4k below 24 GiB; larger cards get 32k / 256k-class windows). Agents, web search, and long repos often want 64k if VRAM allows.
      3. Headroom: desktop, browser, and IDE also use RAM. On Apple Silicon, weights and the OS share one unified pool.

      Practice: ollama run a small Q4 first and check ollama ps for 100% GPU. Heavy CPU offload means: smaller tag, shorter context, or Cloud.

      Cloud models

      Cloud models do not require a 70B/100B+ file on disk, but they still use the same CLI and API. Official cloud examples include gemma4:cloud and gpt-oss:120b-cloud; the live list is on Library.

      ollama signin
ollama run gemma4:cloud
# or
ollama pull gpt-oss:120b-cloud

      You can also skip the local daemon and call https://ollama.com/api with an API key (REST chapter). Cloud tags retire over time; official pages list replacements. Local pulls are not removed when a cloud SKU retires.

      For fully offline use, disable cloud features (see Cloud “Local only”) and only run tags without :cloud.

      Copy, rename, clean up

      Give a legacy tool an OpenAI-looking name (from the compatibility docs):

      ollama pull llama3.2
ollama cp llama3.2 gpt-3.5-turbo

      Then /v1/chat/completions can send "model": "gpt-3.5-turbo". Cleaner: put the real Ollama name in the client.

      ollama ls
ollama rm unused-tag

      rm deletes the local copy only. Change the store root with OLLAMA_MODELS (Installation).

      Importing weights

      Library is enough for most people. Advanced paths:

      • Modelfile FROM: an existing local name, a GGUF path, or some Safetensors trees—see Modelfile
      • GGUF on Hugging Face: learn license and quant on the Hugging Face course, then FROM ./model.gguf
      • Embedding models: do not embed with a chat model—see Embeddings

      Picking a starting point

      JobStarting point (still verify on Library)
      Match official docs, general chatgemma4
      Chinese writing / local KBStrong Chinese tags such as qwen2.5
      Local coding / small completionMid-size instruct / coder tags
      Huge reasoning, tiny VRAM*:cloud
      RAG vectorsembeddinggemma or another Library embed model

      Next steps

      评论