Kenhuang Academy
  • English

      • vLLM vs Ollama

      One line: Ollama optimizes local developer experience; vLLM optimizes GPU throughput in production. Both can expose OpenAI-style /v1, so you can write the app against Ollama and later switch base_url to vLLM. Weights and model cards still live on Hugging Face.

      Comparison

      OllamavLLM
      Product goalInstaller + ollama run for local (or Cloud) chatInference engine + HTTP gateway to saturate datacenter GPUs
      Happy hardwareConsumer NVIDIA / Apple Silicon / CPU is OKNVIDIA CUDA GPU first; CPU is not the point
      BatchingInteractive, single machineContinuous batching, many in-flight requests
      KV cacheRuntime-managedPagedAttention paging
      Model formatLibrary tags, typically GGUF quantizationsHF checkpoints, AWQ / GPTQ / FP8, …
      Default port11434 (/api and /v1)8000 (/v1)
      Swap modelsollama pull / change a tagChange --model, reload the process
      Multi-GPULimited, desktop-oriented--tensor-parallel-size and friends
      OpsTray app, light systemdCUDA, shared memory, replicas, rate limits
      Privacy defaultLoopback; data need not leave the machineAlso self-hosted; cluster boundaries are yours

      When to use Ollama

      • Prompt iteration, demos, and offline drafts on a laptop
      • You want a reply from ollama run in minutes
      • No NVIDIA server, or you only have Apple Silicon
      • Agent work needs a “good enough” local OpenAI-compatible endpoint

      Follow the Ollama tutorial: install → run → /v1. You do not need PagedAttention first.

      When to switch to vLLM

      • Concurrent users grow; Ollama queues while the GPU is still underfilled
      • You have A10 / A100 / L40S / H100 and tokens/s is an SLA
      • Inference must stay in your VPC, with existing OpenAI SDK / LangChain clients
      • Offline jobs over many prompts (synthetic data, evals) that benefit from continuous batching

      Migration is usually: same chat.completions code, new base_url and model string, then vllm serve on the GPU box.

      Where Hugging Face fits

      QuestionGo here
      Find ungated models, licenses, tokenizersHugging Face tutorial and the Hub
      Fine-tune LoRA / change architectureTransformers + PyTorch—not vLLM’s main job
      Serve a trained Instruct repo at high QPSvLLM (or a similar engine)
      Let teammates chat locallyOllama (if a GGUF / import path exists) or a Transformers script

      Mental model: Hub stores weights → Ollama for local play → vLLM for serving.

      Mix-up pitfalls

      • Passing an Ollama library name (gemma4) to vLLM --model: vLLM wants a Hub ID or a local directory.
      • Assuming quant files are interchangeable: GGUF ≠ AWQ.
      • Running Ollama and vLLM on the same consumer GPU: they fight for VRAM—ollama stop / kill the other process first.
      • Treating the vLLM Docker image as a graphical model store: there is no Ollama Library UX.

      Migration checklist

      1. Prove the product path against Ollama with the OpenAI SDK.
      2. serve an ungated HF Instruct repo on vLLM (this course: Qwen/Qwen2.5-0.5B-Instruct).
      3. Change only base_url=http://<gpu-host>:8000/v1 and model=.
      4. Compare quality on production prompts; then add quantization / tp.
      5. Add a reverse proxy and rate limits; do not rely on --api-key alone.

      Next steps

      评论