#Introduction to Ollama

#What is Ollama?

Ollama is an open-source local LLM runtime: one command downloads a model, runs it on your machine (or Ollama Cloud), and exposes an HTTP API. It is maintained by Ollama; the engine lives at ollama/ollama.

It removes the “I want an open model but I do not want to assemble CUDA, vLLM, a tokenizer, and a gateway first” problem. After the desktop app or CLI is installed, ollama run gemma4 starts a chat. A background server listens on localhost:11434, so apps connect with REST or any OpenAI-compatible client.

#Where it sits in the stack

┌─────────────────────────────────────────────────────────────┐
│                   Your app / agent                          │
│   CLI · Python/JS · LangChain · Dify · Cursor · Hermes      │
└───────────────────────────┬─────────────────────────────────┘
                            │ HTTP
              ┌─────────────┴─────────────┐
              │  localhost:11434          │
              │  /api/*  or  /v1/*        │
              └─────────────┬─────────────┘
                            │
                    ┌───────┴───────┐
                    │    Ollama     │
                    │  schedule / KV│
                    └───────┬───────┘
              ┌─────────────┴─────────────┐
              │ Local weights (GGUF, …)   │
              │ or :cloud offload         │
              └───────────────────────────┘

#Local runtime vs cloud APIs vs vLLM

When to pick what:

• Personal dev, private drafts, offline demos, a local backend for an agent → Ollama
• SLA, global latency, no GPU ops → cloud API
• Many concurrent streams on server GPUs → vLLM (or similar)

Ollama also ships cloud models (for example gemma4:cloud): the same CLI and API, with large weights running in Ollama’s cloud. That is “keep the Ollama interface, move the FLOPs,” not a lock-in to a closed chat product.

#Privacy and boundaries

Default local path: prompts, context, and completions stay in a process on your machine. You do not open a cloud account to try one prompt. That fits experiments on code, draft records, and unpublished docs.

Boundaries you still own:

• ollama run xxx:cloud or https://ollama.com/api sends content to Ollama Cloud (sign-in or API key; their terms apply).
• The server binds to localhost by default. Setting OLLAMA_HOST to 0.0.0.0 or forwarding the port exposes the model on the LAN.
• Model licenses (Gemma, Llama, Qwen, …) are separate from “the file is on disk.” Read the Library page before commercial use.
• Embeddings and RAG store document chunks in a vector DB—backup and ACL are your problem.

#Good fit / be careful

Good fit: local Q&A and coding help, swapping the base URL of an OpenAI-shaped app, a LangChain agent’s local model, a Dify private provider, Cursor with a custom base URL.

Be careful: hundreds of concurrent production streams (prefer vLLM); forcing a 70B onto 8 GB RAM (use a small quant or Cloud); publishing an unauthenticated 11434 on the public internet.

#Next steps

• Installation
• Quick Start
• Models and quantization