Ollama

Overview

Ollama is a lightweight tool for running LLMs locally. It provides a simple CLI and API for downloading, managing, and running models. Ollama uses the GGUF format (from llama.cpp) and is optimized for consumer hardware (CPU and consumer GPUs). It’s the go-to tool for local LLM development and testing.

Key Features

Installation

# macOS / Linux
curl -fsSL https://ollama.com/install.sh | sh

# Or download from ollama.com

Usage

CLI

# Download and run a model
ollama run llama3

# List downloaded models
ollama list

# Pull a model without running
ollama pull mistral

# Remove a model
ollama rm llama3

# Show model info
ollama show llama3

API

# Chat completion (OpenAI-compatible)
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

# Text generation
curl http://localhost:11434/api/generate \
  -d '{"model": "llama3", "prompt": "Hello!"}'

Python

import requests

response = requests.post(
    "http://localhost:11434/v1/chat/completions",
    json={
        "model": "llama3",
        "messages": [{"role": "user", "content": "Hello!"}],
    },
)
print(response.json()["choices"][0]["message"]["content"])

Modelfile

Similar to a Dockerfile, defines model configuration:

FROM llama3

PARAMETER temperature 0.7
PARAMETER top_p 0.9
PARAMETER num_ctx 4096

SYSTEM You are a helpful Python programming assistant.

TEMPLATE """{{ if .System }}<|system|>
{{ .System }}<|end|>
{{ end }}{{ if .Prompt }}<|user|>
{{ .Prompt }}<|end|>
{{ end }}<|assistant|>
{{ .Response }}<|end|>
"""

# Create custom model
ollama create myassistant -f Modelfile
ollama run myassistant

Quantization Levels

Resource Requirements

Interview Questions

Q1: How does Ollama differ from vLLM?

Answer:

• Ollama: Local-first, simple CLI, GGUF format, CPU+GPU, no batching optimization, for development/testing
• vLLM: Production-grade, PagedAttention, continuous batching, FP16/GPTQ/AWQ, for production serving
• Use Ollama for development, vLLM for production.

Q2: What is GGUF and why does Ollama use it?

Answer: GGUF (GPT-Generated Unified Format) is the model format from llama.cpp. It supports various quantization levels (Q2-Q8), runs on CPU and consumer GPUs, and is self-contained (single file with metadata). Ollama uses GGUF because it’s optimized for consumer hardware, supports many quantization levels, and has a large ecosystem of pre-quantized models.

Common Mistakes

• ❌ Using Ollama for production serving (no continuous batching)
• ❌ Not checking available RAM before pulling large models
• ❌ Confusing Ollama (tool) with llama.cpp (library it’s built on)

Summary

Ollama is the simplest way to run LLMs locally. It wraps llama.cpp with a user-friendly CLI and API, supports GGUF quantization for consumer hardware, and is ideal for development and testing. For production, use vLLM or TensorRT-LLM instead.

Cross-References

• Quantization → GGUF format details
• Systems Overview → Comparison with other systems
• vLLM → Production alternative
• TGI
• ML Edge Deployment