Set up Ollama

Run AI models locally — with no API costs, no data leaving your machine, and your choice of model. This guide walks through installing Ollama, running your first model, integrating it into an existing app, and customizing model behavior using Warp.

Prerequisites

Hardware requirements

You'll need enough memory to load the model weights. As a general estimate, plan on roughly 1GB of memory per billion parameters.

Model size	Example models	Minimum RAM/VRAM
3B	Llama 3.2 3B, Phi-3 Mini (3.8B)	4GB
7–8B	Mistral 7B, Llama 3.1 8B	8GB
13B	Llama 2 13B	16GB
22–34B	Codestral (22B), Mixtral (46.7B total)	24GB
70B	Llama 3.1 70B	48GB

On Apple Silicon Macs, unified memory is shared between CPU and GPU. An M2 Mac with 16GB unified memory can run 7–8B models comfortably.

Software requirements

• macOS 14 Sonoma+, Windows 10+, or Ubuntu 20.04+
• No account required — Ollama is free and open source

1. Install Ollama

macOS

Run the install script in your terminal:

curl -fsSL https://ollama.com/install.sh | sh

Or download the macOS app manually from ollama.com/download. The app runs as a background service and appears in your menu bar.

Linux

curl -fsSL https://ollama.com/install.sh | sh

On systemd-based Linux distributions, the installer typically registers Ollama as a service. To start it and enable it on boot, run:

sudo systemctl enable ollama
sudo systemctl start ollama

Windows

Run this in PowerShell:

irm https://ollama.com/install.ps1 | iex

Or download the installer manually from ollama.com/download. Ollama runs as a background service after installation.

2. Run your first model

Pull a model to download it locally. This example uses Llama 3.2 (3B), a lightweight model that runs on most hardware:

ollama pull llama3.2

Then start an interactive chat session:

ollama run llama3.2

Type your prompt and press Enter. Type /bye to exit the session.

To try a larger model with stronger reasoning:

ollama run llama3.1:8b

Use ollama ls to see all models you've downloaded, and ollama ps to see which models are currently loaded in memory.

3. Understand model types

When browsing ollama.com/library, you'll see labels on models that indicate their capabilities:

Label	Meaning
thinking	The model reasons step-by-step before answering; better for complex problems.
tools	The model can call external functions or utilities (e.g., web search).
vision	The model can process and respond to images.
embedding	Converts text to numeric vectors for search or RAG pipelines.

Quantized models (e.g., q4_0, q8_0 variants) use significantly less memory than full-precision originals. Start with a quantized version if you're memory-constrained.

4. Compare model performance

To benchmark models side by side, Warp makes it easy to run prompts in parallel tabs and compare results. Pass a prompt and the --verbose flag to get timing stats alongside the response:

ollama run --verbose llama3.2 "Summarize the Rust ownership model in one paragraph"

Run the same prompt on two different models to compare output quality and throughput before committing to one.

To see which models are currently loaded and their memory usage:

ollama ps

5. Integrate Ollama into your app

Ollama exposes an OpenAI-compatible REST API at http://localhost:11434/v1/. You can drop it in wherever you're currently using the OpenAI API — just update three values:

1. Base URL → http://localhost:11434/v1/
2. API key → any non-empty string (e.g., "ollama" — required by the SDK but ignored by Ollama)
3. Model name → the name of your local model (e.g., "llama3.2")

Python (OpenAI SDK)

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1/",
    api_key="ollama",  # required by the SDK but ignored by Ollama
)

response = client.chat.completions.create(
    model="llama3.2",
    messages=[
        {"role": "user", "content": "Explain async/await in Python"}
    ]
)

print(response.choices[0].message.content)

Node.js (OpenAI SDK)

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "http://localhost:11434/v1/",
  apiKey: "ollama", // required by the SDK but ignored by Ollama
});

const response = await client.chat.completions.create({
  model: "llama3.2",
  messages: [{ role: "user", content: "Explain async/await in Python" }],
});

console.log(response.choices[0].message.content);

Ollama also provides official native libraries if you prefer an SDK built specifically for Ollama rather than the OpenAI compatibility layer: pip install ollama for Python and npm i ollama for JavaScript. See the Ollama README for details.

Open your app's code in Warp and use the agent to locate the OpenAI client initialization, swap in the new values, and test the connection — all from the same terminal session.

6. Customize model behavior

You can create a custom model variant by writing a Modelfile — a configuration file that sets a system prompt, adjusts generation parameters, or builds on any existing Ollama model.

Create a file called Modelfile:

FROM llama3.2

SYSTEM """
You are a focused code review assistant.
Always identify security issues first, then logic errors, then style.
Keep feedback concise and actionable.
"""

PARAMETER temperature 0.3
PARAMETER num_ctx 8192

Then build and run your custom model:

ollama create code-reviewer -f ./Modelfile
ollama run code-reviewer

Your custom model is saved locally and available any time you run ollama run code-reviewer. Ask Warp's agent to generate a Modelfile for a specific use case — describe what you want the model to do, and let the agent write the configuration.

Productivity tips

• Unload to free memory — Use ollama stop <model> to unload a model from memory before loading a larger one.
• Script with the REST API — For quick testing without the SDK: curl http://localhost:11434/api/chat -d '{"model": "llama3.2", "messages": [{"role": "user", "content": "Hello"}], "stream": false}'
• Keep a model inventory — ollama ls shows all locally cached models with their sizes and download dates. Use ollama rm <model> to delete models you no longer need.
• Compare models in parallel — Open side-by-side Warp tabs and run the same prompt against two different models to compare quality and speed before deciding which to use in your app.

Next steps

You have Ollama installed, a model running locally, and a path to integrate it into any app using the OpenAI-compatible API.

Explore related guides and features:

• Set up Claude Code or Set up Codex CLI — use cloud-based agents alongside your local Ollama setup
• Run multiple agents at once — run Ollama and a cloud agent in parallel to compare outputs on the same task
• Ollama model library — browse all available models with size and capability details
• Ollama documentation — advanced configuration, GPU setup, and environment variables