Running open models locally with Ollama

This guide is for readers who want a language model running on their own computer and reachable through a local API. By the end you will have installed Ollama, chatted with a model in the terminal, and sent a request to the local server from Python.

What Ollama is

Ollama is a tool for downloading and running language models on your own hardware. It has a server component and a command-line client. The ollama command talks to the server, and ollama serve starts the server by hand when you need to. The project also publishes a Docker image, ollama/ollama, for readers who prefer containers.^[1][7]

Before you start

Check your hardware first, because it decides which models are practical. The macOS documentation asks for macOS Sonoma (v14) or newer and an Apple M series or x86 CPU, and it keeps models and configuration in ~/.ollama.^[5] On Windows you need Windows 10 22H2 or newer, in the Home or Pro edition, and the API is served at http://localhost:11434.^[6] Nvidia GPUs need compute capability 5.0 or higher and driver version 550 or newer, and Apple GPUs are accelerated through the Metal API.^[13]

Memory is usually the limit you hit first. The quickstart's small Gemma model downloads at about 7.2 GB, and it recommends 8 GB of available VRAM or unified memory on a Mac. Larger context windows need more memory on top of that.^[2]

Install it

Pick the instructions for your system. On macOS, mount ollama.dmg and drag Ollama into the system-wide Applications folder. The app offers to link the ollama command into /usr/local/bin.^[5] On macOS and Linux, the official one-line installer is:^[1]

curl -fsSL https://ollama.com/install.sh | sh

On Windows, run irm https://ollama.com/install.ps1 | iex in PowerShell.^[1]

On Linux, confirm the install with ollama -v, and start the server with ollama serve.^[7] The Linux page also suggests running the server as a service, with a unit file at /etc/systemd/system/ollama.service that sets ExecStart=/usr/bin/ollama serve.^[7] To pin a version, the same page passes it in an environment variable:^[7]

curl -fsSL https://ollama.com/install.sh | OLLAMA_VERSION=0.5.7 sh

On Windows you can move the model store by setting the OLLAMA_MODELS user environment variable, or install to a chosen folder with OllamaSetup.exe /DIR="d:\some\location".^[6]

Run your first model

The quickstart begins with one command for a small Gemma model:^[2]

ollama run gemma4:e2b

That downloads the model and opens a chat on your machine. Type /bye to leave.^[2] To download without starting a chat, use ollama pull gemma4. The CLI reference also lists the housekeeping commands: ollama ls shows installed models, ollama ps shows what is loaded, ollama stop gemma4 unloads a model, and ollama rm gemma4 deletes it.^[3]

The two models in this guide's title, Qwen3 and DeepSeek-R1, each have their own page in the Ollama library.^[14][15] The default Qwen3 model runs with:^[14]

ollama run qwen3

The library page lists qwen3:0.6b at 523MB, qwen3:1.7b at 1.4GB, qwen3:4b at 2.5GB and qwen3:8b at 5.2GB. The bigger variants are qwen3:14b at 9.3GB, qwen3:30b at 19GB, qwen3:32b at 20GB and qwen3:235b at 142GB, and the page lists 58 models in total.^[14]

DeepSeek-R1 runs with ollama run deepseek-r1, and an older local copy is brought up to date with ollama pull deepseek-r1.^[15] The page lists 35 models. The smaller distilled tags include deepseek-r1:1.5b at 1.1GB, deepseek-r1:7b at 4.7GB, deepseek-r1:8b at 5.2GB and deepseek-r1:14b at 9.0GB. The largest variants are deepseek-r1:32b at 20GB, deepseek-r1:70b at 43GB and deepseek-r1:671b at 404GB.^[15] For a laptop, the distilled tags are the place to start, for example with ollama run deepseek-r1:7b. The weights are licensed under the MIT License.^[15]

Call it from code

Ollama exposes two HTTP surfaces on the same port. The native API lives under http://localhost:11434/api, and an OpenAI-compatible API lives under http://localhost:11434/v1.^[8] The chat endpoints are http://localhost:11434/api/chat for native requests and http://localhost:11434/v1/chat/completions for OpenAI-style ones.^[2]

Because the compatible API sits under /v1, the OpenAI Python client can talk to it once its base URL points at the local server. The key must be present, but the local server ignores it.^[9] The compatibility page's example is:^[9]

from openai import OpenAI

client = OpenAI(
    base_url='http://localhost:11434/v1/',
    api_key='ollama',  # required but ignored
)

responses_result = client.responses.create(
  model='qwen3:8b',
  input='Write a short poem about the color blue',
)
print(responses_result.output_text)

The compatible surface includes /v1/chat/completions, /v1/completions, /v1/models, /v1/embeddings and /v1/responses. If a tool insists on an OpenAI model name, ollama cp llama3.2 gpt-3.5-turbo copies a model to that default name.^[9]

Customize a model with a Modelfile

A Modelfile is the blueprint for a customized model. Each line has the form INSTRUCTION arguments. FROM is the only required instruction. The others are PARAMETER, TEMPLATE, SYSTEM, LICENSE, MESSAGE, REQUIRES and CAPABILITY.^[4] The reference gives 2048 as the default num_ctx and 0.8 as the default temperature, and it notes that the Modelfile is not case sensitive.^[4]

A basic example sets PARAMETER temperature 1, PARAMETER num_ctx 4096 and a SYSTEM message. It is built with ollama create choose-a-model-name -f <location of the file e.g. ./Modelfile>.^[4]

To import a model in GGUF format, the Modelfile contains FROM /path/to/file.gguf, and you run ollama create my-model. Sharing goes through ollama cp mymodel myuser/mymodel and then ollama push myuser/mymodel.^[10]

To change the context size for the OpenAI-compatible API, the compatibility page uses a Modelfile line PARAMETER num_ctx <context size> followed by ollama create mymodel.^[9]

Context length and memory

Ollama picks a default context length from GPU memory: 4k below 24 GiB of VRAM, 32k for 24 to 48 GiB, and 256k at 48 GiB or more. To raise it, set OLLAMA_CONTEXT_LENGTH when you start the server:^[11]

OLLAMA_CONTEXT_LENGTH=64000 ollama serve

Bigger contexts cost memory. The FAQ says RAM needs scale with OLLAMA_NUM_PARALLEL, which defaults to 1, and with OLLAMA_CONTEXT_LENGTH.^[12]

Models stay loaded in memory for five minutes by default. You can unload one earlier with ollama stop, or control the timing with the keep_alive parameter.^[12] The server listens on 127.0.0.1 port 11434 by default, so only programs on the same machine can reach it.^[12]

Model files are stored in ~/.ollama/models on macOS, /usr/share/ollama/.ollama/models on Linux and C:\Users\%username%\.ollama\models on Windows.^[12]

Limits and common problems

The range of sizes is wide. The smallest Qwen3 tag is 523MB, while deepseek-r1:671b is a 404GB download, and qwen3:235b is 142GB.^[14][15] Pick a tag by the disk space and memory you actually have, not by the model name.

On a GPU with less than 24 GiB of VRAM, the default context is 4k, which can be too short for a long document. Raising it costs memory, as described above.^[11][12]

The hardware page lists Nvidia GPUs with compute capability 5.0 or higher and driver 550 or newer, plus Apple Metal acceleration. If your card falls outside that list, check the hardware page before you expect acceleration.^[13]

Because the server binds to 127.0.0.1 by default, a client on another machine cannot reach it without a change to that setting.^[12]

Next steps

The CLI reference is the next stop. It lists the full command set, including integrations started with ollama launch, such as ollama launch claude --model qwen3.5.^[3] For the API, read the introduction and the OpenAI compatibility page.^[8][9] To define your own models, read the Modelfile reference.^[4] Before you buy hardware for a larger model, read the hardware support page.^[13]