Ollama API Examples: curl, Python, and OpenAI SDK

The Ollama API examples worth keeping cover five jobs on http://localhost:11434: POST /api/chat for conversation, /api/generate for one-shot completion, /api/embed for vectors, a JSON schema in format for structured output, and a tools array for function calling. Existing OpenAI SDK code also works once you point it at /v1.

Getting a response is easy. The trouble starts once a model sits behind a service: weights unloaded after five idle minutes, JSON that won’t parse, a prompt quietly cut at the context limit. Every example below returns the timing fields worth logging, and the second half of the post wires them into Prometheus.

Which Ollama API endpoint should you call?

Use /api/chat for anything conversational or tool-using, /api/generate for single prompts and fill-in-the-middle via suffix, and /api/embed for vectors. The rest of the API reference covers model management. The server listens on port 11434 and binds to 127.0.0.1 unless you set OLLAMA_HOST, per the Ollama FAQ.

In a container, the bind address is usually the first thing to break. Running Ollama in Docker covers the port mapping.

Example 1: non-streaming chat with curl

Streaming is the default on chat and generate. For scripts and health checks, turn it off so you get back a single JSON object:

curl -s http://localhost:11434/api/chat -d '{
  "model": "llama3.1:8b",
  "messages": [
    {"role": "system", "content": "Answer in one sentence."},
    {"role": "user", "content": "What is a KV cache?"}
  ],
  "stream": false,
  "keep_alive": "30m",
  "options": {"temperature": 0, "num_ctx": 8192, "seed": 42}
}'

The reply is at message.content, next to done_reason, prompt_eval_count, eval_count, and four durations (total_duration, load_duration, prompt_eval_duration, eval_duration), all in nanoseconds according to the chat endpoint docs.

num_ctx sets the context window for the request. Leave it out and you get the server default, which the context length docs tie to VRAM: 4k under 24 GiB, 32k from 24 to 48 GiB, 256k at 48 GiB and up. A bigger window needs a bigger KV cache, and how much VRAM you need does that arithmetic. keep_alive overrides the default five-minute idle unload.

Example 2: streaming NDJSON in Python

With stream: true the server returns newline-delimited JSON (application/x-ndjson), one object per chunk, and the final done: true object carries the timing fields. Parsing the stream yourself with requests lets you measure time-to-first-token (TTFT) from the client:

import json
import time
import requests

def stream_chat(model, messages, host="http://localhost:11434"):
    t0 = time.perf_counter()
    ttft = None
    payload = {"model": model, "messages": messages, "stream": True}
    with requests.post(f"{host}/api/chat", json=payload, stream=True, timeout=(5, 300)) as r:
        r.raise_for_status()
        for line in r.iter_lines():
            if not line:
                continue
            chunk = json.loads(line)
            if chunk.get("done"):
                return ttft, chunk  # durations and counts live here
            token = chunk["message"].get("content", "")
            if ttft is None and token:
                ttft = time.perf_counter() - t0
            print(token, end="", flush=True)

The official Python library (pip install ollama) handles this for you with chat(..., stream=True), and it has an AsyncClient for asyncio services.

Example 3: structured output with a JSON schema

Setting "format": "json" gets you valid JSON in no particular shape. Passing a full JSON schema constrains the shape too, and Pydantic will generate the schema:

from ollama import chat
from pydantic import BaseModel

class Incident(BaseModel):
    service: str
    severity: int
    root_cause: str | None

resp = chat(
    model="llama3.1:8b",
    messages=[{"role": "user", "content": "Extract: checkout-api paged at sev2, OOM on the GPU node."}],
    format=Incident.model_json_schema(),
    options={"temperature": 0},
)
incident = Incident.model_validate_json(resp.message.content)

The structured outputs guide recommends temperature 0 and repeating the schema in the prompt. Keep model_validate_json anyway. A response cut off by num_predict or the context limit is still truncated JSON, and you want that counted as a validation error, not surfacing as a KeyError three functions later.

Example 4: a tool-calling loop

The Python library turns typed, docstringed functions into tool schemas. This loop follows the agent example in the tool calling docs:

from ollama import chat

def get_gpu_util(node: str) -> int:
    """Return current GPU utilization percent for a node"""
    return 87  # stub: query your DCGM exporter here

tools = {"get_gpu_util": get_gpu_util}
messages = [{"role": "user", "content": "Is gpu-node-3 saturated?"}]

for _ in range(5):  # hard cap on tool rounds
    resp = chat(model="qwen3", messages=messages, tools=list(tools.values()))
    messages.append(resp.message)
    if not resp.message.tool_calls:
        break
    for call in resp.message.tool_calls:
        fn = tools.get(call.function.name)
        result = fn(**call.function.arguments) if fn else "unknown tool"
        messages.append({"role": "tool", "tool_name": call.function.name, "content": str(result)})

print(resp.message.content)

The docs use while True. The cap here is deliberate, because a small model that keeps calling the same tool will spin until your timeout fires. Only models trained for tool use emit tool_calls.

Example 5: batched embeddings for RAG

/api/embed accepts a string or an array and returns embeddings, one vector per input:

curl -s http://localhost:11434/api/embed -d '{
  "model": "embeddinggemma",
  "input": ["p99 latency spiked after the deploy", "KV cache filled at 32k context"],
  "truncate": false
}'

Send chunks in batches, not one request each, and set truncate: false while indexing. The default of true trims an oversized input to fit the context window without any warning. With false, the API docs say the request returns an error, which is your signal to re-chunk.

Example 6: the OpenAI SDK against a local server

For code already written against OpenAI, just change the base URL. According to the OpenAI compatibility docs, a local server requires an API key but ignores its value:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1/", api_key="ollama")
resp = client.chat.completions.create(
    model="llama3.1:8b",
    messages=[{"role": "user", "content": "Say hello in one sentence."}],
)
print(resp.choices[0].message.content)

Chat completions, completions, models, embeddings, and responses are supported. tool_choice, logit_bias, logprobs, and image URLs are not. The OpenAI response shape also has no field for Ollama’s nanosecond durations, which is the main reason to use the native endpoints for anything you monitor. Ollama vs LM Studio compares the two local servers.

The metric that matters: decode rate per request

Wall-clock latency is the obvious metric, but it’s the wrong one to alert on because it scales with output length. Log rates instead:

• decode rate = eval_count / eval_duration * 1e9 tokens/sec
• prefill rate = prompt_eval_count / prompt_eval_duration * 1e9 tokens/sec
• cold-load cost = load_duration / 1e9 seconds

Decode rate doesn’t depend on output length, and it moves when the hardware situation changes: a model partly spilled to CPU, a context window big enough to push layers off the GPU, or a second model competing for VRAM. Prefill rate catches long-prompt regressions in RAG pipelines. load_duration shows whether keep_alive is working. Ollama tokens per second explains where decode rate should land on given hardware.

Wiring it up

Record histograms, not summaries. Prometheus histograms aggregate across instances through histogram_quantile(), and precomputed summary quantiles don’t.

from prometheus_client import Counter, Histogram, start_http_server

TTFT = Histogram("ollama_ttft_seconds", "Client-observed TTFT",
                 ["model"], buckets=(0.1, 0.25, 0.5, 1, 2, 5, 10, 30))
DECODE = Histogram("ollama_decode_tokens_per_second", "eval_count / eval_duration",
                   ["model"], buckets=(5, 10, 20, 40, 60, 80, 120, 200))
LOAD = Histogram("ollama_load_seconds", "load_duration",
                 ["model"], buckets=(0.01, 0.1, 0.5, 1, 5, 15, 60))
LENGTH_STOPS = Counter("ollama_length_stops_total", "done_reason == length", ["model"])

def record(model, ttft, final):
    if ttft is not None:
        TTFT.labels(model).observe(ttft)
    if final.get("eval_duration"):
        DECODE.labels(model).observe(final["eval_count"] / final["eval_duration"] * 1e9)
    LOAD.labels(model).observe(final.get("load_duration", 0) / 1e9)
    if final.get("done_reason") == "length":
        LENGTH_STOPS.labels(model).inc()

start_http_server(9105)

Feed it the (ttft, final) tuple from stream_chat. p95 TTFT per model is then:

histogram_quantile(0.95, sum by (le, model) (rate(ollama_ttft_seconds_bucket[5m])))

What you’ll see

On a healthy system, each model’s decode-rate histogram is a tight band that holds steady as traffic changes. ollama_load_seconds stays in its lowest bucket, apart from a spike after any idle gap longer than keep_alive. TTFT p95 tracks prompt length.

Unhealthy patterns, roughly in the order you’ll meet them:

• Bimodal load histogram. Either two models are evicting each other or keep_alive is shorter than the gap between requests. GET /api/ps shows what’s resident.
• Decode rate steps down with no code change. The model is partly on CPU. ollama ps shows the split, and Ollama not using GPU walks through the fixes.
• TTFT climbs while decode rate holds. That’s queueing. OLLAMA_NUM_PARALLEL defaults to 1 request per model, and once OLLAMA_MAX_QUEUE (default 512) fills, new requests are rejected.
• Length stops rising. Responses are hitting num_predict or the context window. Thinking models spend output budget before they write any visible content.

Caveats

Silent truncation. A prompt longer than num_ctx doesn’t fail loudly. If prompt_eval_count sits pinned near your num_ctx, assume the input was cut.

Prefill under caching. On multi-turn chats, part of the prompt is served from cache (prompt_eval_cached_count), and prompt_eval_duration covers only the uncached part. That makes a naive prefill rate read high, so treat it as a trend line rather than a hardware number.

TTFT on thinking models. The clock above stops at the first content token, so reasoning time is included. Label it so nobody compares it to TTFT from a non-thinking model.

Cardinality. Label by model only. The Prometheus naming guidance warns against unbounded label values, so session and user IDs belong in logs.

Exposure. The local server has no authentication of its own. Binding OLLAMA_HOST to 0.0.0.0 exposes every endpoint, including /api/delete, to anyone who can reach the port.

FAQ

what port does the ollama api use

The Ollama API listens on port 11434 at http://localhost:11434 and binds to 127.0.0.1 by default. To accept connections from other machines or containers, set OLLAMA_HOST, for example to 0.0.0.0:11434. The server has no authentication of its own, so put a reverse proxy in front before exposing it.

is the ollama api compatible with openai

Yes. Ollama serves OpenAI-compatible endpoints under http://localhost:11434/v1/, covering chat completions, completions, embeddings, models, and responses. Set the OpenAI SDK’s base_url there and use any placeholder API key. Unsupported features include tool_choice, logit_bias, logprobs, and image URLs, so images have to be sent as base64.

how do i turn off streaming in the ollama api

Set "stream": false in the request body to get a single JSON object instead of newline-delimited chunks. Streaming is the default on /api/chat and /api/generate. In the Python library you just leave out stream=True, because non-streaming is its default. Use non-streaming for scripts and health checks.

how do i keep an ollama model loaded in memory

Pass "keep_alive": -1 on a request, or a duration such as "30m", to override the default five-minute idle unload. "keep_alive": 0 unloads the model immediately. To preload a model without generating anything, send a request with only the model name to /api/generate. GET /api/ps confirms what is loaded.

Related across the network

• LangChain with Ollama: Local Setup, Tools, and RAG — langchainguide.com
• LM Studio vs Ollama for Local LLMs: How to Actually Choose — lmstudioguide.com