How to Run Ollama in Docker: CPU, NVIDIA and AMD GPU Setup

The short version of how to run Ollama in Docker is one command: docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama. It starts, it answers on port 11434, and on most machines it is also quietly running every model on the CPU. A container cannot see a GPU unless the host runtime hands one over, and Ollama does not error when that hand-over is missing. It just runs slow. This guide covers the CPU one-liner, NVIDIA and AMD passthrough, how to confirm the GPU actually landed, a Compose file, and the network exposure the default command creates.

The CPU-only container

The flags matter, since every later variant builds on them. From the official Docker docs:

docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

• -v ollama:/root/.ollama is a named volume for the model store. Without it, every pulled model dies with the container. The default llama3.2 tag is 2.0 GB per its library page, and larger models run to tens of gigabytes, so put the volume somewhere with room.
• -p 11434:11434 publishes the API on every host interface. More on that below.
• ollama/ollama is the Docker Hub image. It ships linux/amd64 and linux/arm64 builds and version tags such as 0.33.2 alongside latest, so pin one in anything you expect to keep running.

Pull and run a model through the container’s own CLI:

docker exec -it ollama ollama run llama3.2

Or through the API from the host:

curl http://localhost:11434/api/pull -d '{"model": "llama3.2"}'
curl http://localhost:11434/api/generate -d '{"model": "llama3.2", "prompt": "Why is the sky blue?"}'

What to expect from a CPU-only container is a memory-bandwidth question, not a Docker one. Ollama without a GPU covers the numbers.

NVIDIA GPU passthrough

Docker cannot see a GPU by itself. The NVIDIA Container Toolkit installs a runtime hook that maps the driver and device nodes into the container. Install the host driver first, then on Debian and Ubuntu:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Fedora and RHEL take the .repo file from the same guide and dnf install nvidia-container-toolkit, then the same nvidia-ctk and restart lines. Confirm the runtime works before involving Ollama, using the check from Docker’s GPU docs:

docker run -it --rm --gpus all ubuntu nvidia-smi

If that prints your card, start Ollama with the same flag:

docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

--gpus device=0 or --gpus '"device=0,2"' restricts the container to particular cards. Ollama also honours CUDA_VISIBLE_DEVICES per its GPU docs, which list compute capability 5.0 and driver version 550 as the floor.

Two platforms need a different answer. On macOS, Ollama’s own Docker image announcement says to run it as a standalone application, because Docker Desktop does not pass the GPU through. On Windows, Docker Desktop’s GPU support exists only on the WSL 2 backend with an NVIDIA card and a driver that supports GPU paravirtualization.

AMD GPU passthrough

AMD cards use a separate image tag and device mounts instead of the --gpus flag:

docker run -d --device /dev/kfd --device /dev/dri -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama:rocm

Per AMD’s container docs, /dev/kfd is the compute interface shared by all GPUs and /dev/dri holds the per-GPU render nodes. The rocm tag is linux/amd64 only. Ollama’s GPU docs require the ROCm v7 driver on Linux, list the supported LLVM targets, and give HSA_OVERRIDE_GFX_VERSION as the override for cards outside that list. On SELinux hosts the same page recommends sudo setsebool container_use_devices=1 when the container cannot open the devices. The default image also bundles Vulkan, enabled whenever the container can reach the GPU device nodes, with OLLAMA_VULKAN=0 to turn it off.

Confirm the GPU actually landed

A container started with a GPU flag and a model running on the CPU is the normal failure. Nothing in the API response tells you. Three checks, cheapest first:

docker logs ollama
docker exec -it ollama ollama ps
curl http://localhost:11434/api/ps

The startup log lists the compute libraries Ollama found. ollama ps has a PROCESSOR column that should read 100% GPU for a model that fits. A CPU/GPU split means only part of it loaded, which is a capacity problem covered in Ollama not using GPU and the VRAM sizing guide. The /api/ps response carries size_vram in bytes, the field to scrape if you graph this. Ollama’s troubleshooting page holds the container-specific fixes: reload nvidia_uvm if the toolkit test passes but Ollama sees nothing, and if the GPU works at start and then drops to CPU later, switch Docker to the cgroupfs cgroup driver in /etc/docker/daemon.json.

A Compose file that survives reboots

For anything left running, a Compose file records the flags and restarts with the host. The reservation syntax comes from Docker’s Compose GPU docs, where capabilities is mandatory and count and device_ids are mutually exclusive:

services:
  ollama:
    image: ollama/ollama:0.33.2
    container_name: ollama
    restart: unless-stopped
    ports:
      - "127.0.0.1:11434:11434"
    volumes:
      - ollama:/root/.ollama
    environment:
      OLLAMA_KEEP_ALIVE: "30m"
      OLLAMA_MAX_LOADED_MODELS: "1"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

volumes:
  ollama:

The environment block changes two defaults from the Ollama FAQ. Models unload after 5 minutes idle, which means a cold reload and a slow first token after any lull. A longer keep-alive trades idle VRAM for that. The loaded-model cap defaults to 3 per GPU, and pinning it to 1 stops a second model from evicting the one you care about on a single card. OLLAMA_NUM_PARALLEL stays at its default of 1. On AMD, swap the deploy block for devices entries mapping /dev/kfd and /dev/dri, and use the rocm tag.

Podman users on Fedora and RHEL get the NVIDIA device through CDI instead: sudo nvidia-ctk cdi generate --output=/var/run/cdi/nvidia.yaml, then --device nvidia.com/gpu=all --security-opt=label=disable on the run line, per the toolkit’s CDI guide.

Port 11434 is not a login screen

The Compose file above binds the port to 127.0.0.1 on purpose. The stock docker run line publishes on every host interface, which Docker’s port-publishing docs describe as insecure by default. Inside the container the image listens on 0.0.0.0, the process runs as root, and Ollama has no authentication. Wiz’s CVE-2024-37032 disclosure put those together: a path traversal in /api/pull gave remote code execution as root on exposed Docker deployments, was fixed in 0.1.34, and their scan found over 1,000 vulnerable instances on the public internet. The serving layer keeps producing entries like it, which the AI Alert CVE roundup tracks. Bind to localhost, put a reverse proxy with authentication in front for anything remote (the FAQ has an nginx example), and pin the image tag so an upgrade is a decision rather than a surprise on the next docker pull.