Ollama Not Using GPU: How to Diagnose and Fix It

Ollama does not refuse to run a model that is too large for your GPU. It loads as many layers as fit and quietly runs the rest on the CPU. Nothing errors, nothing warns in the chat interface, and the only symptom is that generation is five to twenty times slower than expected. This is by far the most common cause of “my local model is slow”, and it is diagnosable in about thirty seconds.

Step 1: read the split, before changing anything

With a model loaded, run:

ollama ps

The PROCESSOR column reports where the layers actually landed:

100% GPU means full offload. Your GPU is being used, and slow generation here is a different problem, covered in Ollama tokens per second.
47%/53% CPU/GPU or similar means partial offload. Part of every token is being computed at system-RAM speed.
100% CPU means the GPU is not being used at all, which points at a detection or support problem rather than a capacity one. On a machine with no discrete GPU installed this is simply expected, and Ollama without a GPU covers what to expect from CPU-only inference instead.

The same output carries a CONTEXT column reporting the window actually allocated for the loaded model. That figure is worth reading alongside the split, because a larger context than you intended is a common reason a model that should fit does not.

Those three outcomes have different causes and different fixes, so establish which one you have before touching a setting. Skipping this step is how people spend an afternoon tuning sampling parameters on a model that was never on the GPU.

The server log carries the reasoning behind the decision. On Linux with a systemd service:

journalctl -u ollama --no-pager | tail -n 60

Look for the lines reporting detected devices, available memory, and the layer count offloaded. That log states what Ollama found and what it decided, which is more reliable than inference from symptoms.

Step 2: is the GPU visible at all?

If the split is 100% CPU, the question is whether the device was detected.

nvidia-smi          # NVIDIA
rocm-smi            # AMD with ROCm

If those commands fail or report nothing, the problem sits below Ollama and no Ollama setting will fix it. Driver installation, a reboot after a driver update, or a container missing GPU passthrough are the usual causes.

Common causes of an invisible GPU:

Driver not loaded after an update. A kernel or driver upgrade without a reboot leaves the userspace library and kernel module mismatched. nvidia-smi reports the mismatch directly.
Running in Docker without GPU access. The container needs the NVIDIA container toolkit and --gpus=all, or the equivalent device flags for AMD. Without them the container simply has no GPU.
Unsupported compute capability, or a driver below the floor. The hardware support documentation gives two requirements, and the second is the one people miss: compute capability 5.0 or higher, and driver version 550 or newer, rising to 570 or newer for the oldest supported cards in the 5.0 to 6.2 range. A card that qualifies on paper will still be ignored on a distribution shipping an older driver branch. Check the version in the top-right of nvidia-smi output before concluding the card is unsupported.
CUDA_VISIBLE_DEVICES set to an empty or wrong value. A leftover export in a shell profile or systemd unit hides every device. Check the service environment as well as your shell.
The service runs as a different user. A systemd unit running as a user without access to the render or video group may not open the device even though your interactive shell can.

Step 3: it is detected, but the model does not fit

If nvidia-smi works and the split still shows CPU involvement, this is a capacity problem. The model plus its KV cache plus the compute buffers exceed free VRAM.

Check what is free, not what is installed:

nvidia-smi --query-gpu=memory.total,memory.used,memory.free --format=csv

A desktop session, a browser with hardware acceleration, or another loaded model can hold gigabytes before Ollama starts. On a machine with a display attached, that overhead is real and it is the difference between fitting and not fitting more often than people expect.

Then check whether the model should fit. At the Q4_K_M default, weights run about 0.6 GB per billion parameters, plus 1 to 3 GB for cache and buffers. The VRAM calculator will do that arithmetic for a specific model, quantisation, and context length, and how much VRAM you need to run local LLMs with Ollama explains where each term comes from.

Fixes, in the order that costs you the least quality:

Close whatever else holds VRAM. Another resident model is the most common culprit. ollama ps lists them; ollama stop <model> unloads one.
Reduce the context window. The KV cache grows linearly with context length, so this is often the cheapest large saving. Set num_ctx per request, or OLLAMA_CONTEXT_LENGTH server-wide.
Quantise the KV cache. OLLAMA_KV_CACHE_TYPE=q8_0 roughly halves cache memory at negligible quality cost; q4_0 saves more and is likelier to show on long contexts. This requires flash attention to be active.
Step down a model size. An 8B model fully on the GPU beats a 14B model half on the CPU, comfortably. On an 8 GB card in particular, the 14B class does not fit at Q4 by any arrangement of settings; best Ollama models for 8GB VRAM sets out what that tier does hold.
Only then step down a quantisation level. Q4_K_M to Q3 is where quality loss becomes noticeable, particularly for structured output and tool calls.

Step 4: context length as a hidden cause

A model that loads cleanly can fall out of full GPU offload part-way through a long conversation, because the KV cache grows as the context fills. If ollama ps reported 100% GPU at load and generation slowed dramatically after a long document was pasted in, this is the mechanism.

Ollama’s default context window scales with available VRAM rather than being fixed, so the same model behaves differently on different machines. Raising num_ctx to a model’s advertised maximum on a card that cannot hold the resulting cache is a reliable way to force partial offload.

Step 5: AMD-specific checks

AMD support runs through ROCm, with wider device coverage through Vulkan. The officially supported device list in the ROCm system requirements is narrower than AMD’s shipping product range, and unsupported-but-similar cards frequently need HSA_OVERRIDE_GFX_VERSION set to a supported architecture string before ROCm will use them.

If ROCm cannot be made to work on a given card, the Vulkan backend is the fallback and covers a wider range of devices, generally at some performance cost. Confirm which backend is active from the server log rather than assuming.

Step 6: forcing the behaviour you want

Ollama estimates how many layers fit. The estimate is usually good, and occasionally conservative. To pin it:

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.1:8b",
  "messages": [{"role": "user", "content": "hello"}],
  "options": { "num_gpu": 33, "num_ctx": 8192 }
}'

num_gpu sets the number of layers placed on the GPU. Setting it above what fits will cause an out-of-memory failure rather than a graceful fallback, which is arguably the more useful behaviour: an explicit error beats silent slowness.

Two environment variables are worth knowing:

OLLAMA_FLASH_ATTENTION=1 forces flash attention on, reducing attention memory as context grows. Ollama enables it automatically where supported, so this matters mainly when confirming it is active.
OLLAMA_SCHED_SPREAD=1 spreads a model across all available GPUs rather than fitting it onto the fewest. On a multi-GPU machine this trades some efficiency for capacity.

For multi-GPU systems where one card should be reserved, CUDA_VISIBLE_DEVICES restricts which devices Ollama sees. Set it in the service environment, not just your shell, or the daemon will not inherit it.

The diagnostic order that works

Symptom	First check	Likely cause
`100% CPU` in `ollama ps`	`nvidia-smi` / `rocm-smi`	Driver, container, or device support
Partial split at load	Free VRAM vs model size	Model plus cache exceeds capacity
Full GPU, then slows mid-chat	`num_ctx` and cache growth	KV cache spilling as context fills
Worked yesterday, not today	Driver version, resident models	Update without reboot, or memory held
`100% GPU` and still slow	Memory bandwidth of the card	Not an offload problem at all

That last row matters. Once the split reads 100% GPU, offloading is no longer the issue and the speed you are getting is the speed the hardware provides. What sets that number is covered in Ollama tokens per second, and which cards deliver what is compared in best GPU for Ollama.

The general principle: verify the split first, verify capacity second, and change settings third. Reversing that order turns a thirty-second diagnosis into an afternoon.