Ollama Without a GPU: CPU-Only Speed and RAM

Ollama runs without a GPU. It does not require one, it does not warn you, and it will happily serve a 70B model on a laptop CPU at a rate measured in seconds per token. So the useful question is not whether CPU-only inference works. It is which model sizes remain usable, how much system RAM each one needs, and at what point the experience stops being worth having.

This is a different situation from a machine that has a GPU which is not being used. If a card is installed and ollama ps still reports CPU involvement, that is a fault to diagnose, and Ollama not using GPU covers it. What follows assumes there is genuinely no discrete GPU to use.

Why CPU inference is slow, precisely

Generating a token requires reading the entire active weight set out of memory. That makes single-stream generation bound by memory bandwidth rather than by processing power, which is why core counts predict CPU inference speed poorly and memory configuration predicts it well.

System memory bandwidth is straightforward arithmetic: channels multiplied by 8 bytes multiplied by the transfer rate.

Memory configuration	Calculation	Bandwidth
Dual-channel DDR4-3200	2 × 8 × 3200	~51 GB/s
Dual-channel DDR5-5600	2 × 8 × 5600	~90 GB/s
Dual-channel DDR5-6400	2 × 8 × 6400	~102 GB/s
Eight-channel DDR5-4800 (server)	8 × 8 × 4800	~307 GB/s
Twelve-channel DDR5-4800 (server)	12 × 8 × 4800	~461 GB/s

A discrete GPU sits in the hundreds of gigabytes per second and, at the top end, past a terabyte. A typical desktop sits near 90. That roughly tenfold gap is the entire performance story, and no amount of CPU tuning closes it.

Two consequences follow immediately. First, a single-channel configuration halves your inference speed, which makes a laptop with one memory stick installed a materially worse machine for this than the same laptop with two. Second, server platforms are unusually good at CPU inference relative to their core counts, because eight or twelve memory channels put them within range of an entry-level discrete card.

What speed to expect

Dividing bandwidth by weight bytes gives a ceiling. At Q4_K_M, weights run about 0.6 GB per billion parameters.

Model class	Weights at Q4	Ceiling on ~51 GB/s	Ceiling on ~90 GB/s	Ceiling on ~307 GB/s
1B	0.6 GB	~85 tok/s	~150 tok/s	~511 tok/s
3B	1.8 GB	~28 tok/s	~50 tok/s	~170 tok/s
8B	4.8 GB	~11 tok/s	~19 tok/s	~64 tok/s
14B	8.4 GB	~6 tok/s	~11 tok/s	~37 tok/s
32B	19 GB	~2.7 tok/s	~4.7 tok/s	~16 tok/s
70B	42 GB	~1.2 tok/s	~2.1 tok/s	~7 tok/s

Treat these as upper bounds, and pessimistic ones at that. A CPU typically achieves a lower fraction of its theoretical bandwidth than a GPU does, because the memory controller is shared with everything else the system is doing and the access pattern is less favourable. Expect real rates somewhere around half to two-thirds of the figures above.

Against a reading speed of roughly 5 to 10 tokens per second, the practical dividing lines fall out:

• Under 4B parameters: genuinely usable. Fast enough to read along with on ordinary desktop memory.
• 7-8B: tolerable on DDR5, tedious on DDR4. Around reading speed at best. Fine for short answers, wearing for long ones.
• 14B and above: not interactive. Workable for batch jobs that run unattended, not for conversation.
• 70B: measured in minutes per response. It runs. That is the most that can be said.

RAM sizing

The arithmetic is the same as for VRAM, worked through in how much VRAM you need to run local LLMs with Ollama, except the memory in question is system RAM shared with the operating system. Weights plus KV cache plus buffers, then leave room for everything else the machine is doing.

Model class	Weights at Q4	Sensible total system RAM
1-3B	0.6 to 1.8 GB	8 GB
7-8B	~4.8 GB	16 GB
12-14B	~8.4 GB	16 to 32 GB
27-32B	16 to 19 GB	32 to 48 GB
70B	~42 GB	64 GB

Two allocations are easy to forget. The KV cache is charged against the same pool, so a long context window costs system RAM here exactly as it costs VRAM on a card. And per the Ollama FAQ, OLLAMA_MAX_LOADED_MODELS defaults to 3 for CPU inference, meaning three separate models can sit resident simultaneously. On a memory-constrained machine, lowering that is worthwhile.

If the total exceeds physical RAM, the operating system begins swapping to disk, and that is categorically worse than slow inference. Response times move from slow to unusable. Size so the model fits in RAM with room left over, or run a smaller one.

What actually helps

More memory channels. The single highest-impact change. Populating both channels on a dual-channel board is close to a doubling.

Faster memory. DDR5-6400 over DDR5-4800 is a direct proportional gain, unlike most memory-speed upgrades where the effect is marginal.

A smaller model. Speed scales inversely with parameter count almost exactly. Dropping from 8B to 3B is roughly a 2.5x speedup.

Lower quantisation, more than usual. On a GPU, quantisation is primarily a capacity decision. On a CPU it is a speed decision too, because fewer bits per weight means fewer bytes read per token. Q4_K_M over Q8_0 is close to a doubling.

Mixture-of-experts models. These activate only a subset of parameters per token, so bytes read per token are far below total model size. An MoE model with a small active parameter count generates much faster than a dense model of the same footprint, which makes MoE architectures disproportionately attractive for CPU inference specifically.

Instruction set support. Ollama’s inference runs on llama.cpp, which uses vectorised kernels selected for the available instruction set. AVX2 is the practical baseline on x86; hardware without it will be considerably slower. This mostly affects prompt processing, which is compute-bound, rather than generation.

What does not help much

• Core count, past a point. Generation saturates memory bandwidth well before it saturates cores. Beyond roughly the physical core count there is little to gain, and oversubscribing threads can hurt. The num_thread option sets it if the automatic choice is wrong.
• CPU clock speed. A faster core still waits on the same memory bus.
• A faster SSD. It changes model load time, not inference speed. Discard the first run when timing anything, since load_duration on a cold start is dominated by disk.
• An integrated GPU, for generation. This one surprises people. Ollama can use an iGPU through Vulkan, but an iGPU reads the same system memory over the same bus, so the bandwidth ceiling is identical. The gain is in prefill, which is compute-bound, not in generation. Faster prompt processing, same tokens per second.

Running CPU-only deliberately

Sometimes the point is to keep a GPU free for something else. The hardware support documentation notes that an invalid GPU ID forces CPU usage:

CUDA_VISIBLE_DEVICES=-1 ollama serve      # NVIDIA
ROCR_VISIBLE_DEVICES=-1 ollama serve      # AMD ROCm

Vulkan is enabled by default and covers a wide range of devices including integrated graphics, so on a machine with an iGPU it may pick that up even with the discrete card hidden. Disable it as well:

OLLAMA_VULKAN=0 ollama serve

Set these in the service environment rather than an interactive shell, or a systemd-managed daemon will not inherit them. Confirm the result with ollama ps, which should report 100% CPU.

Where CPU-only genuinely makes sense

It is a poor way to run a chat assistant and a perfectly good way to run several other things:

• Embeddings. Embedding models are small and the work is a single forward pass with no autoregressive loop, so the bandwidth penalty barely applies. CPU embedding generation is entirely practical.
• Batch and offline jobs. Classification, extraction, and summarisation over a queue overnight do not care about latency.
• Small models in a pipeline. A 1-3B model doing routing, tagging, or structured extraction runs at a useful rate on any modern desktop.
• Development and testing. Getting an integration working against a real endpoint does not require production speed.
• Privacy-constrained work on existing hardware. Slow and local can beat fast and remote when the data cannot leave.

When to stop and buy a card

If the goal is interactive use of a 7B model or larger, a discrete GPU is the answer, and a modest one is transformative rather than incremental. Even an entry-level card with 8 GB delivers several times the memory bandwidth of dual-channel desktop RAM, which is the difference between reading along with the output and waiting for it. Best Ollama models for 8GB VRAM covers what that tier holds, and best GPU for Ollama compares capacity against bandwidth across tiers.

Two alternatives deserve a mention before buying. A Mac with unified memory reaches large-model capacity without a discrete card at all, at bandwidth between desktop RAM and a GPU, as Ollama on Apple Silicon sets out. And a server platform with eight or more memory channels is a legitimate CPU inference machine rather than a compromise, which is worth knowing if such hardware is already available.

Before deciding either way, measure what the current machine does. The VRAM calculator gives the memory a given model and context needs, and ollama run <model> --verbose prints the actual eval rate after every response. The method for reading those numbers correctly is in Ollama tokens per second. A measured 12 tokens per second on a 3B model may well be enough, in which case there is nothing to buy.