Ollama Tokens per Second: What Sets Your Speed

Ask what tokens per second a given model will produce and the honest first answer is another question: on what memory, at what quantisation, with how much context already in the window. Generation speed for a single conversation is not primarily a function of how much compute a GPU has. It is a function of how fast that GPU can read memory.

Understanding why makes the number predictable in advance rather than something you discover after buying hardware.

Two phases, two different bottlenecks

An inference request has two distinct stages, and they are limited by different things.

Prefill, also called prompt evaluation, processes the tokens you sent. Every token in the prompt can be processed in parallel, which saturates the arithmetic units. This phase is compute-bound, and it is where a card’s raw processing throughput matters. Prefill rates are typically an order of magnitude higher than generation rates, which is why a long prompt often costs less waiting than the reply to it.

Decode, the generation phase, produces one token at a time. Each token depends on the one before it, so there is nothing to parallelise across. To produce a single token, the hardware must read the entire active weight set out of memory. This phase is memory-bandwidth-bound, and it is what people mean when they say “tokens per second”.

The distinction explains a common confusion: a card can be excellent at chewing through a long document and unremarkable at writing the summary.

The ceiling, and how to estimate it

Because decode reads the whole active weight set per token, the theoretical maximum is:

tokens per second ≈ memory bandwidth ÷ bytes of active weights

An 8B model at Q4_K_M occupies roughly 4.8 GB. On a card with about 1,000 GB/s of bandwidth, the ceiling is somewhere around 200 tokens per second. Real rates land lower, commonly in the 60 to 80 percent range of that ceiling, because attention over the KV cache, sampling, and framework overhead all consume time the simple division ignores.

Working the same arithmetic across memory classes shows the scale of the differences, using published bandwidth specifications from the TechPowerUp database:

Memory	Typical bandwidth	Ceiling for a 4.8 GB model	Rough expectation
Dual-channel DDR5 system RAM	~80 GB/s	~17 tok/s	Painful for chat
RTX 4060 Ti 16GB (128-bit GDDR6)	~288 GB/s	~60 tok/s	Usable
RTX 3060 12GB (192-bit GDDR6)	~360 GB/s	~75 tok/s	Comfortable
Apple M4 Max (unified)	~546 GB/s	~113 tok/s	Fast
RTX 3090 (384-bit GDDR6X)	~936 GB/s	~195 tok/s	Very fast
RTX 5090 (512-bit GDDR7)	~1,792 GB/s	~373 tok/s	Faster than you can read

The unified-memory row behaves differently from the rest: Apple Silicon reaches that bandwidth for a pool large enough to hold models no consumer card can, but the same chip ships in more than one memory-interface binning, so capacity and bandwidth have to be checked separately. Ollama on Apple Silicon tabulates both per chip. The system-RAM row at the top is the CPU-only case, worked through in Ollama without a GPU, where memory channel count rather than core count sets the ceiling.

Two things follow immediately. First, the roughly tenfold gap between system RAM and discrete GPU memory is why partial CPU offload is catastrophic rather than merely unfortunate. Second, past a certain point the number stops mattering, because human reading speed is roughly 5 to 10 tokens per second and anything above about 30 already streams faster than you can follow.

What actually changes the number

Quantisation. Dropping from Q8_0 to Q4_K_M nearly halves the bytes read per token, so it nearly doubles generation speed. This is the largest single lever, and unlike most speed optimisations it costs memory footprint rather than adding to it. The quality cost at Q4_K_M is small; below 4 bits it stops being small.

Model size. A 14B model reads roughly twice the bytes per token of a 7B model at the same quantisation, so it generates at roughly half the rate. Speed scales inversely with parameter count, almost exactly, as long as both fit in VRAM.

Whether it fits. This dominates everything else. A model at 47%/53% CPU/GPU is not running at 47 percent of full speed; it is running much closer to the CPU’s rate, because the slowest link sets the pace. Confirming the split with ollama ps is the first thing to check on any speed complaint, as covered in Ollama not using GPU.

Context length, gradually. The KV cache grows linearly with context, and attention must read it for every generated token. Speed therefore degrades as a conversation lengthens. The effect is modest at 4K and pronounced at 64K and beyond. Flash attention reduces it; KV cache quantisation reduces the memory cost that drives it.

Mixture-of-experts architecture. MoE models activate only a subset of their parameters per token. A 47B-parameter MoE with roughly 13B active reads far less per token than its total size suggests, so it generates at close to the speed of a 13B dense model while requiring VRAM closer to its full size. Capacity and speed come apart for these models in a way they do not for dense ones.

What does not change the number much

• CPU core count, once the model is fully on the GPU. The CPU is coordinating, not computing.
• PCIe generation, for a single card. Weights cross the bus once at load time. It matters greatly for multi-GPU layer splits and enormously for CPU offload.
• Sampling parameters. Temperature, top-p, and repeat penalty change output, not throughput.
• System RAM speed, when nothing has spilled into it. When something has, it becomes the only number that matters.

Measuring your own rate

Do not estimate when you can read the actual figure. The CLI reports it directly:

ollama run llama3.1:8b --verbose

After each response it prints total duration, load duration, prompt evaluation count and rate, and evaluation count and rate. The eval rate line is your tokens per second for generation; the prompt eval rate is your prefill throughput. Reading both separately is the point, because a slow-feeling session with a fast eval rate is a prefill problem, not a generation one.

The API returns the same data as fields on the final response object, with durations in nanoseconds:

curl -s http://localhost:11434/api/chat -d '{
  "model": "llama3.1:8b",
  "messages": [{"role": "user", "content": "Write one paragraph about memory bandwidth."}],
  "stream": false
}' | python3 -c "import sys,json; d=json.load(sys.stdin); print(round(d['eval_count']/d['eval_duration']*1e9, 1), 'tok/s')"

Two measurement pitfalls are worth avoiding. First, load_duration on a cold start includes reading the model off disk, which can dwarf everything else and has nothing to do with inference speed; discard the first run. Second, measure at a realistic context length. A benchmark taken on a two-token prompt overstates what you will see on a real document by a comfortable margin.

Reading numbers other people publish

Published tokens-per-second figures are frequently not comparable, because the variables that matter most are the ones most often omitted. Before treating a number as relevant to your setup, check that the quantisation is stated, the context length is stated, the split was full GPU offload, and the figure is generation rate rather than an average that folds in prefill.

A figure that specifies “8B at Q4_K_M, 4K context, 100% GPU” is useful. A figure that says “8B on an RTX 4090” is not, because it could plausibly differ by a factor of four depending on the details left out.

Setting expectations before you buy

The practical sequence is to establish capacity first and speed second, because a model that does not fit has no speed worth discussing. Work out the VRAM requirement with the VRAM calculator or the arithmetic in how much VRAM you need to run local LLMs with Ollama, filter hardware to what clears that bar, then divide bandwidth by model size to estimate the rate. The card comparison in best GPU for Ollama lays out both numbers side by side.

For multi-user serving, where requests are batched and the same weight read serves several tokens at once, the bandwidth arithmetic above stops applying and aggregate throughput becomes the metric that matters. That is a different discipline, covered at llmops.report.