Best GPU for Ollama: 8GB to 48GB Cards Compared

Two numbers on a graphics card’s spec sheet decide almost everything about how it runs local models. VRAM capacity decides which models load at all. Memory bandwidth decides how fast they answer once loaded. Core counts, ray-tracing units, and clock speeds barely register by comparison, because single-stream token generation is a memory-bound workload rather than a compute-bound one.

That makes GPU selection for Ollama unusually simple to reason about, and unusually easy to get wrong if you shop by the numbers gaming reviews emphasise.

Capacity: what a card can hold

A model’s weights have to fit in VRAM, and so does the KV cache that grows with your conversation. Below roughly 4 bits per weight, quality degrades in ways that show up first in structured output and long-context reasoning, so capacity is a hard floor rather than something you can compress your way past indefinitely.

At the Q4_K_M quantisation most of Ollama’s library ships by default, budget about 0.6 GB per billion parameters for weights, then add 1 to 3 GB for cache and buffers. The full arithmetic, including how context length changes the answer, is worked through in how much VRAM you need to run local LLMs with Ollama, and the VRAM calculator on this site will do the sum for a specific model, quantisation, and context window.

The practical tiers that fall out of that:

• 8 GB holds a 7-8B model at Q4 with a modest context window, and little else. Best Ollama models for 8GB VRAM covers that tier in detail, including how much of the 8 GB the desktop takes before Ollama starts.
• 12 GB holds the same 7-8B class comfortably with room for a longer window, or a 13-14B model on a short one.
• 16 GB holds 13-14B at Q4 with real headroom.
• 24 GB holds 27-32B at Q4, which is where local output starts to feel qualitatively different.
• 32 GB and up opens the 70B class at aggressive quantisation, and 48 GB across two cards holds 70B at Q4 properly.

Bandwidth: how fast it answers

Generating each token requires reading the active weight set out of memory. That makes the theoretical ceiling for single-stream decoding roughly memory bandwidth divided by bytes read per token, and real-world rates land somewhere in the region of 60 to 80 percent of that ceiling once overheads are counted.

The consequence is blunt: a card with more VRAM but less bandwidth will load a bigger model and run it slower. That trade is often still worth taking, because a model that fits runs an order of magnitude faster than one that spills to system RAM, but it should be a decision rather than a surprise. The mechanics are covered in Ollama tokens per second.

The comparison

Capacity and bandwidth from published manufacturer specifications, cross-referenced against the TechPowerUp GPU database. Bandwidth figures are stock reference values.

Card	VRAM	Bus width	Memory bandwidth	Comfortable model class
RTX 3060 12GB	12 GB GDDR6	192-bit	~360 GB/s	7-8B at Q4, longer context
RTX 4060 Ti 16GB	16 GB GDDR6	128-bit	~288 GB/s	13-14B at Q4, slowly
RTX 5060 Ti 16GB	16 GB GDDR7	128-bit	~448 GB/s	13-14B at Q4
RTX 4070 Ti SUPER	16 GB GDDR6X	256-bit	~672 GB/s	13-14B at Q4, quickly
RTX 3090	24 GB GDDR6X	384-bit	~936 GB/s	27-32B at Q4
RTX 4090	24 GB GDDR6X	384-bit	~1,008 GB/s	27-32B at Q4
RTX 5090	32 GB GDDR7	512-bit	~1,792 GB/s	32B at Q8, or 70B tightly
RX 7900 XTX	24 GB GDDR6	384-bit	~960 GB/s	27-32B at Q4, ROCm permitting
Apple M4 Max	up to 128 GB unified	—	~546 GB/s	70B at Q4 with headroom
Apple M3 Ultra	up to 512 GB unified	—	~819 GB/s	70B at Q8, or larger

Two rows in that table deserve attention because they look like near-twins and are not. The RTX 4060 Ti 16GB and the RTX 4070 Ti SUPER carry the same 16 GB of VRAM, so they load the same models, but the narrow 128-bit bus on the 4060 Ti gives it well under half the bandwidth. Same capability, materially different speed. The RTX 5060 Ti closes much of that gap with GDDR7 on the same narrow bus, which is a good illustration of why bus width alone is not the number to read either.

Choosing by tier

Cheapest useful entry: 12 GB. An RTX 3060 12GB is frequently the lowest-cost path to a card that runs an 8B model entirely on the GPU with a usable context window. Its bandwidth is modest, so it is not fast, but “slow on the GPU” is a different category of experience from “fast on paper, half offloaded to the CPU”.

The awkward middle: 16 GB. This tier is where the spec sheet matters most, because the bandwidth spread across 16 GB cards is enormous. Buying 16 GB purely for the capacity number, on a narrow memory bus, produces a machine that loads 14B models and generates at a pace that discourages using them.

The sweet spot: 24 GB. A 24 GB card at high bandwidth runs the 27-32B class at Q4 entirely on the GPU, which is the point where local models stop feeling like a compromise for a lot of work. Used RTX 3090 cards remain the most commonly recommended route into this tier precisely because they pair 24 GB with 936 GB/s, a combination newer mid-range cards do not offer at any price.

Above that: capacity strategy changes. A single 32 GB card is fast but still cannot hold a 70B model at Q4 with real context. Two 24 GB cards can. Which route is better depends on whether you value throughput on mid-size models or the ability to load large ones at all.

AMD and Apple, with caveats

Ollama supports AMD hardware through ROCm, with broader device coverage available via Vulkan, per the hardware support documentation. The important caveat is that ROCm’s officially supported device list is narrower than AMD’s product range, and the ROCm system requirements are the document to check before buying rather than after. Cards outside that list frequently work, sometimes with an architecture-override environment variable, but “frequently works” is a different purchase decision from “supported”.

Apple Silicon is a genuinely different shape of answer. Unified memory means the GPU addresses system RAM directly, so a 64 GB machine can hold a 70B model at Q4 with no discrete card involved. Bandwidth sits between desktop system RAM and a high-end discrete GPU. For large models that would otherwise require multiple cards and a power budget to match, that is often the better trade; for mid-size models where a 24 GB card already fits, it is usually not.

Two details change the sizing on a Mac and have no equivalent on a discrete card: the GPU is allocated only part of the unified pool rather than all of it, and the same chip ships in more than one memory-interface binning, so identical capacity can mean substantially different bandwidth. Ollama on Apple Silicon sets out both, with a per-chip table.

NVIDIA cards need compute capability 5.0 or higher, which the CUDA GPUs list enumerates per model. Anything modern qualifies; the check matters only for older hardware being repurposed.

What not to optimise for

• CUDA core counts and gaming benchmarks. They predict prompt processing speed reasonably and token generation speed poorly. The bottleneck during generation is the memory subsystem.
• PCIe generation, for a single card. Weights are transferred once at load time. After that, a single-GPU setup barely touches the bus. PCIe bandwidth starts to matter when layers are split across multiple cards, and matters enormously when layers are split with system RAM.
• Cooling and case airflow, but only to a point. Sustained inference is a real thermal load, though a lighter one than sustained gaming on the same card.
• Total system RAM, once the model fits in VRAM. It matters a great deal if the model does not fit, because that is the memory the overflow lands in.

A buying checklist

1. Decide the largest model class you actually intend to run, not the largest you might one day try.
2. Compute its VRAM requirement at Q4_K_M, at the context length you need, using the VRAM calculator.
3. Add 1 to 3 GB of headroom, plus whatever your desktop session holds.
4. Filter cards to those meeting that capacity.
5. Among those, sort by memory bandwidth and buy the highest you can justify.
6. Confirm driver support before purchase, particularly for AMD.

Getting step 4 and step 5 in that order is the whole discipline. Capacity is a gate; bandwidth is a dial. A card that fails the gate cannot be fixed by any amount of tuning, which is why the most common local-LLM disappointment is a fast card that is two gigabytes too small. If that has already happened, Ollama not using GPU covers how to confirm the split and what to change.

For running this workload inside a virtual machine rather than on bare metal, proxmoxguide.com covers GPU passthrough tradeoffs. For serving models to more than one user, where batching changes the bandwidth arithmetic entirely, llmops.report covers deployment patterns.