How Much VRAM Do You Need to Run Local LLMs with Ollama?

Almost every question about running a local model comes down to one constraint: does it fit in your GPU’s memory, with room left for the conversation? Model choice, speed, and context length are all downstream of that. Get the arithmetic right and a local setup feels instant. Get it wrong by two gigabytes and the same model crawls, because Ollama has quietly pushed half the layers onto your CPU.

The one number that decides everything: weights in memory

A model’s weights have to live somewhere. The size of that block is parameter count multiplied by bits per weight, and quantisation is simply the knob that sets bits per weight.

Most models in Ollama’s library ship at Q4_K_M, which lands near 4.8 bits per weight once you account for K-quants keeping certain tensors at higher precision. In practical terms that is roughly 0.6 GB per billion parameters. An 8-bit build (Q8_0) is roughly 1.1 GB per billion, and an unquantised FP16 build is about 2 GB per billion. A few newer releases use other formats, such as the MXFP4 builds of gpt-oss, so check the tag.

Model size	Q4_K_M (default)	Q8_0	FP16
3B	~1.9 GB	~3.2 GB	~6.4 GB
7-8B	~4.8 GB	~8.5 GB	~16 GB
13-14B	~8.4 GB	~15 GB	~28 GB
27-32B	~19 GB	~34 GB	~64 GB
70B	~42 GB	~75 GB	~140 GB

Those are weights only. Budget 1 to 3 GB on top for the KV cache, compute buffers, and whatever the desktop already holds. On Linux with a display attached, check nvidia-smi first: a desktop session can hold several hundred megabytes before Ollama loads a layer.

To skip the arithmetic for a specific combination, the VRAM calculator on this site takes a model size, quantisation level, and context window and returns the memory total along with an estimated generation rate for a given card.

Quantisation: what you actually give up

Quantisation stores each weight in fewer bits, trading fidelity for memory and bandwidth, and the curve is not linear.

• Q8_0 is effectively indistinguishable from FP16 for most tasks, and rarely worth the doubled memory unless you are precision sensitive.
• Q5_K_M and Q4_K_M are the practical sweet spot. Q4_K_M loses little enough that most users never notice it in chat, summarisation, or code assistance.
• Q3 and below is where degradation becomes obvious: sloppier instruction following, structured output (JSON, tool calls) that starts breaking, and long-context reasoning that fails first. Squeezing a bigger model onto a small card this way is usually a bad trade.

The rule that holds up: a larger model at Q4 usually beats a smaller model at Q8 when both fit. Parameters buy you more than precision does, right up until you drop below about 4 bits.

For GGUF files from outside the Ollama library, the Hugging Face GGUF documentation explains the naming scheme, and a Modelfile can point at any local file with FROM ./model.gguf.

Context length is the second memory bill

The KV cache holds attention keys and values for every token in the conversation, and it grows linearly with context length. That catches people out: a model that loads fine at default settings can spill to CPU part-way into a long document.

Ollama’s default context window is not the model’s advertised maximum. Older releases pinned it at 2048 tokens, later ones at 4096. Current versions scale the default with available VRAM: the documentation gives roughly 4K below 24 GiB, 32K from 24 to 48 GiB, and 256K above that. On an 8 or 12 GB card that still means a 4K window, which is why a model advertising 128K context can behave as if it forgot the top of your document. Raise it per request with num_ctx, or server-wide with OLLAMA_CONTEXT_LENGTH.

Two levers reduce the cost of a long window:

• Flash attention cuts attention memory as context grows. Ollama enables it automatically when the backend and device support it; OLLAMA_FLASH_ATTENTION=1 forces it on and =0 turns it off.
• KV cache quantisation, set with OLLAMA_KV_CACHE_TYPE to q8_0 or q4_0 instead of the f16 default, roughly halves or quarters cache memory. It needs flash attention active, and it is server-wide rather than per request. q8_0 is safe; q4_0 saves more but is likelier to show as quality loss on long contexts.

What fits on your card

VRAM	Comfortable choice	Stretch
8 GB	7-8B at Q4_K_M, modest context	8B at Q4 with 8K context, tight
12 GB	7-8B at Q4 with long context	13-14B at Q4, short context
16 GB	13-14B at Q4_K_M with real headroom	8B at Q8_0
24 GB	27-32B at Q4_K_M	32B at Q4 with extended context
48 GB	70B at Q4_K_M, moderate context	70B with long context, tight

The 8 GB row is the one most people land on, and it is also the tier where the desktop’s own memory use decides whether a model fits. Best Ollama models for 8GB VRAM works through the realistic budget on that card and what to run against it.

Multiple GPUs help: Ollama splits a model’s layers across all visible cards automatically, so two 12 GB cards hold roughly what one 24 GB card holds. They do not match one card for bandwidth, since layers run in sequence and traffic crosses PCIe at the boundaries. Capacity scales cleanly, speed does not.

Capacity is only half of a purchase decision, though. Two cards with identical VRAM can differ by more than a factor of two in memory bandwidth, which sets generation speed once the model fits. Best GPU for Ollama compares both numbers by tier, from 8 GB up to 48 GB across two cards.

When it does not fit: partial offload

Ollama does not refuse to run a model that is too large. It loads as many layers as fit on the GPU and runs the rest on the CPU, which is the single most common cause of “why is my local model so slow.”

Check what happened:

ollama ps

The PROCESSOR column reports the split, for example 100% GPU or 47%/53% CPU/GPU. Anything short of 100% GPU means part of every token runs at system-RAM speed.

Single-stream token generation is bound by memory bandwidth, not compute, because each token reads the entire active weight set. Published specs make the gap plain: dual-channel DDR5 sits in the tens of gigabytes per second, while a discrete GPU on GDDR6, GDDR6X, or GDDR7 sits in the hundreds and, at the top end, past a terabyte. Roughly an order of magnitude, which is why one offloaded layer is a rounding error and forty are a wall.

If you land in partial offload, try these in order: drop to a smaller parameter count, reduce num_ctx, quantise the KV cache, then step down a quantisation level. Pin the layer count with the num_gpu option when you want deterministic behaviour instead of the automatic estimate. Ollama not using GPU walks the full diagnostic path, including the case where the card is not detected at all rather than merely too small.

Once the split does read 100% GPU, the speed you get is the speed the memory subsystem provides, and it becomes predictable: divide bandwidth by the size of the active weight set. Ollama tokens per second works through that estimate and how to measure the real figure with --verbose.

GPU, CPU, and unified memory

CPU-only inference works and is usable for 3B class models and batch jobs where latency does not matter. It is painful for interactive use at 13B and above, for the bandwidth reason above. The full arithmetic, including how memory channels change the answer and which workloads remain practical, is in Ollama without a GPU.

Ollama supports NVIDIA GPUs from compute capability 5.0 upward on a recent driver, AMD cards through ROCm with wider coverage via Vulkan, and Apple Silicon through Metal, per the hardware support notes. Apple Silicon is a special case: unified memory lets a 64 GB machine hold a 70B model at Q4 without a discrete card, and its bandwidth beats desktop system RAM while still trailing a high-end GPU. For large models that would otherwise need multiple cards, that trade is often worth it. The catch is that the GPU does not get the whole pool, which Ollama on Apple Silicon quantifies alongside the bandwidth figures for each chip.

Driving it from the API

The daemon listens on 127.0.0.1:11434. The two endpoints that matter are /api/generate for single completions and /api/chat for multi-turn, both covered in the API reference.

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.1:8b",
  "messages": [{"role": "user", "content": "Summarise this log line."}],
  "stream": false,
  "options": { "num_ctx": 8192, "temperature": 0.2 },
  "keep_alive": "30m"
}'

Three things about that payload:

• options takes per-request overrides including num_ctx, num_gpu, temperature, and seed. This is where you raise the context window without touching the environment.
• keep_alive controls how long the model stays resident after the request. The default is five minutes; -1 keeps it loaded indefinitely, 0 unloads immediately. On a shared workstation, an indefinite keep-alive on a 20 GB model gives up that memory permanently.
• stream defaults to true and returns newline-delimited JSON objects. Set it to false for a single response body.

There is also an OpenAI-compatible surface at http://localhost:11434/v1, so most SDK code can point at a local model by changing the base URL and passing any non-empty API key. Compatibility is partial, so verify anything using advanced parameters.

A sizing checklist

1. Decide the context length you actually need, not the model’s advertised maximum.
2. Multiply parameters by 0.6 GB for a Q4_K_M estimate.
3. Add 1 to 3 GB for KV cache and buffers, more for long context.
4. Subtract whatever your desktop session is already using.
5. Load the model, run ollama ps, and confirm 100% GPU.
6. Only then start tuning quantisation or context.

The order matters. Most people tune first and check the split last, which is how you end up optimising sampling parameters on a model that is running half on the CPU.

For production rather than workstation use, llmops.report covers telemetry and deployment patterns, and proxmoxguide.com helps if you are weighing GPU passthrough to a VM against bare metal.