Home / Learn / Data & AI / Offline AI / Lesson 17.3
All lessons

Module 17: Pros, cons and limits · Lesson 17.3 · about 20 min

Memory limits: model, context and cache

What fills your machine's memory when a model runs (weights, KV cache, working space), why long contexts are the hidden cost, and every way to make a model fit.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 17 · about 20 minutes · read first: 4.1, 8.4 and 10.3. You’ll be able to tell why a model does not fit and pick the cheapest fix.

Weights, the KV cache for the context and the engine's working space must all fit in fast memory at once, and the cache, which grows with every token of context, is the part people forget.

1. Why this matters

“Out of memory” is the most common local AI error. The second most common is a model that quietly spills into slower normal memory and crawls. Both come from the same budget: everything must fit in the fast memory at once.

2. The three shares

Picture a desk. The weights are the reference books that never leave it; the KV cache is the pile of notes that grows with every page you read; working space is the elbow room you need to write.

  1. Weights: parameters × bits per weight ÷ 8. A 27B model at 4.5 bits is about 15 GB. Fixed.
  2. KV cache: the model’s notes on every token it has read so far, so it does not reread them (Lesson 8.4). It grows with every token of context, for every parallel user (Lesson 15.1).
  3. Working space: the engine’s scratch memory, plus whatever the operating system and other apps hold. On a Mac or AI box, the system shares the same memory.

Try a model, quant, context and device:

Try it: will it fit?

model weights conversation memory (KV cache) working room · line = what this device can give the model

Rough rules: weights = billions of parameters × bits ÷ 8; conversation memory ≈ 0.01 GB per billion parameters per 1,000 tokens; 1 GB working room; top speed ≈ memory bandwidth ÷ weights read per token. Real numbers vary by model and engine by 20% or more. Usable memory leaves room for the system (Macs about three quarters). Device speeds are makers' figures and the sources in this lesson.

And the KV cache on its own, from real model designs:

Calculator: the KV cache

Formula: 2 × layers × KV heads × head size × bytes × tokens. In config.json these are num_hidden_layers, num_key_value_heads and head_dim (or hidden size ÷ attention heads). Some models (Gemma, gpt-oss) look only at a recent window on some layers and need less: the engine's log shows the real figure. 8-bit and 4-bit include the small scale overhead of llama.cpp's q8_0 and q4_0.

The weights. Sam's 16 GB card holds a 12B model at Q4: about 7.7 GB. That part never changes.

A huge context. Sam sets 131,072 tokens. At about 0.125 MB per token (16-bit cache, a round number for illustration) the engine wants 131,072 × 0.125 MB = 16 GB of cache. 7.7 + 16 is more than the card: it refuses to load.

Right-size the context. 32,768 tokens is plenty for Sam's work. Cache: 32,768 × 0.125 MB = 4 GB. With about 1.5 GB of working space: 13.2 GB.

Shrink the cache. An 8-bit cache halves it to 2 GB. Total about 11.2 GB, leaving room for the system, with little loss of quality.

3. Ways to make it fit, cheapest first

  1. Use only the context you need. Engines reserve memory for the full context you set (Lesson 10.3).
  2. Quantise the KV cache to 8-bit (--cache-type-k q8_0 --cache-type-v q8_0 in llama.cpp; Ollama has a similar setting).
  3. A smaller quant of the weights (Q4_K_M instead of Q6_K; Lesson 8.5).
  4. Fewer parallel slots on a shared server.
  5. A smaller or MoE model whose active part fits (Lesson 2.5).
  6. Offload some layers to normal memory: it fits, but it is much slower.
  7. More memory: a bigger card, a second card (Lesson 15.3) or a unified-memory machine.

4. Hard limits

Some things memory cannot fix at home. Frontier-scale open models at full quality need hundreds of gigabytes to terabytes, and a million-token context on a large model needs far more memory than any home machine has. Those jobs belong in the cloud or on rented hardware (Lesson 15.7).

Maths and principle

Principle: fast memory needed = weights + KV cache + working space, and all of it must fit at once.

Equation: KV cache = 2 × layers × KV heads × head size × bytes per number × tokens (× users).

Analogy: a desk. The weights are the reference books that never leave; the KV cache is the pile of notes, growing with every page you read.

Try it: an 8B model with 0.125 MB per token, at 32,768 tokens, 16-bit cache? (32,768 × 0.125 MB = 4 GB, on top of the weights. At 8-bit, about 2 GB.)

KV=2×L×Hkv×d×b×T×U
2
one key and one value are stored for every token
L
layers in the model; each layer keeps its own notes
Hkv
key-value heads per layer (often fewer than the attention heads, which saves memory)
d
head size: how many numbers describe one token in one head (often 128)
b
bytes per number: 2 at 16-bit, 1 at 8-bit
T × U
tokens of context times parallel users
Physical meaning
Every token the model has read leaves a small, fixed-size note in every layer. Double the context and the notes double; serve four people and they quadruple. The weights are the book; the cache is your highlighting, and long books get a lot of it.

“If the model file fits in memory, the model fits.” The file is only the weights. The KV cache for your context and the engine’s working space come on top, and at long contexts the cache can be as big as the model itself. Leave room, or set a smaller context.

5. Worked example

Made-up machine. Sam’s 16 GB card holds a 12B model at Q4 (about 7.7 GB). With a 131,072-token context the engine refuses to load. Setting the context to 32,768 and an 8-bit KV cache brings the total under 13 GB, leaving room for the system. For the rare 100-page document, Sam uses document search instead of pasting it all in.

6. Practice

  1. Add up weights, cache and working space for your favourite model at 8,192 and 65,536 tokens.
  2. Which fix would you try first for a shared server that runs out of memory at busy times?
  3. Why is offloading layers a last resort?
Answers
  1. Use the two calculators above; add 1 to 2 GB of working space.
  2. Fewer parallel slots, or a smaller context per slot.
  3. Normal memory is many times slower, so speed falls sharply.

New words: working space, KV cache quantisation, offloading, out of memory.

  • Fast memory must hold the weights, the KV cache and working space all at once.
  • KV cache = 2 × layers × KV heads × head size × bytes × tokens × users: it grows with context and users.
  • Cheapest fixes first: a smaller context, an 8-bit cache, a smaller quant, fewer parallel slots.
  • Offloading layers to normal memory makes it fit but much slower.
  • Frontier-size models and million-token contexts are beyond any home machine.

Check yourself

1What three things share the memory when a model runs?
2A model fits with a 4,096-token context but crashes at 131,072. Why?
3Which change frees the most memory with the least loss of quality, for a long-context job?

Sources

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Prices and model names change fast: check the official source before you buy.