Home / Learn / Data & AI / Offline AI / Lesson 4.1
All lessons

Module 4: The parts that matter for AI · Lesson 4.1 · about 25 min

Memory is king: the model must fit

The single most important number for local AI is how much fast memory your machine can give the model. Work out what fits on yours, with a simulation.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 4 · about 25 minutes · read first: 3.3. You’ll be able to work out which models fit on any machine.

A model only runs fast when all of it fits in fast memory, so the first question about any machine is "how much fast memory can the model have?"

1. Why this matters

Most buying mistakes, most “which model?” questions and most speed problems come back to one question: does the model fit in fast memory? Get this right and you will never buy the wrong machine, or download a model your computer cannot run.

2. Two kinds of memory

Think of a kitchen. The worktop next to the cooker is small, but everything on it is in reach. The pantry down the hall holds more, but every trip costs time. Computers have both.

  • Graphics memory (VRAM) sits on a separate graphics card. It is very fast, but usually only 8 to 32 GB. This is the worktop.
  • Main memory (RAM) is larger but much slower for AI on most PCs. This is the pantry.
  • Unified memory (Apple Silicon Macs, and AI boxes such as DGX Spark or Ryzen AI Max machines) is one large pool that the graphics side can use at good speed, up to 128 GB or more. A big worktop.

The model runs fastest when all of it, plus the conversation’s memory, sits in the fast kind.

3. What a model needs

A model needs room for three things:

  1. Weights. These are the model’s learned numbers. Billions of parameters × bits per weight ÷ 8 = GB (8 bits make one byte). A 12B model at 4 bits is about 6 GB of weights. Real files are a little bigger (7.7 GB for Gemma 4 12B) because some parts are kept at higher precision.
  2. Conversation memory (KV cache). The model keeps notes on every token in the chat so far. It grows with the context length (Lesson 8.4).
  3. Working room for the engine, about 1 GB.

Here is a 16 GB graphics card filling up, one part at a time:

An empty card. The card has 16 GB. The screen and system take about 1 GB, so about 15 GB is left for the model.
Load the weights. Gemma 4 12B at 4-bit is a 7.7 GB file. It goes on the card first. 8.7 GB used.
Add the conversation. For a chat of about 8,000 tokens, the KV cache takes roughly 1.5 GB. 10.2 GB used.
Add working room. The engine needs about 1 GB for its sums. Total 11.2 GB: it fits, with almost 5 GB to spare.
Try a 27B model. At 4-bit its weights are about 27 × 4.8 ÷ 8 ≈ 16.2 GB. The total is now about 19.7 GB. Nearly 4 GB spills over into slow main memory, and writing slows down a lot.

The sum in one line, where P is the number of parameters in billions and b is the bits per weight:

GB needed≈P×b8+KV cache+1

Now try combinations of your own here:

Try it: will it fit?

model weights conversation memory (KV cache) working room · line = what this device can give the model

Rough rules: weights = billions of parameters × bits ÷ 8; conversation memory ≈ 0.01 GB per billion parameters per 1,000 tokens; 1 GB working room; top speed ≈ memory bandwidth ÷ weights read per token. Real numbers vary by model and engine by 20% or more. Usable memory leaves room for the system (Macs about three quarters). Device speeds are makers' figures and the sources in this lesson.
P
The number of parameters (weights) in the model, in billions. "12B" means P = 12.
b
Bits per weight: how precisely each number is stored. 16 is full size; 4 to 5 is the usual shrunk size (Module 8).
÷ 8
Turns bits into bytes, so billions of bytes come out as GB.
KV cache
The model's notes on the conversation so far. Longer chats need more.
Physical meaning
Every one of those billions of numbers must sit in fast memory, ready to be read for every word the model writes. If they do not all fit, some live in slow memory and every word waits for them.

4. The headline rules

  • 8 GB laptop: small models only (Gemma 4 E2B class).
  • 16 GB laptop: up to about 12B at 4-bit.
  • 16 GB graphics card: up to about 14B fully on the card, fast.
  • 24 to 32 GB graphics card, or 32 GB Mac: 27B to 31B models at 4-bit.
  • 96 to 128 GB unified memory: large Mixture-of-Experts models such as gpt-oss-120b (65 GB download).

“If the model runs, it fits.” Not quite. Most engines will still start a model that is too big: they quietly put the part that does not fit in main memory. It works, but it can be several times slower (Lesson 4.3). Check that the whole model, plus the conversation, sits in fast memory.

Maths and principle

Principle: fit first, then speed. A model that does not fit in fast memory is slow however fast the chip.

Equation: needed ≈ (parameters × bits ÷ 8) + KV cache + about 1 GB ≤ usable fast memory.

Analogy: a desk and a library. Work goes quickly when every book you need is on the desk. A bigger desk lets you take on bigger projects; a faster reader only helps once the books are there.

Try it: a 27B model at 4.8 bits with 1.5 GB of KV cache? (27 × 4.8 ÷ 8 ≈ 16.2 GB + 1.5 + 1 ≈ 18.7 GB: a 24 GB card, yes; a 16 GB card, no.)

5. Worked example

Made-up numbers. Divya plans to buy either a laptop with 16 GB or one with 32 GB, 200 dollars apart. In the simulation, the 16 GB laptop runs Gemma 4 12B at 4-bit with 8K context, but the 27B is red. She wants better document answers, so the 32 GB model is worth it to her.

6. Practice

  1. Set the simulation to your own machine. What is the largest model that fits with 8,000 tokens of context?
  2. Change only the context to 128,000 tokens. What happens?
  3. On a 16 GB graphics card, which is faster: a 12B at 4-bit or a 27B at 3-bit that only just fits? Why?
Answers
  1. Note it down: this is your model size class.
  2. The conversation memory grows a lot, and models that fitted may no longer.
  3. Usually the 12B: fewer bytes to read per token, and room to spare. The 27B may give better answers but writes more slowly, and 3-bit loses some quality (Lesson 8.2).

New words: VRAM (graphics card memory), RAM (main memory), unified memory, usable memory (what is left for the model after the system’s share).

  • A model needs room for its weights, its conversation memory (KV cache) and about 1 GB of working room, all in fast memory.
  • Weights in GB ≈ billions of parameters × bits per weight ÷ 8. A 12B model at 4 bits is about 6 GB.
  • Graphics memory is fast but small; unified memory on Macs and AI boxes is large and fairly fast; ordinary main memory is slow for AI.
  • If a model does not fit, it still runs, but the part in slow memory sets the pace.

Check yourself

1A model needs 14 GB. Your PC has 32 GB of main memory and an 8 GB graphics card. Where can it run fast?
2Why can a 32 GB Mac run bigger models than a PC with an 8 GB graphics card?
3What is the weight memory of an 8-billion-parameter model at 4 bits per weight?

Sources

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Prices and model names change fast: check the official source before you buy.