Home / Learn / Data & AI / Offline AI / Lesson 8.5
All lessons

Module 8: Quantisation: shrinking models to fit · Lesson 8.5 · about 20 min

Choosing the right size for your machine

A step-by-step way to pick the model size, quantisation and context length that give the best answers your machine can run, using the simulation and a quick test.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 8 · about 20 minutes · read first: 4.1 and 8.1. You’ll be able to choose a model size, quantisation and context length for your machine in a few minutes.

Your best model is the biggest one whose weights, plus the memory for your context, fit in your fast memory with a little room to spare, at a speed you can live with.

1. Why this matters

Every machine has a best combination: the most capable model that still fits, at a speed you can live with, with enough context for your work. Finding it once saves a lot of trial and error.

It is like packing a suitcase for a flight: you want the most useful things inside, under the weight limit, and you still need to be able to close the lid.

2. Five steps

  1. Usable memory: your graphics card’s memory, or about three quarters of a Mac’s, or your laptop’s memory minus about 4 to 5 GB for the system.
  2. Context you need: chat and short emails, 8K; long documents, 16K to 32K; whole reports or codebases, more.
  3. Start at 4 bits (Q4_K_M or MLX 4-bit) and find the biggest model that fits with that context and 10 to 20% to spare. Use the simulation below.
  4. Check speed: at least about 10 tokens per second for chat; for overnight jobs, slower is fine.
  5. Test on your own tasks: compare with one size smaller at higher precision (for example 12B at 4 bits against 31B at 3 bits, or 12B at 6 bits). Keep the winner.

Headroom is memory you leave free on purpose: the chat grows, and your browser and system need space too.

Try it: will it fit?

model weights conversation memory (KV cache) working room · line = what this device can give the model

Rough rules: weights = billions of parameters × bits ÷ 8; conversation memory ≈ 0.01 GB per billion parameters per 1,000 tokens; 1 GB working room; top speed ≈ memory bandwidth ÷ weights read per token. Real numbers vary by model and engine by 20% or more. Usable memory leaves room for the system (Macs about three quarters). Device speeds are makers' figures and the sources in this lesson.

Watch it happen: Sana’s 16 GB laptop

Usable memory. 16 GB minus about 5 GB for Windows and apps: 11 GB usable.

Context. Sana summarises reports, so 16K tokens. For a 12B model that costs about 1.9 GB of KV cache.

Try 12B at 4.8 bits. Weights 12 × 4.8 ÷ 8 = 7.2 GB. Add cache and engine: 7.2 + 1.9 + 1 = 10.1 GB. Headroom 0.9 GB: tight, but it fits.

Try 31B at 4.8 bits. Weights 31 × 4.8 ÷ 8 = 18.6 GB. Far over 11 GB. Out.

Check speed and test. The 12B writes about 9 tokens per second: fine for a summary script. For quick questions she keeps E4B too, which is twice as fast.

3. Quick picks (November 2026)

Usable fast memory First choice Also try
about 4 GB Gemma 4 E2B, 4-bit —
about 11 GB Gemma 4 12B, 4-bit Gemma 4 E4B for speed
about 15 GB (16 GB card) Gemma 4 12B at 6 or 8 bits Gemma 4 26B A4B, 4-bit, short context
about 23 to 31 GB Gemma 4 31B or Qwen3.8-27B, 4-bit 6-bit with a short context
about 96 GB or more gpt-oss-120b several smaller models side by side
Maths and principle

Principle: choose the point that maximises quality under two limits: memory and speed.

Equation: maximise quality(size, bits) subject to memory(size, bits, context) ≤ usable memory and speed ≥ your minimum.

headroom=U−P×b8−k×C−1

Analogy: choosing a suitcase for a flight: the most useful things you can take, under the weight limit and still able to carry it.

Try it: usable memory 11 GB, need 16K context (about 1.9 GB of KV cache for a 12B model). Does a 12B at 4.8 bits fit? (7.2 + 1.9 + 1 ≈ 10.1 GB: yes, only just.)

U
usable fast memory in GB (card memory, three quarters of a Mac, or laptop memory minus the system)
P × b ÷ 8
the weights: billions of parameters × bits per weight ÷ 8
k × C
the KV cache: GB per 1,000 tokens × thousands of tokens of context
1
about 1 GB the engine needs for its own work
Physical meaning
Headroom is the gap between what you pack and the suitcase's limit. Aim for 10% to 20% of your memory: less and the machine starts swapping when a chat grows; much more and you could have run a bigger model.

“If the file is smaller than my memory, it will run fine.” The file is only the weights. The context’s cache, the engine and your other apps need memory too, and a model that “just fits” can still run painfully slowly. Add them up and keep 10% to 20% free.

4. Worked example

Made-up numbers. Sana has a 16 GB laptop and wants document summaries with 16K of context. Usable memory is about 11 GB. The simulation says Gemma 4 12B at 4-bit fits with 16K. A test on five real documents shows good summaries at 9 tokens per second, fine for a script that runs while she does something else. For quick questions she keeps Gemma 4 E4B, which is twice as fast.

5. Practice

  1. Work through the five steps for your own machine and write down your pick.
  2. Test your pick and one alternative on three real tasks. Which wins?
  3. Your needs change to 64K context. What has to give?
Answers
  1. Keep the note; it is your default model.
  2. Count wins, and note speed; a slightly worse but twice-as-fast model can be the better tool.
  3. The conversation memory grows a lot, so pick a smaller model, fewer bits, or more memory.

New words: usable memory, headroom (memory left free on purpose), default model (the one you reach for first).

  • Usable memory: the card’s memory, three quarters of a Mac, or laptop memory minus 4 to 5 GB.
  • Start at 4 bits and pick the biggest model that fits with your context and 10% to 20% headroom.
  • Headroom = usable − weights − KV cache − about 1 GB for the engine.
  • Aim for at least 10 tokens per second for chat, then test your pick against one alternative on real tasks.

Check yourself

1You can just fit a 27B model at 3 bits, or a 12B at 6 bits. What should decide?
2Why leave 10 to 20 percent of memory free?
3Your model fits but writes at 3 tokens per second. What is the first thing to try?

Sources

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Prices and model names change fast: check the official source before you buy.