Module 8: Quantisation: shrinking models to fit · Lesson 8.2 · about 20 min
Bits per weight and what you lose
How much quality a model loses at 8, 6, 4, 3 and 2 bits per weight, how it is measured, and why a bigger model squeezed small often beats a smaller one kept large.
Module 8 · about 20 minutes · read first: 8.1. You’ll be able to choose how far to shrink a model without losing what you need.
1. Why this matters
Fewer bits mean a smaller, faster model (Lesson 4.2). Too few, and the model starts to make mistakes it did not make before. Knowing where the line is lets you run bigger, better models on the same machine.
2. What you lose at each level
Rough guide for GGUF files of 7B to 70B models (smaller models lose more at each step):
| Bits per weight | Example | Size against 16-bit | Quality |
|---|---|---|---|
| 16 | F16, BF16 | 100% | the original |
| 8 | Q8_0 | about 53% | practically the same |
| 6 | Q6_K | about 41% | very hard to tell apart |
| about 4.8 | Q4_K_M | about 30% | small loss; the usual choice |
| about 3.9 | Q3_K_M | about 24% | noticeable on maths, code and other languages |
| about 2.6 to 3 | Q2_K | about 17% to 19% | clear loss; last resort for very large models |
Why do the levels matter so much? Each bit doubles the number of allowed values: 8 bits give 256 values, 4 bits give 16, 2 bits only 4. With 4 values, almost every weight is moved a long way.
Watch it happen: one weight at four bit levels
Take a block whose largest weight is 1.0 and follow the weight 0.30 as the bits drop. (Signed scheme: at b bits the whole numbers go from −(2^(b−1) − 1) to 2^(b−1) − 1.)
8 bits. Whole numbers −127 to 127, step 1 ÷ 127 = 0.0079. 0.30 ÷ 0.0079 = 38.1 → 38 → 0.299. Error 0.001: invisible.
6 bits. −31 to 31, step 0.032. 0.30 ÷ 0.032 = 9.3 → 9 → 0.290. Error 0.010: still tiny.
4 bits. −7 to 7, step 0.143. 0.30 ÷ 0.143 = 2.1 → 2 → 0.286. Error 0.014: small, and errors in other weights partly cancel out.
2 bits. Only −1, 0, 1 (step 1.0). 0.30 → 0 → 0. The weight is gone completely. Do that to millions of weights and the model starts to make real mistakes.
3. How quality loss is measured
- Perplexity: how surprised the model is by real text. A score of 10 means it is, on average, as unsure as if it were choosing between 10 equally likely next tokens. Lower is better; quantised copies score a little higher.
- KL divergence: how far the quantised model’s next-word choices drift from the original’s. Closer to zero is better.
- Your own tasks (Lesson 7.4): the measure that counts.
4. Bigger and smaller, or smaller and bigger?
At the same memory, a bigger model with fewer bits usually beats a smaller model with more bits, down to about 4 bits. Below about 3 bits, the bigger model’s advantage shrinks or reverses. Mixture-of-experts models and very large models tolerate low bits better than small dense ones.
model weights conversation memory (KV cache) working room · line = what this device can give the model
Maths and principle
Principle: each weight is rounded to one of 2^bits levels; fewer levels mean more rounding error, which adds up over many layers.
Equation: levels = 2^bits. 8 bits: 256 levels; 4 bits: 16; 2 bits: 4.
Analogy: describing paint colours with 256 names against 16 against 4. With 16 you can still paint a good picture; with 4 the sunset looks wrong.
Try it: how many levels does 3 bits give? (2^3 = 8.)
- b
- bits per weight: about 8.5 for Q8_0, 6.6 for Q6_K, 4.8 for Q4_K_M, 3.9 for Q3_K_M
- 2b
- how many different values each weight can take
- Pmax
- the largest model, in billions of parameters, whose weights fit
- reserve
- memory kept free for the conversation (KV cache, Lesson 8.4) and the system
- Physical meaning
- Fewer bits buy you a bigger model in the same memory, and a bigger model knows more. The trade pays off until the rounding gets so coarse, below about 3 bits, that the bigger model's extra knowledge is lost in the noise.
“More bits always means a better answer, so use the biggest file I can.” On the same memory, a model twice the size at 4 bits usually beats a smaller one at 8 bits. Spend memory on parameters first, bits second, and stop going lower at about 4 bits.
5. Worked example
Made-up numbers. Chen has 24 GB. Options: a 12B at Q8_0 (about 13 GB) or a 27B at Q4_K_M (about 17 GB). On his 20 test tasks the 27B scores 17 and the 12B scores 14, but the 12B writes faster. He keeps both: the 27B for hard jobs, the 12B for quick ones.
6. Practice
- In the simulation, find the largest model that fits your machine at 4 bits, and at 8 bits.
- Why are small models hurt more by low bits?
- Which bits would you choose for a model that writes code for you?
Answers
- Usually about twice the parameters at 4 bits.
- They have fewer weights to spread the error over, so each rounding matters more.
- 5 to 6 bits if it fits (Q5_K_M or Q6_K); code is exact work.
New words: bits per weight, perplexity, KL divergence, rounding error.
- Each bit removed halves the allowed values: 256 at 8 bits, 16 at 4, 4 at 2.
- 8 and 6 bits are almost lossless; about 4.8 (Q4_K_M) is a small loss; 3 bits and below hurt exact work.
- Perplexity and KL divergence measure the loss; your own tasks decide.
- On the same memory, a bigger model at 4 bits usually beats a smaller one at 8.
- Largest model that fits ≈ (memory − reserve) × 8 ÷ bits per weight.
Check yourself
Sources
- llama.cpp quantisation tool and its quality measurements (perplexity and KL divergence), project documentation (official page), https://github.com/ggml-org/llama.cpp/tree/master/tools/quantize
- Dettmers, T. and Zettlemoyer, L. (2023), The case for 4-bit precision: k-bit Inference Scaling Laws, ICML 2023 https://arxiv.org/abs/2212.09720
- Jurafsky, D. and Martin, J. H., Speech and Language Processing, 3rd edition draft, chapter on n-gram language models (perplexity) https://web.stanford.edu/~jurafsky/slp3/
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Prices and model names change fast: check the official source before you buy.