Home / Learn / Data & AI / Offline AI / Lesson 8.3
All lessons

Module 8: Quantisation: shrinking models to fit · Lesson 8.3 · about 20 min

Common formats: Q4_K_M, AWQ, MLX and FP4

Decode the quantisation names you see on model pages, from GGUF k-quants and i-quants to AWQ, GPTQ, MLX 4-bit and the new FP8 and FP4 formats.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 8 · about 20 minutes · read first: 8.1 and 8.2. You’ll be able to read any quant name and know whether it suits your machine.

Every quant name is a recipe for the same idea, small numbers per weight plus a scale per block; the name tells you roughly how many bits, which method, and which engine can read it.

1. Why this matters

Model pages list names like Q4_K_M, IQ3_XS, AWQ, GPTQ-Int4, MLX-4bit and FP8. They are different ways of shrinking the same model. Each engine reads only some of them, so reading the name saves a wasted download.

2. GGUF names (llama.cpp, Ollama, LM Studio)

  • Q8_0, Q6_K, Q5_K_M, Q4_K_M, Q3_K_M, Q2_K: the number is the rough bits; K means the “k-quant” method, which stores blocks of weights with their own scales; S, M, L (small, medium, large) say how many important layers keep extra bits.
  • IQ4_XS, IQ3_M, IQ2_XS: “i-quants”, which pack weights more cleverly. Better quality at very low bits, a little slower on some machines.
  • imatrix: an importance matrix made from sample text, so the quantiser keeps the most-used weights more precise. Good for 3 bits and below.

Watch it happen: decoding Q4_K_M

Q4. "Q" for quantised, "4" for about 4 bits per weight for most of the weights.

K. The k-quant method: weights go in small blocks and every block keeps its own scale, so a block of tiny weights and a block of big ones are both rounded finely.

The scale's cost. A block of 32 four-bit weights takes 32 × 4 = 128 bits, plus a 16-bit scale: 144 bits ÷ 32 weights = 4.5 bits per weight.

M. Medium: a few sensitive parts of the model keep 5 or 6 bits. That lifts the average to about 4.8 bits, which is why a 12B Q4_K_M file is about 12 × 4.8 ÷ 8 = 7.2 GB.

Who reads it. It is a GGUF name, so llama.cpp, Ollama and LM Studio. A vLLM server on an NVIDIA card would want AWQ or GPTQ instead.

3. Graphics card formats (vLLM and similar)

  • AWQ and GPTQ: 4-bit methods that use sample data to protect the important weights. Common for serving on NVIDIA cards (Module 15).
  • FP8: 8-bit floating point; supported by recent NVIDIA cards; near-original quality.
  • FP4 (MXFP4, NVFP4): 4-bit floating point with shared scales; recent NVIDIA cards run it natively. gpt-oss shipped its expert weights in MXFP4.

Floating point means each number stores a tiny exponent as well as its digits, like writing 3 × 10⁻² instead of 0.03. It handles very small and very large weights in the same block better than plain whole numbers do.

4. Mac formats

MLX 4-bit, 6-bit, 8-bit: quantised for MLX (Lesson 9.4), usually under the mlx-community name. LM Studio on a Mac offers them next to GGUF.

You see Rough bits Engine
Q4_K_M, Q5_K_M, IQ3_M 4.8, 5.7, 3.7 llama.cpp, Ollama, LM Studio
AWQ, GPTQ-Int4 about 4 vLLM and similar, NVIDIA
FP8, MXFP4, NVFP4 8, about 4 recent NVIDIA cards
MLX 4bit, 8bit about 4.5, 8.5 MLX, LM Studio on a Mac
Maths and principle

Principle: store small numbers per weight plus one scale per block of weights; the scale brings them back to roughly the right size.

Equation: bits per weight = bits per number + (bits of the scale ÷ weights per block). A block of 32 four-bit weights with a 16-bit scale: 4 + 16 ÷ 32 = 4.5 bits.

beff=b+sk

Analogy: a recipe written for 4 people with one line at the top saying “times 3”. Short numbers plus one multiplier.

Try it: 8-bit numbers in blocks of 32 with a 16-bit scale? (8 + 16 ÷ 32 = 8.5 bits, which is Q8_0.)

b
bits for each stored number (4 in a 4-bit format)
s
bits used by the block's scale, often 16
k
how many weights share one scale (the block size), often 32
beff
the real average bits per weight, the number that sets the file size
Physical meaning
The scale is a small overhead (half a bit per weight for blocks of 32) that buys a lot of accuracy, because each block is rounded on its own ruler instead of one ruler for the whole model.

“Q4 means the file is 4 GB” or “version 4 of the model.” The number is bits per weight. A Q4_K_M file of a 31B model is about 19 GB; of a 4B model, about 2.5 GB. The model version is in the model’s own name, not the quant name.

5. Worked example

Made-up numbers. Sofia wants Qwen3.8-27B on an RTX 3090 (24 GB). For Ollama she picks Q4_K_M GGUF. Later, to serve her team with vLLM, she uses an AWQ 4-bit copy of the same model. Same model, two formats, each for its engine.

6. Practice

  1. What do the parts of Q5_K_M mean?
  2. You need 3-bit to fit. Which GGUF option should you look for?
  3. Which formats can LM Studio on a Mac use?
Answers
  1. About 5 bits, k-quant method, medium size of extra-precision parts.
  2. An i-quant made with an imatrix, such as IQ3_M.
  3. MLX and GGUF.

New words: k-quant, i-quant, imatrix, AWQ, GPTQ, FP8, MXFP4, floating point.

  • In GGUF names the number is rough bits; K is the block method; S, M, L say how much is kept at higher bits.
  • i-quants with an imatrix are the best choice at 3 bits and below.
  • AWQ, GPTQ, FP8 and FP4 are for graphics-card servers such as vLLM; MLX formats are for Macs.
  • Real bits per weight = bits per number + scale bits ÷ block size (4 + 16 ÷ 32 = 4.5).

Check yourself

1In Q4_K_M, what does the 4 tell you?
2Which formats are mainly for NVIDIA graphics cards with Python serving engines?
3gpt-oss was released in MXFP4. What is that?

Sources

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Prices and model names change fast: check the official source before you buy.