Module 3: Tokens: the unit of everything · Lesson 3.5 · about 20 min
Tokens per second: reading speed and writing speed
Local models have two speeds, a fast reading speed (prefill) and a slower writing speed (decode). What sets each, what is a good number, and how to measure yours.
Module 3 · about 20 minutes · read first: 3.4. You’ll be able to read the speed numbers your engine shows and know which part of your hardware sets each one.
1. Why this matters
“How fast is it?” has two answers. A model may read a long document in seconds but write slowly, or the other way round. Knowing which speed limits your job tells you whether a bigger graphics card, more memory or a smaller model will help.
2. Two phases
- Prefill (reading): the model processes all your input tokens together. Because the work is done in parallel, it is fast, often hundreds to thousands of tokens per second on a graphics card. It is limited by compute, how much maths the chip does per second.
- Decode (writing): it writes one token at a time, and each needs a pass through the model’s weights. It is limited by memory bandwidth, how fast those weights can be read.
You see prefill as the pause before the first word (“time to first token”), and decode as the speed of the words after it.
Picture a translator with a long letter. Reading it through takes a few seconds, because the eyes take in many words at once. Writing the translation is slower: one word at a time, and before each word they flip through their dictionary.
You press Enter. Chen's 12-page document is 8,000 tokens of input.
Prefill. The laptop reads all 8,000 together at 500 tokens per second: 8,000 ÷ 500 = 16 seconds of waiting before anything appears.
First token. After 16 seconds the first word shows. That pause is the time to first token.
Decode. Now words appear at 12 tokens per second. A 300-token summary takes 300 ÷ 12 = 25 seconds.
Total: 41 seconds. To cut the 16-second wait, more compute (a graphics card) helps. To speed up the writing, a smaller or more tightly squeezed model helps.
3. What good looks like
- Reading along: 10 tokens per second or more feels smooth; people read about 4 to 8.
- Batch jobs: every extra token per second shortens the run.
- A real example: one test measured gpt-oss-120b writing at about 88 tokens per second on a 128 GB M5 Max Mac (4K context, MLX). It is a Mixture-of-Experts model, so only part of it is read per token (Lesson 2.5).
Speeds fall as the context grows, because there is more to look back over (often by a quarter or more at long lengths).
“A faster graphics chip always means faster writing.” Writing is usually held back by memory speed, not by the chip’s maths. A chip twice as powerful with the same memory bandwidth writes at about the same speed; it only reads your input faster. When you compare machines for chat speed, compare memory bandwidth (GB/s) first.
Maths and principle
Principle: decode reads every active weight once per token, so its ceiling is bandwidth divided by bytes read.
Equation: decode speed ≤ memory bandwidth (GB/s) ÷ active weights (GB).
Analogy: a reader who must flip through the whole reference book before writing each word. A thinner book (smaller model) or faster hands (more bandwidth) means more words per minute.
Try it: a graphics card at 448 GB/s and a 7.7 GB model? (448 ÷ 7.7 ≈ 58 tokens per second at most; real speeds are often 60 to 80% of the ceiling.)
- prefill
- reading all the input tokens together; limited by compute (maths per second)
- decode
- writing one token at a time; limited by memory bandwidth
- bandwidth (GB/s)
- how many gigabytes the memory can deliver each second
- active weights (GB)
- how much of the model must be read for each token (all of it for a dense model)
- Physical meaning
- Every written token needs one trip through the whole model in memory. Divide how fast memory can be read by how much must be read, and you have the most tokens per second possible. Real engines reach about 60 to 80% of it.
4. Worked example
Made-up numbers. Chen’s 12-page document (8,000 tokens) takes 16 seconds before the first word appears: prefill at 500 tokens per second on his laptop. The summary then writes at 12 tokens per second. To speed up the start, a graphics card would help most; to speed up the writing, a smaller or more tightly quantised model would.
5. Practice
- In Ollama, run
ollama run gemma4:e4b --verboseand ask a question. Find “prompt eval rate” (prefill) and “eval rate” (decode). - A Mac with 170 GB/s and a 16 GB model. What is the decode ceiling?
- Your job reads 50 long documents and writes one line each. Which speed matters more?
Answers
- The prompt eval rate is usually many times the eval rate.
- About 10 tokens per second.
- Prefill (reading), because almost all the tokens are input.
New words: prefill (reading the input), decode (writing the output), time to first token (the pause before the answer starts), compute-bound and memory-bound.
- Prefill reads all input at once and is limited by compute; you feel it as the time to first token.
- Decode writes one token at a time and is limited by memory bandwidth.
- Decode ceiling ≈ bandwidth (GB/s) ÷ active weights (GB); real speeds are about 60 to 80% of it.
- About 10 tokens per second or more feels smooth to read along with.
- Speeds fall as the context grows; in Ollama, —verbose shows both rates.
Check yourself
Sources
- llama.cpp, official repository (llama-bench reports prompt processing and generation speeds separately) https://github.com/ggml-org/llama.cpp
- Ollama API documentation (prompt_eval_duration and eval_duration) https://github.com/ollama/ollama/blob/main/docs/api.md
- Kwon, W. et al. (2023), Efficient Memory Management for Large Language Model Serving with PagedAttention (prefill and decode phases) https://arxiv.org/abs/2309.06180
- Mac speeds measured with MLX (M5 Max, 128 GB, 4K context): ComputingForGeeks, checked 27 October 2026, https://computingforgeeks.com/best-mac-for-local-llm/
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Prices and model names change fast: check the official source before you buy.