Module 4: The parts that matter for AI · Lesson 4.2 · about 20 min
Memory bandwidth: the real speed limit
Why writing speed depends on how fast memory can be read, not on how fast the chip is, and how to estimate tokens per second for any machine.
Module 4 · about 20 minutes · read first: 4.1 and 3.5. You’ll be able to guess a machine’s writing speed before you buy it.
1. Why this matters
Lesson 4.1 asked “does it fit?”. This lesson asks “how fast will it write?”. The answer is mostly one number on the spec sheet, memory bandwidth, which shops rarely print in big letters. Knowing it saves you from paying for a fast chip that sits waiting.
2. One token, one full read
To write each new token, the chip combines the conversation with every weight in the model (Lesson 2.1). The weights live in memory. So for every token, all of them travel from memory to the chip. A 7.7 GB model means 7.7 GB read per token.
The chip itself could do the maths much faster. It spends most of its time waiting for the next batch of weights to arrive. That is why we say writing is memory-bound: memory is the bottleneck.
Picture a cook who needs every jar in the pantry for every single dish. A faster cook does not help if the pantry door is narrow. A wider door does. Bandwidth is the width of the door.
In symbols, with the bandwidth in GB/s and the GB read for each token:
- B (bandwidth)
- How many GB the memory can hand to the chip each second. It is on the spec sheet as GB/s.
- M
- GB read for each token: the whole model for a normal (dense) model, only the active part for a Mixture-of-Experts model.
- Memory-bound
- The chip spends its time waiting for memory, not doing sums.
- Physical meaning
- Every word the AI writes is a full trip through its memory. A machine with twice the bandwidth makes that trip twice as fast and writes about twice as many words per second, whatever its chip is called.
3. Reading the speed off a spec sheet
Memory bandwidth is written in GB/s (gigabytes per second):
- Laptop memory (two DDR5 sticks): about 60 to 100 GB/s.
- Base Mac mini class: about 120 GB/s; Pro and Max chips: 270 to over 600 GB/s.
- 128 GB AI boxes (DGX Spark, Ryzen AI Max class): about 256 to 273 GB/s.
- Graphics cards: RTX 5060 Ti 448 GB/s, used RTX 3090 936 GB/s, RTX 5090 1,792 GB/s.
A Mixture-of-Experts model (Lesson 2.5) reads only its active experts for each token, so is much smaller than the file. That is why big MoE models can write quickly even on medium bandwidth.
“A faster processor makes the AI write faster.” While writing, the chip mostly waits for memory. A higher clock speed or more “AI TOPS” with the same memory changes little. Look for the GB/s number instead.
4. Reading time versus writing time
Reading your prompt (prefill, Lesson 3.5) is different. The chip handles many prompt tokens together, so each weight it reads is used many times. Prefill is compute-bound: here the chip’s raw power matters, and graphics cards beat Macs and AI boxes by a wide margin. That is why a long document can take a while to “start” on a Mac even when the writing is quick.
Maths and principle
Principle: writing speed is capped by how many times per second memory can hand over every active weight.
Equation: top tokens per second ≈ bandwidth (GB/s) ÷ bytes read per token (GB). Real engines reach about 50 to 80% of it.
Analogy: a cook who needs every ingredient in the pantry for each dish. A faster cook does not help if the pantry door is narrow; a wider door does.
Try it: a 16 GB model on a 1,792 GB/s card? (1,792 ÷ 16 = 112 tokens per second ceiling, about 56 to 90 in practice.)
5. Worked example
Made-up numbers. Arjun compares a 128 GB AI box (273 GB/s) with a Mac Studio (about 614 GB/s) for a 27B model at 4-bit (about 16 GB). The box: 273 ÷ 16 ≈ 17 tokens per second ceiling, so about 9 to 14 in practice. The Mac: 614 ÷ 16 ≈ 38, so about 19 to 31. For a mixture-of-experts model that reads only 3 GB per token, both feel quick, and the box’s lower price wins for him.
6. Practice
- Your laptop has 80 GB/s. How fast could Gemma 4 E2B (about 1.6 GB) write at best?
- Why does a long pasted report feel slower to start on a Mac than on a graphics card?
- Two cards both have 16 GB: one 448 GB/s, one 896 GB/s. Which models can each run, and how do they differ?
Answers
- 80 ÷ 1.6 = 50 tokens per second ceiling, about 25 to 40 in practice.
- Reading the prompt is compute-bound, and graphics cards have much more raw compute.
- The same models (same memory), but the 896 GB/s card writes about twice as fast.
New words: bandwidth (GB moved per second), memory-bound (waiting on memory), compute-bound (waiting on the maths).
- Every new token needs every active weight read from memory once.
- Speed ceiling ≈ bandwidth (GB/s) ÷ GB read per token; real engines reach about 50 to 80% of it.
- Writing is memory-bound, so faster memory helps more than a faster chip.
- Reading a long prompt is compute-bound, which is where graphics cards pull far ahead.
- Mixture-of-Experts models read only their active part, so they write fast for their size.
Check yourself
Sources
- Graphics card memory speeds: NVIDIA product pages via Compute Market, prices from September 2026, updated 1 November 2026, https://compute-market.com
- DGX Spark memory and bandwidth: LAVX News, 13 June 2026, https://news.lavx.hu/article/amd-prices-ryzen-ai-halo-at-3-999-as-dgx-spark-moves-to-4-699
- Model download sizes: Ollama library, read 16 November 2026 via BenchLM, https://benchlm.ai/best/ollama-models
- Williams, S., Waterman, A. and Patterson, D. (2009), Roofline: an insightful visual performance model for multicore architectures, Communications of the ACM 52(4), 65-76 (memory-bound and compute-bound work) https://doi.org/10.1145/1498765.1498785
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Prices and model names change fast: check the official source before you buy.