Module 2: How a language model works · Lesson 2.5 · about 20 min
Dense models and Mixture-of-Experts
Dense models use every parameter for every token; Mixture-of-Experts models use only a few experts. Why that makes big MoE models fast, and why they still need lots of memory.
Module 2 · about 20 minutes · read first: 2.2. You’ll be able to tell dense and MoE models apart and pick the right hardware for each.
1. Why this matters
Many of the best open models are now Mixture-of-Experts (MoE). They break the simple rule “bigger means slower”, and they change which hardware is the better buy.
2. Dense: everyone works on everything
In a dense model, every parameter is used for every token. A 31B dense model reads all 31B for each token. Simple and predictable: memory and speed both follow its size. It is like a small office where everyone reads every letter.
3. MoE: a team of specialists
An MoE model splits the feed-forward part of each layer (Lesson 2.4) into many experts, plus a small router. For each token, the router picks a few experts (often 2 to 8 out of dozens or more) and only those run. It is like a hospital reception desk that sends each patient to two of its many specialists.
A token arrives at one layer: say the token " 7" in "what is 3 + 4? It is 7".
The router scores the experts. This layer has 8 experts. The router gives each a score; experts 3 and 6 score highest.
Pick the top 2. Their scores become shares: expert 3 gets 0.6, expert 6 gets 0.4. The other six do nothing for this token.
Blend. The two experts each work on the token, and their results are mixed 60 : 40. Only 2 of 8 experts' numbers were read from memory.
Next token, new choice. The next token may go to experts 1 and 5. Any expert could be picked, so all 8 must stay in memory, even though each token reads only 2.
So two numbers matter:
- Total parameters (all experts): sets memory. Gemma 4 26B A4B needs memory for 26B.
- Active parameters (experts used per token): sets speed. It writes like a 4B model.
4. What it means for buying
- Small fast graphics card (8 to 16 GB): dense models of the right size are the better fit; big MoE models do not fit.
- Large unified memory (64 to 128 GB Macs, AI boxes): big MoE models shine: plenty of room, and only a little has to be read per token. gpt-oss-120b (117B total, 5.1B active) is the classic example.
“A 26B A4B model only needs the memory of a 4B model.” No: it needs memory for all 26B, because the router may pick any expert for the next token. What the 4B tells you is the speed: only about 4B are read per token. Big memory, small reading: that is the MoE deal.
Maths and principle
Principle: memory scales with total parameters; time per token scales with active parameters.
Equation: memory ≈ total × bits ÷ 8; speed ceiling ≈ bandwidth ÷ (active × bits ÷ 8).
Analogy: a hospital. The building must hold every specialist, but each patient sees two, so patients flow quickly.
Try it: 117B total, 5.1B active, 4 bits, 273 GB/s? (Memory ≈ 58.5 GB; speed ceiling ≈ 273 ÷ 2.55 ≈ 107 tokens per second.)
- total
- all parameters, every expert included, in billions
- active
- the parameters actually used for one token: the shared parts plus the chosen experts
- bits ÷ 8
- bytes per parameter (0.5 at 4 bits)
- bandwidth
- how many GB per second the memory can deliver
- Physical meaning
- The building must hold every specialist (memory follows total), but each patient sees only two (speed follows active). So a big MoE model on a roomy but not super-fast machine can write as quickly as a small dense model.
5. Worked example
Made-up numbers. Two machines, same price: one with a 24 GB graphics card at 936 GB/s, one with 128 GB unified memory at 273 GB/s. A 31B dense model at 4 bits (16 GB) runs about three times faster on the graphics card. gpt-oss-120b (about 65 GB) does not fit on the card at all, but runs smoothly on the 128 GB machine.
6. Practice
- What are the memory and speed classes of “60B A6B”?
- Which machine above would you pick for a coding helper using a 27B dense model? For gpt-oss-120b?
- Why don’t all models use MoE?
Answers
- Memory like a 60B model (about 30 to 36 GB at 4 bits); speed like a 6B model.
- The graphics card for the 27B; the 128 GB machine for gpt-oss-120b.
- MoE needs more memory for the same ability, is harder to train well, and is less efficient on small machines.
New words: dense model, Mixture-of-Experts (MoE), expert, router, total and active parameters.
- A dense model uses every parameter for every token; memory and speed both follow its size.
- An MoE model has many experts and a router that picks a few for each token.
- Memory ≈ total × bits ÷ 8, because any expert may be needed next.
- Speed ceiling ≈ bandwidth ÷ (active × bits ÷ 8), because only the chosen experts are read.
- Big MoE models suit machines with lots of memory; dense models suit small, fast graphics cards.
Check yourself
Sources
- Shazeer, N. et al. (2017), Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer https://arxiv.org/abs/1701.06538
- Fedus, W., Zoph, B. and Shazeer, N. (2021), Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity https://arxiv.org/abs/2101.03961
- Jiang, A. Q. et al. (2024), Mixtral of Experts https://arxiv.org/abs/2401.04088
- gpt-oss-120b (117B total, 5.1B active) and other MoE sizes: Morph, November 2026, https://www.morphllm.com/best-open-source-llm
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Prices and model names change fast: check the official source before you buy.