Module 10: From words to an LLM · Lesson 10.5 · about 25 min
The transformer block
A transformer is the same block stacked many times: attention, add and normalise, a small feed-forward network, add and normalise. Step through one from words to next-word probabilities and count its parameters.
Module 10 · about 25 minutes · read first: 9.3, 10.4. You’ll be able to name the parts of a transformer block in order, say what each one does to the numbers, and estimate a model’s size from its settings.
1. Why this matters
Every well-known LLM (GPT, Gemini, Claude, Llama, Gemma, Mistral) is a stack of transformer blocks. When you read that a model has “32 layers, model width 4,096 and 8 billion parameters”, this lesson tells you what those numbers mean. From them you can guess how much memory it needs and how fast it can run (the Offline AI course uses exactly this).
2. One block
Every token enters as a list of numbers: its embedding plus a signal for its position. is called the model width. One block does four things:
- Attention (Lesson 10.4): each token mixes in information from the earlier tokens it needs, using several heads.
- Add and normalise: add the block’s input back on (a residual connection), then rescale each token’s list to average 0 and spread 1 (layer normalisation).
- Feed-forward network: a small two-layer network (Lesson 9.3) that works on each token on its own: widen to about numbers, bend, shrink back to .
- Add and normalise again.
- Tokenstext → 5 token ids
- Embedding + position5 × 512 numbers
- Attentionstill 5 × 512
- Add and normalisestill 5 × 512
- Feed-forward5 × 2,048 → 5 × 512
- Add and normalisestill 5 × 512
- × N blocksstill 5 × 512
- Next-word scores512 → 50,000 scores
- Softmax50,000 probabilities
Attention moves information between tokens. The feed-forward part works on each token’s information alone. Stack the block times, each copy with its own weights. Finally, score every vocabulary word from the last token’s numbers, and softmax the scores into next-word probabilities.
Follow one token, “sat”, through a block. Real models use thousands of numbers per token; here we use 4.
In. "sat" arrives as 4 numbers: its embedding plus a position signal saying it is the third word.
Listen (attention). "sat" looks back at "the" and "cat" and mixes in mostly "cat": who did the sitting. Out comes a change to add.
Add and normalise. Input plus change gives, say, (1, 2, 3, 6). Mean 3, standard deviation 1.871, so it becomes (−1.07, −0.53, 0, 1.60): same pattern, steady size.
Think alone (feed-forward). Widen the 4 numbers to 16, set the negative ones to 0 (the bend, ReLU), and shrink back to 4. This is where much of the model's stored knowledge acts.
Add and normalise again. Out come 4 numbers, the same shape as went in, ready for the next block.
After the last block. The final token's numbers give one score per vocabulary word, and softmax turns the scores into next-word probabilities.
3. Why the extra parts
- Position: attention is a weighted average, and averages ignore order. Adding a position signal to each embedding (or rotating queries and keys by position, as most recent models do) lets “dog bites man” differ from “man bites dog”.
- Residuals: as with very deep networks (Lesson 9.2), gradients fade as they pass back through many layers. gives them a straight road back, so 100-block models still train.
- Normalisation: keeps the numbers in a steady range from block to block, so the learning rate that suits one block suits all.
Maths and principle
Principle: alternate “talk to other tokens” (attention) and “think on your own” (feed-forward), with a shortcut round each.
One block, for the token vectors :
with . (Many models normalise before each part instead of after; the idea is the same.)
Layer normalisation of one token’s vector, with its own mean and standard deviation : , then a learned scale and shift.
Size: attention has 4 matrices of (queries, keys, values, output); the feed-forward part has and . So one block holds about
Analogy: a meeting run in rounds. In each round everyone first listens to whoever is relevant (attention), then thinks alone for a moment (feed-forward), keeping their notes from before (residual).
Try it: a model with and 32 blocks. Weights in the blocks? ( million per block; × 32 ≈ 6.4 billion, before the embedding table.)
- d
- model width: how many numbers describe each token inside the model
- N
- how many blocks are stacked (the "layers" in a model's description)
- 12d²
- weights in one block: 4d² in attention plus 8d² in the feed-forward network
- V × d
- weights in the embedding table, one row of d numbers per vocabulary token
- Physical meaning
- The parameter count is the model's size on disk and in memory. At 2 bytes per weight, an 8-billion-parameter model needs about 16 GB just to load, which is why width and depth decide whether a model fits on your laptop.
4. Worked example
A small model like the one in the simulation: , 12 blocks, vocabulary 50,000.
- One block: weights, about 3.1 million.
- 12 blocks: about 37.7 million.
- Embedding table: million (often shared with the final scoring layer).
- Total: about 63 million parameters, about half the size of the first GPT model (2018).
Layer normalisation of one token’s vector (1, 2, 3, 6): mean 3, variance (4 + 1 + 0 + 9)/4 = 3.5, standard deviation 1.871, so the result is (−1.07, −0.53, 0, 1.60).
“More layers is always what makes a model bigger.” Width matters more per block: the weights grow with , so doubling the width (same number of blocks) gives about 4 times the weights, while doubling the blocks only doubles them.
5. Practice
- In the simulation, at which step does information move between tokens? At which step does each token change alone?
- Why can the same kind of block be stacked 12 or 100 times?
- Doubling (with the same number of blocks): roughly how many more weights?
Answers
- Between tokens: attention. Alone: the feed-forward network (and the normalisation).
- Its output has the same shape as its input ( numbers per token), and residual connections keep deep stacks trainable.
- About 4 times, since the count grows with .
Try it on your own data: the neural network toys lets you build and train tiny networks, the feed-forward part of every block.
New words: transformer block, residual connection, layer normalisation, feed-forward network, positional encoding, model width, parameters, logits.
- A transformer block does attention, add and normalise, a feed-forward network, add and normalise.
- Attention moves information between tokens; the feed-forward part works on each token alone.
- Residual connections (x + f(x)) and layer normalisation keep deep stacks trainable and steady.
- A block’s output has the same shape as its input, so the block is stacked N times.
- One block holds about 12d² weights, so model size grows with the square of the width.
Check yourself
Sources
- Vaswani, A. et al. (2017), Attention Is All You Need, Advances in Neural Information Processing Systems 30 https://arxiv.org/abs/1706.03762
- Ba, J. L., Kiros, J. R. and Hinton, G. E. (2016), Layer Normalization https://arxiv.org/abs/1607.06450
- He, K., Zhang, X., Ren, S. and Sun, J. (2016), Deep Residual Learning for Image Recognition, CVPR https://doi.org/10.1109/CVPR.2016.90
- Radford, A. et al. (2019), Language Models are Unsupervised Multitask Learners (GPT-2), OpenAI technical report
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.