Home / Learn / Data & AI / ML foundations / Lesson 10.5
All lessons

Module 10: From words to an LLM · Lesson 10.5 · about 25 min

The transformer block

A transformer is the same block stacked many times: attention, add and normalise, a small feed-forward network, add and normalise. Step through one from words to next-word probabilities and count its parameters.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 10 · about 25 minutes · read first: 9.3, 10.4. You’ll be able to name the parts of a transformer block in order, say what each one does to the numbers, and estimate a model’s size from its settings.

A transformer is one simple block copied many times: each token listens to the others, then thinks on its own, and keeps its old notes so nothing gets lost.

1. Why this matters

Every well-known LLM (GPT, Gemini, Claude, Llama, Gemma, Mistral) is a stack of transformer blocks. When you read that a model has “32 layers, model width 4,096 and 8 billion parameters”, this lesson tells you what those numbers mean. From them you can guess how much memory it needs and how fast it can run (the Offline AI course uses exactly this).

2. One block

Every token enters as a list of d numbers: its embedding plus a signal for its position. d is called the model width. One block does four things:

  1. Attention (Lesson 10.4): each token mixes in information from the earlier tokens it needs, using several heads.
  2. Add and normalise: add the block’s input back on (a residual connection), then rescale each token’s list to average 0 and spread 1 (layer normalisation).
  3. Feed-forward network: a small two-layer network (Lesson 9.3) that works on each token on its own: widen to about 4d numbers, bend, shrink back to d.
  4. Add and normalise again.
Try it: step through a transformer
  1. Tokenstext → 5 token ids
  2. Embedding + position5 × 512 numbers
  3. Attentionstill 5 × 512
  4. Add and normalisestill 5 × 512
  5. Feed-forward5 × 2,048 → 5 × 512
  6. Add and normalisestill 5 × 512
  7. × N blocksstill 5 × 512
  8. Next-word scores512 → 50,000 scores
  9. Softmax50,000 probabilities

The four boxes with a side line form one transformer block. Shapes are for a small made-up model: 5 tokens, 512 numbers per token, a 50,000-token vocabulary. A model's "size" (for example 8 billion parameters) counts every weight in the embedding table, the attention and feed-forward layers of every block, and the final scoring layer.

Attention moves information between tokens. The feed-forward part works on each token’s information alone. Stack the block N times, each copy with its own weights. Finally, score every vocabulary word from the last token’s numbers, and softmax the scores into next-word probabilities.

Follow one token, “sat”, through a block. Real models use thousands of numbers per token; here we use 4.

In. "sat" arrives as 4 numbers: its embedding plus a position signal saying it is the third word.

Listen (attention). "sat" looks back at "the" and "cat" and mixes in mostly "cat": who did the sitting. Out comes a change to add.

Add and normalise. Input plus change gives, say, (1, 2, 3, 6). Mean 3, standard deviation 1.871, so it becomes (−1.07, −0.53, 0, 1.60): same pattern, steady size.

Think alone (feed-forward). Widen the 4 numbers to 16, set the negative ones to 0 (the bend, ReLU), and shrink back to 4. This is where much of the model's stored knowledge acts.

Add and normalise again. Out come 4 numbers, the same shape as went in, ready for the next block.

After the last block. The final token's numbers give one score per vocabulary word, and softmax turns the scores into next-word probabilities.

3. Why the extra parts

  • Position: attention is a weighted average, and averages ignore order. Adding a position signal to each embedding (or rotating queries and keys by position, as most recent models do) lets “dog bites man” differ from “man bites dog”.
  • Residuals: as with very deep networks (Lesson 9.2), gradients fade as they pass back through many layers. 𝐱+f(𝐱) gives them a straight road back, so 100-block models still train.
  • Normalisation: keeps the numbers in a steady range from block to block, so the learning rate that suits one block suits all.
Maths and principle

Principle: alternate “talk to other tokens” (attention) and “think on your own” (feed-forward), with a shortcut round each.

One block, for the token vectors 𝐗:

𝐗′=LN⁡(𝐗+Attn⁡(𝐗))
𝐘=LN⁡(𝐗′+FFN⁡(𝐗′))

with FFN⁡(𝐱)=𝐖2ReLU⁡(𝐖1𝐱+𝐛1)+𝐛2. (Many models normalise before each part instead of after; the idea is the same.)

Layer normalisation of one token’s vector, with its own mean μ and standard deviation σ: LN⁡(xk)=(xk−μ)/σ, then a learned scale and shift.

Size: attention has 4 matrices of d×d (queries, keys, values, output); the feed-forward part has d×4d and 4d×d. So one block holds about

4d2+8d2=12d2 weights

Analogy: a meeting run in rounds. In each round everyone first listens to whoever is relevant (attention), then thinks alone for a moment (feed-forward), keeping their notes from before (residual).

Try it: a model with d=4,096 and 32 blocks. Weights in the blocks? (12×4,0962≈201 million per block; × 32 ≈ 6.4 billion, before the embedding table.)

d
model width: how many numbers describe each token inside the model
N
how many blocks are stacked (the "layers" in a model's description)
12d²
weights in one block: 4d² in attention plus 8d² in the feed-forward network
V × d
weights in the embedding table, one row of d numbers per vocabulary token
Physical meaning
The parameter count is the model's size on disk and in memory. At 2 bytes per weight, an 8-billion-parameter model needs about 16 GB just to load, which is why width and depth decide whether a model fits on your laptop.

4. Worked example

A small model like the one in the simulation: d=512, 12 blocks, vocabulary 50,000.

  • One block: 12×5122=3,145,728 weights, about 3.1 million.
  • 12 blocks: about 37.7 million.
  • Embedding table: 50,000×512=25.6 million (often shared with the final scoring layer).
  • Total: about 63 million parameters, about half the size of the first GPT model (2018).

Layer normalisation of one token’s vector (1, 2, 3, 6): mean 3, variance (4 + 1 + 0 + 9)/4 = 3.5, standard deviation 1.871, so the result is (−1.07, −0.53, 0, 1.60).

“More layers is always what makes a model bigger.” Width matters more per block: the weights grow with d2, so doubling the width (same number of blocks) gives about 4 times the weights, while doubling the blocks only doubles them.

5. Practice

  1. In the simulation, at which step does information move between tokens? At which step does each token change alone?
  2. Why can the same kind of block be stacked 12 or 100 times?
  3. Doubling d (with the same number of blocks): roughly how many more weights?
Answers
  1. Between tokens: attention. Alone: the feed-forward network (and the normalisation).
  2. Its output has the same shape as its input (d numbers per token), and residual connections keep deep stacks trainable.
  3. About 4 times, since the count grows with d2.

Try it on your own data: the neural network toys lets you build and train tiny networks, the feed-forward part of every block.

New words: transformer block, residual connection, layer normalisation, feed-forward network, positional encoding, model width, parameters, logits.

  • A transformer block does attention, add and normalise, a feed-forward network, add and normalise.
  • Attention moves information between tokens; the feed-forward part works on each token alone.
  • Residual connections (x + f(x)) and layer normalisation keep deep stacks trainable and steady.
  • A block’s output has the same shape as its input, so the block is stacked N times.
  • One block holds about 12d² weights, so model size grows with the square of the width.

Check yourself

1The output of a transformer block has:
2A residual connection:
3With d = 512 numbers per token, roughly how many weights are in one block?

Sources

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.