Module 2: How a language model works · Lesson 2.4 · about 25 min
Transformers and attention, explained with a classroom
The transformer is the design inside almost every language model. Attention lets each word look back at the words that matter to it. A plain-words picture, then the maths.
Module 2 · about 25 minutes · read first: 2.1 and 2.2. You’ll be able to describe what happens inside a model in plain words, and why long context costs memory and time.
1. Why this matters
You do not need the maths to run models. But this picture explains three practical things:
- why context costs memory (the KV cache);
- why long documents slow down;
- why models lose track of the middle of very long texts.
2. The classroom picture
Imagine each token is a student in a row, holding a card with its word. To understand “it” in “The cat sat on the mat because it was tired”, the student holding “it” needs to know who “it” is.
- Every student writes a question (“I’m a pronoun, who am I about?”), a label (“I’m an animal, a noun”) and some notes to share.
- The “it” student compares its question with every earlier student’s label. “cat” matches best, “mat” a little, “the” hardly at all.
- It then takes a mix of everyone’s notes, mostly from “cat”. Now “it” carries the meaning “the cat”.
That is attention. In model terms the question, label and notes are the query, key and value. Each token can only look back, never forward.
Here it is with small made-up numbers. To keep it short, “it” looks at just three earlier words.
Ask. "it" holds up its question: "who am I about?"
Compare. It checks the question against each earlier label. Match scores: "cat" 2, "mat" 1, "the" 0.
Turn scores into shares. Softmax: e2 = 7.39, e1 = 2.72, e0 = 1, total 11.11. Shares: "cat" 67%, "mat" 24%, "the" 9%.
Blend the notes. "it" takes 67% of cat's notes, 24% of mat's and 9% of the's. Its card now mostly says "animal, the one that sat".
Keep the labels and notes. The engine stores each word's key and value (the KV cache), so the next word can compare against them without working them out again.
3. The stack
A transformer stacks this, layer upon layer (often 30 to 90 layers). After each attention step, a small calculation (the feed-forward part, where most parameters live) works on each token. Early layers handle grammar, later ones meaning and reasoning. At the top, the last token’s result becomes the scores for the next token (Lesson 2.1).
4. What it means for you
- KV cache: to avoid recomputing, the engine stores every earlier token’s keys and values. That is the conversation memory that grows with context (Lesson 8.4).
- Longer context, slower tokens: each new token compares with every stored key.
- Lost in the middle: with very long inputs, attention spreads thin, and middle parts get less weight (Lesson 17.4).
Maths and principle
Principle: attention is a weighted average of earlier tokens’ values, with weights from how well their keys match the current query.
Equation: Attention(Q, K, V) = softmax(Q Kᵀ ÷ √d) V, where d is the length of each key.
Analogy: a search engine inside the sentence: the query is matched against every key, and results are blended by how well they match.
Try it: a query matches three keys with scores 2, 0 and 0. With softmax, how much weight goes to the first? (e² ÷ (e² + 1 + 1) ≈ 7.39 ÷ 9.39 ≈ 79%.)
- Q (query)
- what the current token is looking for: the question on its card
- K (key)
- what each earlier token offers: its label
- V (value)
- the information each earlier token passes on if chosen: its notes
- Q Kᵀ ÷ √d
- the match scores, one per earlier token; dividing by √d (d is the length of a key) keeps the scores from getting too big
- softmax(...) V
- turn scores into shares that add up to 1, then blend the values by those shares
- Physical meaning
- Each token decides whom to listen to. After attention, the token for "it" carries the cat's meaning, so the model can use it correctly. Every new token must compare with every stored key, which is why long chats use more memory and get slower.
5. Worked example
Made-up numbers. A model reads a 40-page contract. The answer to Gita’s question is on page 21. The model misses it. She moves the question to the end of the prompt and asks it to quote the clause first; it now finds it. Attention at the start and end of long texts is stronger than in the middle.
“If it fits in the context window, the model reads every part equally carefully.” Not quite. Attention shares are spread over every token, and tests (Liu and colleagues, 2023, in the sources) found models use facts at the start and end of a long input much better than facts in the middle. For long documents, put your question last, ask for quotes, or send only the relevant parts (Module 12).
6. Practice
- In “The trophy did not fit in the suitcase because it was too big”, what should “it” attend to most?
- Why can’t a token attend to tokens after it?
- Why does doubling the context roughly double the KV cache?
Answers
- “trophy”. (Change “big” to “small” and it becomes “suitcase”, a classic test of understanding.)
- The model writes left to right, so later tokens do not exist yet when it predicts.
- The cache stores keys and values for every token, so twice the tokens means twice the storage.
Go deeper: the maths of attention, with a live attention map you can change, is in Attention: each word looks at the others, part of the free course From a straight line to an LLM.
New words: transformer, attention, query, key, value, layer, feed-forward, KV cache.
- Attention: each token compares its query with earlier tokens’ keys, turns the scores into shares with softmax, and blends their values.
- A transformer stacks dozens of layers, each attention plus a feed-forward step where most parameters live.
- Tokens only look back, never forward, because later tokens do not exist yet.
- The KV cache stores every earlier token’s keys and values, so it grows with the length of the conversation.
- Long inputs are slower and the middle gets less attention: put the question last and ask for quotes.
Check yourself
Sources
- Vaswani, A. et al. (2017), Attention Is All You Need https://arxiv.org/abs/1706.03762
- Liu, N. F. et al. (2023), Lost in the Middle: How Language Models Use Long Contexts https://arxiv.org/abs/2307.03172
- Jurafsky, D. and Martin, J. H., Speech and Language Processing, 3rd edition draft, chapter on transformers https://web.stanford.edu/~jurafsky/slp3/
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Prices and model names change fast: check the official source before you buy.