Module 2: How a language model works · Lesson 2.1 · about 20 min
Text in, text out: predicting the next token
A language model does one thing, over and over, which is to guess the next token. See how that one trick produces answers, code and essays, and what temperature does to it.
Module 2 · about 20 minutes · read first: 3.1. You’ll be able to explain how a model produces text, and what the temperature setting changes.
1. Why this matters
Every behaviour you will see, good and bad, follows from one simple loop. Once you see it, a lot makes sense: settings such as temperature, problems such as made-up facts, and limits such as speed.
Think of the phone keyboard that suggests your next word. A language model is that idea grown enormous: it has read far more text, so its suggestions can carry on a whole essay.
2. The loop
- The model reads all the tokens so far (your prompt, plus anything it has written).
- It gives every token in its vocabulary a score, then turns the scores into probabilities that add up to 100%.
- One token is chosen from those probabilities.
- That token is added to the text, and the loop starts again, until the model writes a special “stop” token or reaches a length limit.
That is all. Answers, poems, code and translations are all this loop, run hundreds of times. The skill is in the scores, which come from patterns learned in training (Lesson 2.3).
Watch one turn of the loop. The text so far is “The cat sat on the”, and to keep it small we look at only four of the many thousands of possible tokens.
Read. The model reads every token so far: "The", " cat", " sat", " on", " the".
Score. It gives each possible next token a score: " mat" 3, " sofa" 2, " roof" 1, " moon" 0. Higher means "fits better here".
Turn scores into chances. Softmax: e3 = 20.1, e2 = 7.4, e1 = 2.7, e0 = 1, total 31.2. Divide each by the total: 64%, 24%, 9%, 3%. They add up to 100%.
Pick. At temperature 1 the model draws from these chances, like a weighted lucky dip: " mat" most times, " sofa" now and then. At temperature 0 it always takes " mat".
Add and repeat. " mat" joins the text: "The cat sat on the mat". The model reads it all again and scores the next token, perhaps "." or " and".
3. Choosing the token: temperature
Temperature controls how the choice is made. Think of it as a “boldness” dial:
- 0: always the most likely token. Same input, same output. Best for facts, data extraction and code.
- about 0.7 to 1: mostly likely tokens, sometimes others. Natural for writing and chat.
- above about 1.2: unlikely tokens appear often. Creative, then rambling, then nonsense.
Lesson 10.1 covers the other dials (top-p, top-k, repeat penalty).
Maths and principle
Principle: scores become probabilities through the softmax function, and temperature divides the scores before that.
Equation: P(token i) = e^(sᵢ/T) ÷ Σⱼ e^(sⱼ/T), where sᵢ is the score and T the temperature.
Analogy: a crowd voting with louder and quieter voices. Low temperature lets the loudest voice win every time; high temperature turns everyone to the same volume.
Try it: two tokens with scores 2 and 1 at T = 1? (e² ≈ 7.39, e¹ ≈ 2.72; P = 7.39 ÷ 10.11 ≈ 73% and 27%.)
- si
- the score the model gives token i: how well it fits next
- T
- temperature: the boldness dial. Small T sharpens the gap between scores, large T flattens it
- es/T
- "e to the power": makes every score positive and stretches bigger scores further ahead
- P(token i)
- its share of the total: the chance it is picked. All the chances add up to 1 (100%)
- Physical meaning
- The model never "knows" the next word; it only has chances. At temperature 0 you always get the favourite, so the same question gives the same answer. Turn the dial up and the less likely words get a real chance, which reads as creativity, then as nonsense.
At temperature 0.5 the same scores become 6, 4, 2 and 0, and the chances become about 86%, 12%, 2% and 0.2%: ” mat” wins far more often. Try one yourself.
4. Worked example
Made-up numbers. Hana asks a model to pull the invoice number and total from 50 emails. At temperature 0.8 it sometimes rewords the total (“about 300 dollars”). At temperature 0 it copies the exact figures every time. She keeps 0 for extraction and 0.7 for drafting replies.
“The model thinks up the whole answer first, then types it out slowly.” It doesn’t keep a hidden finished draft. Each token is chosen in turn, using everything written so far, and once a token is written it shapes everything after it. That is why a wrong early word can send an answer off track, and why asking the model to plan or reason step by step first often helps: the plan becomes part of the text it reads.
5. Practice
- In the simulation, set temperature to 0 and pick ten times. Then 1.5. What changes?
- With scores 3 and 1 at T = 0.5, what is the probability of the first? (Use the box.)
- Which temperature would you use for a summary of a legal letter?
Answers
- At 0 you always get the top word; at 1.5 unlikely words appear often.
- Divide by T: 6 and 2. e⁶ ≈ 403, e² ≈ 7.4, so about 98%.
- Low, 0 to 0.3: you want faithful, repeatable wording.
New words: probability distribution (chances for every possible next token), softmax (the formula that turns scores into probabilities), temperature (how adventurous the choice is), stop token.
- A language model repeats one loop: read all tokens so far, score every possible next token, pick one, add it, repeat.
- Softmax turns scores into chances: P(i) = e^(sᵢ/T) ÷ the sum of e^(sⱼ/T) over all tokens.
- Temperature 0 always takes the most likely token (repeatable; best for facts and code); about 0.7 to 1 suits writing.
- A 300-token answer means 300 turns of the loop, which is why writing is slower than reading.
- All the skill is in the scores, which come from patterns learned in training.
Check yourself
Sources
- Jurafsky, D. and Martin, J. H., Speech and Language Processing, 3rd edition draft, chapters on language models and large language models https://web.stanford.edu/~jurafsky/slp3/
- Holtzman, A. et al. (2019), The Curious Case of Neural Text Degeneration (on sampling and temperature) https://arxiv.org/abs/1904.09751
- Brown, T. et al. (2020), Language Models are Few-Shot Learners https://arxiv.org/abs/2005.14165
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Prices and model names change fast: check the official source before you buy.