Home / Learn / Data & AI / Offline AI / Lesson 2.7
All lessons

Module 2: How a language model works · Lesson 2.7 · about 20 min

Why models make things up

Models predict likely text, not true text, so they sometimes produce confident errors. Why it happens, when it is most likely, and the habits that reduce it.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 2 · about 20 minutes · read first: 2.1. You’ll be able to predict when a model is likely to make things up and set up your work to catch it.

A model picks likely-sounding words, not checked facts, so when it does not know something it can still produce a fluent, confident answer that is wrong.

1. Why this matters

A wrong figure in a report or a made-up legal reference can cost far more than the time the model saved. Every serious workflow needs a plan for errors.

2. Why it happens

The loop in Lesson 2.1 picks likely text, not checked facts. When the model has learned a fact well, likely and true are the same. When it has not (rare names, recent events, exact numbers, references), the most fluent continuation can be a confident invention. This is called hallucination.

Think of a student in an exam who never learned the answer but must write something. A well-practised student writes a smooth, convincing paragraph anyway. The model is always that student: it must pick some next token.

A question it can't know. "Which library won the 2019 regional award for small branches?" The model never read the answer.

No clear favourite. For the first word of the name, many tokens score about the same: "Maple" 12%, "River" 10%, "Oak" 9%, and so on. None is strong.

It still picks one. The loop must choose. Say it picks "Maple".

Then everything flows. After "Maple", "Grove" is very likely (80%), then "Library" (90%). Each word fits the words before it, so the whole answer reads confidently, though only the first choice was a guess.

Give it the source. Paste the award page into the prompt and the right name scores 95%: it is right there to copy. This is why grounding works.

It is most likely with:

  • exact numbers, dates, quotes and references;
  • things after the model’s knowledge cut-off;
  • niche topics, small models, and high temperature;
  • questions that assume something false (“Why did X win in 2025?”).

3. Habits that reduce it

  1. Give it the source (paste, or document search in Module 12) and say “answer only from this text”.
  2. Ask for quotes or page numbers for each claim, so you can check.
  3. Allow “I don’t know”: say so in the system prompt.
  4. Use low temperature for facts.
  5. Check numbers with tools, not the model (a calculator, a spreadsheet, or tool calling in Lesson 11.3).
  6. A second pass: ask the model (or another one) to check each claim against the source.
Maths and principle

Principle: if each claim is right with probability p, a long answer with n independent claims is fully right with probability pⁿ.

Equation: P(all correct) = pⁿ.

Analogy: a chain of stepping stones: each is probably solid, but cross enough of them and the chance that one wobbles grows fast.

Try it: 20 claims each 98% likely to be right? (0.98²⁰ ≈ 0.67: one answer in three has at least one error. That is why checking matters.)

p
the chance that one claim in the answer is right, for example 0.98 (98%)
n
how many separate facts the answer states
pⁿ
p multiplied by itself n times: the chance every single claim is right
Physical meaning
Small error rates add up. A model that is right 98% of the time per fact still gives a long report with at least one mistake about a third of the time. Long, fact-heavy answers need checking; short answers grounded in a source you supplied need much less.

4. Worked example

Made-up numbers. A small charity asks a local model for “five grants for literacy projects in Ontario with deadlines”. It lists five, two of which do not exist. Next time they paste the funders’ pages and ask “list grants mentioned in this text, quoting the deadline line”. Every answer now comes with a quote they can check.

“If the model sounds sure, it’s probably right.” Tone tells you nothing. The words “definitely” and “according to” are just likely tokens like any others. A made-up reference reads exactly as smoothly as a real one. Judge an answer by whether you can check it against a source, not by how confident it sounds.

5. Practice

  1. Ask your model about a small local business or person it cannot know. What does it say?
  2. Add “If the text does not say, answer ‘not stated’” to a prompt. Does the answer change?
  3. Using the box, 10 claims at 95% each?
Answers
  1. Small models often invent plausible details; this is the behaviour to design around.
  2. Usually it starts admitting gaps instead of guessing.
  3. 0.95¹⁰ ≈ 0.60.

New words: hallucination (confident, made-up content), grounding (answering from supplied sources), knowledge cut-off.

  • Models pick likely text, not checked facts; when they never learned a fact, a fluent guess can still win.
  • Errors are most likely with exact numbers, dates, quotes, references, recent events, niche topics and small models.
  • If each claim is right with chance p, an answer with n claims is all right with chance pⁿ (0.98²⁰ ≈ 0.67).
  • Ground answers in text you supply, ask for quotes, allow “not stated”, and use low temperature for facts.
  • Check numbers with a calculator or spreadsheet, not the model, and never judge by how confident it sounds.

Check yourself

1Why do language models sometimes state false facts confidently?
2Which helps most to reduce made-up facts in answers about your documents?
3Are local models more or less prone to this than cloud models?

Sources

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Prices and model names change fast: check the official source before you buy.