Home / Learn / Data & AI / Offline AI / Lesson 2.3
All lessons

Module 2: How a language model works · Lesson 2.3 · about 20 min

Training and inference: learning versus using

Training a model takes months on thousands of chips; using it takes seconds on your laptop. Why the two are so different, and why you only ever do the second at home.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 2 · about 20 minutes · read first: 2.1 and 2.2. You’ll be able to explain why running a model is cheap and making one is not.

Training sets the model's numbers once, at huge cost; inference only uses those numbers, so it is cheap and never changes what the model knows.

1. Why this matters

People often expect a local model to “learn” their business by chatting with it. It will not. Knowing the difference between training and inference tells you which tool gives the model what it needs: context, document search or fine-tuning.

2. Training: making the model

A model starts as random numbers. Training shows it text and asks it to predict each next token. It measures how wrong the guess was and nudges every parameter a tiny bit to be less wrong. Repeat over trillions of tokens and the numbers come to hold grammar, facts, reasoning patterns and styles.

It is like learning to throw darts. Each throw, you see how far you missed and adjust your arm a little. After millions of throws your arm “knows” the board.

Show a sentence. The training text says "The cat sat on the mat". The model sees "The cat sat on the" and must guess the next token.

It guesses. Early in training it gives " mat" only 20%. Its favourite is " table" at 30%.

Measure the miss. The right answer was " mat". The error is measured as −ln(0.20) = 1.61 (a perfect guess of 100% would score 0).

Nudge every knob. The training program works out which way each of the billions of parameters should move to raise " mat", and moves each one a tiny bit. Next time " mat" gets 25%; the error falls to 1.39.

Repeat trillions of times, on every token of every sentence. Then training stops and the numbers are frozen. From now on, using the model (inference) reads them but never moves them.

This takes data centres with thousands of specialised chips running for weeks or months. Afterwards, post-training teaches the model to follow instructions, chat and refuse harmful requests, using examples and feedback.

3. Inference: using the model

Inference is running the finished model: read the input, predict tokens, done. The weights do not change. It takes one machine and seconds. This is all you do at home (and all a cloud chatbot does when you use it).

“If I chat with my local model every day, it will learn my business.” It won’t. Its parameters are frozen; chatting is inference, and inference never changes them. Within one chat it can use what you told it, because that text is in its context window (Lesson 3.3). Start a new chat and it is gone. Some apps add a “memory” feature, but that just saves notes and pastes them back into the context.

4. So how does it “know” my stuff?

  • In the context: paste or attach it (Lesson 3.3). Simple, limited by window size.
  • Document search (RAG): find the relevant parts of your files and add them automatically (Module 12).
  • Fine-tuning: a small extra round of training on your examples (Module 14), for style and format more than facts.
Maths and principle

Principle: training adjusts weights to reduce prediction error; inference only uses them.

Equation: rough compute for training ≈ 6 × parameters × training tokens; for inference ≈ 2 × parameters per token generated.

Analogy: writing a textbook versus reading it. Writing takes years and a team; reading a page takes a minute, and reading never changes the book.

Try it: with these rules, how many times more compute does training a 10B model on 10 trillion tokens take than generating 1,000 tokens? (6 × 10¹⁰ × 10¹³ ÷ (2 × 10¹⁰ × 10³) = 3 × 10¹⁰: thirty billion times.)

P
parameters: how many numbers the model has
D
training tokens: how much text the model read while learning, often trillions
6 × P × D
rough number of calculations to train: about 2 per parameter per token to predict, and about 4 more to work out the nudges
2 × P per token
rough number of calculations to write one token at home: only the predicting part
Physical meaning
Training is writing the textbook; inference is reading a page. A company spends months and fortunes once, and then anyone with the file can use the result for the cost of electricity. That is why you can run a model you could never afford to make.

5. Worked example

Made-up numbers. A hotel manager chats with a local model for a week about their room rates, then asks “What is our weekend rate?”. The model guesses wrong: nothing from earlier chats changed it. They add the rate card as a document to search (Module 12) and the model answers correctly every time.

6. Practice

  1. Tell your model a made-up fact, start a new chat and ask about it. What happens?
  2. Which of the three ways would you use for your company’s 200-page policy manual?
  3. Why is it fine to run a model with no internet?
Answers
  1. It does not know: new chats start with an empty context, and weights never change.
  2. Document search (RAG): too long for one context, and facts are better found than trained in.
  3. Inference only needs the weights on your machine.

New words: training, post-training (teaching a trained model to follow instructions), inference, fine-tuning (a small extra training run on your own examples).

  • Training nudges every parameter, over trillions of tokens, to predict the next token better; it takes thousands of chips for weeks or months.
  • Inference uses the frozen parameters to answer; it takes one machine and seconds, and never changes the model.
  • Compute: training ≈ 6 × parameters × training tokens; inference ≈ 2 × parameters per token written.
  • A local model does not learn from your chats; it only sees what is in its context window.
  • To give it your knowledge: put it in the context, use document search (RAG), or fine-tune for style and format.

Check yourself

1What happens during training?
2Does a local model learn from your chats?
3Why is training so much more expensive than inference?

Sources

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Prices and model names change fast: check the official source before you buy.