Home / Learn / Data & AI / Offline AI / Lesson 3.7
All lessons

Module 3: Tokens: the unit of everything · Lesson 3.7 · about 20 min

Saving tokens 2: trimming and summarising context

Long chats and long documents fill the context window. How to summarise history, send only the parts that matter, and split big documents so nothing important falls out.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 3 · about 20 minutes · read first: 3.3. You’ll be able to keep long work inside the context window without losing what matters.

Every turn of a chat re-sends everything before it, so keep the window lean: summarise old talk, send only the parts of a document that matter, and split what is too big.

1. Why this matters

The context window is the model’s whole world. Fill it with old small talk and there is no room for the document that matters. Overflow it and the start falls out. Managing it is the biggest lever on quality, speed and cost in long tasks.

2. Long chats: summarise and restart

Every message re-sends the whole conversation. Watch how fast that adds up, with turns of 200 tokens each.

Turn 1. The model reads your 200-token message. Total read so far: 200.

Turn 2. It reads turn 1 again plus the new 200: 400. Total so far: 200 + 400 = 600.

Turn 3. It reads 600. Total: 200 + 400 + 600 = 1,200, which is 200 × 3 × 4 ÷ 2.

Turn 30. The last turn alone reads 6,000 tokens, and the chat has read 200 × 30 × 31 ÷ 2 = 93,000 in all. Each turn is slower than the one before.

Summarise and restart. Ten bullet points of decisions take about 300 tokens. A new chat that starts from them reads 300 instead of 6,000 on its first turn.

After a long chat:

  1. Ask: “Summarise our decisions and open questions in 10 bullet points.”
  2. Start a new chat and paste the summary.

You keep the substance in a few hundred tokens instead of thousands. Try it in the simulation: fill the window, then tick “Summarise old messages”.

Try it: fill the context window

The bar is the window. The system prompt (S) always stays. When new text arrives and there is no room, the oldest messages fall out and the model cannot see them any more. A summary keeps the gist of old messages in far fewer tokens.

3. Long documents: send only what matters

  • Cut by hand when you know where the answer is: send the section, not the file.
  • Search, then send (RAG): split the document into chunks, find the chunks most related to the question, and send only those (Module 12).
  • Map-reduce when you need the whole thing summarised: summarise each part, then summarise the summaries.

4. Clean the input

Remove repeated headers and footers, signatures, long email chains already answered, page numbers and boilerplate. A cleaning step in your script (Lesson 11.2) can cut a third of the tokens from typical office documents.

Maths and principle

Principle: every turn re-sends the history, so a chat’s total input grows roughly with the square of its length.

Equation: total input over n turns of t tokens each ≈ t × n(n + 1) ÷ 2.

Analogy: reading a whole notebook from page one every time you add a line. Writing a short summary at the top and starting a new notebook keeps each read short.

Try it: 30 turns of 200 tokens? (200 × 30 × 31 ÷ 2 = 93,000 input tokens in total, though the last turn alone is only 6,000.)

t
tokens added in each turn (your message plus the reply)
n
the number of turns so far
n(n + 1) ÷ 2
1 + 2 + 3 + … + n: turn 1 is read n times, turn 2 n − 1 times, and so on
t × n(n + 1) ÷ 2
all the input tokens the chat has read in total
Physical meaning
Doubling the length of a chat roughly quadruples the reading it costs. Locally that is waiting time; in the cloud it is money and plan limits. A short summary resets the count close to zero while keeping what matters.

“Each message only costs the tokens I just typed.” The model has no memory between turns except the text it is sent, so every turn sends the whole conversation again. A one-line question at the end of a long chat can cost thousands of input tokens. That is why long chats get slower and drain plan limits faster.

5. Worked example

Made-up numbers. An engineer summarises a 120-page site report. Pasting it fails: about 80,000 tokens against a 16,000 window. Her script splits it into 12 sections, summarises each in 150 words, then summarises the 12 summaries into one page. Each request stays under 10,000 tokens, and the whole run takes 14 minutes on her laptop.

6. Practice

  1. In the simulation, fill a 4,000-token window with and without summarising. How many messages are lost?
  2. Using the box, how many input tokens do 50 turns of 300 tokens send in total?
  3. Which approach for “What does section 7.3 say about refunds?” in a 90-page policy?
Answers
  1. With summarising, far fewer are lost; the summary keeps their gist.
  2. 300 × 50 × 51 ÷ 2 = 382,500 tokens.
  3. Cut by hand (send section 7.3) or search for it (RAG). Map-reduce is for whole-document summaries.

New words: summarise and restart, chunk (a piece of a document), RAG (retrieval-augmented generation: search, then send), map-reduce (summarise parts, then combine).

  • Every turn re-sends the whole history: total input ≈ t × n(n + 1) ÷ 2 over n turns of t tokens.
  • After a long chat, ask for a short summary of decisions and open questions, then start fresh with it.
  • For long documents: cut by hand, search and send only relevant chunks (RAG), or map-reduce for whole summaries.
  • Clean the input: headers, footers, signatures and old email chains can be a third of the tokens.
  • A lean window is faster, cheaper and gives better answers.

Check yourself

1A chat has gone on for two hours and the model is slow and forgetful. What is the best fix?
2A 200-page manual will not fit. What is the usual approach?
3Summarising each chapter, then summarising the summaries, is called...

Sources

  • Lewis, P. et al. (2020), Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks https://arxiv.org/abs/2005.11401
  • Liu, N. F. et al. (2023), Lost in the Middle: How Language Models Use Long Contexts https://arxiv.org/abs/2307.03172
  • Dean, J. and Ghemawat, S. (2004), MapReduce: Simplified Data Processing on Large Clusters, OSDI 2004 (the idea behind map-reduce summarising)

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Prices and model names change fast: check the official source before you buy.