Home / Learn / Data & AI / Offline AI / Lesson 12.3
All lessons

Module 12: Private document search (RAG), fully offline · Lesson 12.3 · about 25 min

Cutting documents into chunks

Cut long documents into pieces small enough to search and big enough to make sense. Choose chunk size and overlap with a calculator, and write a 10-line chunker in Python.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 12 · about 25 minutes · read first: 12.2. You’ll be able to choose a chunk size and overlap, and cut any text into chunks with Python.

Cut each document into pieces small enough to be about one thing but big enough to make sense alone, with a little overlap so no fact is sliced in half.

1. Why this matters

Chunking is the most underrated setting in RAG. The same model and the same files can give good or poor answers just from how the documents were cut. It is also the setting you can tune for free.

2. Too small, too big, just right

  • Too small (one sentence): each chunk is precise but loses its context. “It must be signed within 30 days” no longer says what it is.
  • Too big (whole chapters): each vector blurs many topics together, search gets vague, and five of them may not fit in your context window.
  • Just right: usually 200 to 500 words (about 250 to 650 tokens), one idea or a few paragraphs, with 10 to 20% overlap.

Think of cutting a long photo strip into frames for an album index. Tiny frames show sharp details but no scene. Huge frames show the whole party, but the index card can only say “a party”.

Better still, cut along the document’s own seams: headings, then paragraphs, and only then by length. Keep each chunk’s file name and page number beside it: you need them for citations.

Calculator: chunk size

Assumes about 500 words a page and 1.33 tokens per English word; other languages and code use more tokens (Lesson 3.2). Index size counts the vectors only; the stored text adds about as much again. The right size for your files comes from testing (Lesson 12.7), not from this rule of thumb.

3. A chunker in ten lines

Save this as chunker.py in a new folder called askdocs. The next lessons build the rest of the project beside it.

def chunk_text(text, size=300, overlap=50):
    """Cut text into pieces of about `size` words; each piece repeats the last `overlap` words of the one before."""
    words = text.split()
    step = size - overlap
    pieces = []
    for start in range(0, max(len(words) - overlap, 1), step):
        piece = " ".join(words[start:start + size])
        if piece:
            pieces.append(piece)
    return pieces

It counts words, not tokens, because that needs no tokenizer; 300 English words is roughly 400 tokens. Test it: print([len(p.split()) for p in chunk_text(" ".join(["w"] * 700))]) gives [300, 300, 200].

The document. 700 words. Settings: size=300, overlap=50, so each new chunk starts 300 − 50 = 250 words after the last.

Chunk 1. Words 1 to 300.

Chunk 2. Starts at word 251 and runs to 550. Words 251 to 300 are in both chunks: that is the overlap. A sentence cut at word 300 appears whole in chunk 2.

Chunk 3. Starts at word 501 and runs to the end, word 700, so it holds 200 words. The loop stops: the next start (751) is past the end.

The count. (700 − 50) ÷ 250 = 2.6, rounded up to 3 chunks, matching [300, 300, 200].

Maths and principle

Principle: chunk size trades precision of each match against context carried by each match; overlap guards the seams.

Equation: for a document of W words, chunk size s and overlap o,

number of chunks=⌈W−os−o⌉

(⌈ ⌉ means round up). Prompt sources ≈ k × s.

Analogy: cutting a long photo strip into frames for an album index. Tiny frames show sharp details but no scene; huge frames show the whole scene but the index card for each can only say “a party”. Overlap is printing the edge of each frame on the next, so nobody’s face is cut in half. Where it breaks: tables and code do not have word-level meaning; cut them by rows or functions instead.

Try it: a 10,000-word report, s = 300, o = 50. About how many chunks? ((10,000 − 50) ÷ 250 ≈ 39.8, so 40.)

W
the number of words in the document
s (chunk size)
the most words in one chunk
o (overlap)
how many words each chunk repeats from the end of the one before
s − o (step)
how far along the text each new chunk starts
Physical meaning
Chunking is cutting a long scroll into index cards. Each card should hold one idea so its label is sharp, and the edges overlap a little so no sentence is torn in half. More overlap means more cards and a bigger index, but fewer facts lost at the seams.

“Bigger chunks give the model more context, so they are always better.” One vector has to stand for the whole chunk. Mix ten topics into it and it matches none of them well, and five big chunks may not fit in your context window. Test sizes with your own questions (Lesson 12.7); 200 to 500 words is a good start.

4. Worked example

Made-up numbers. An HR team tests three settings on 50 known questions (the method is in Lesson 12.7). 100-word chunks: right passage in the top 5 for 33 of 50 questions. 300 words with 50 overlap: 44 of 50. 1,000 words: 36 of 50, and prompts three times longer. They keep 300/50.

5. Practice

  1. Chunk a document of 1,000 words with size 300 and overlap 0, then with overlap 100. How many chunks each time?
  2. Why might legal contracts like a slightly larger chunk size than chat logs?
  3. Change chunk_text so it first splits on blank lines (paragraphs) and only cuts paragraphs longer than size.
Answers
  1. Overlap 0: 4 chunks (300, 300, 300, 100). Overlap 100: 5 chunks (step 200).
  2. Clauses are long and refer back within the same section; chat messages are short and self-contained.
  3. for para in text.split("\n\n"): then pieces += chunk_text(para, size, overlap) if len(para.split()) > size else [para.strip()], skipping empty paragraphs. Very short paragraphs can be joined to the next one.

New words: chunk size (how long each piece is), overlap (words shared between neighbouring chunks), top-k (how many chunks the search returns).

  • Chunk size trades a sharp match (small) against enough context to make sense (big); 200 to 500 words is a good start.
  • Overlap of 10 to 20% keeps facts that straddle a cut whole in at least one chunk.
  • Cut along headings and paragraphs first, and only then by length; tables and code by rows or functions.
  • Number of chunks = ⌈(W − o) ÷ (s − o)⌉; prompt sources ≈ k × s.
  • Keep each chunk’s file name and page beside it for citations.

Check yourself

1Your chunks are 3,000 tokens each. What is the likely problem?
2Why let neighbouring chunks overlap a little?
3Top-5 chunks of 400 tokens. Roughly how many tokens of sources go into each prompt?

Sources

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Prices and model names change fast: check the official source before you buy.