Home / Learn / Data & AI / ML foundations / Lesson 10.1
All lessons

Module 10: From words to an LLM · Lesson 10.1 · about 20 min

Text as numbers: bag of words and TF-IDF

Models need numbers, so text must be turned into numbers first. Count the words, weight the rare ones up with TF-IDF, and compare documents by the angle between their rows. See it on four short reviews.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 10 · about 20 minutes · read first: 1.4, 4.8. You’ll be able to turn a set of documents into a table of numbers, work out TF-IDF by hand, and compare documents with cosine similarity.

A computer can't read, but it can count: turn each document into a row of word counts, give rare words more weight, and similar documents become rows that point the same way.

1. Why this matters

Reviews, support tickets, emails, contracts and survey comments are all text, and most of the world’s data is text. A model only understands numbers, so text has to become numbers first.

The simple methods in this lesson still run search boxes, spam filters and ticket routing. They are the honest baseline a language model has to beat. They are also the first step on the road to embeddings (Lesson 10.2) and LLMs.

2. Bag of words

First make a vocabulary: a sorted list of every different word in the collection. Each document then becomes a row with one column per vocabulary word. Each cell holds how many times that word appears in the document.

Word order is thrown away, as if you tipped the words into a bag and shook it. So “dog bites man” and “man bites dog” get exactly the same row. Naive Bayes (Lesson 4.8) worked like this.

Most cleaning happens before counting:

  • make everything lower case, so “Great” and “great” are one word;
  • split on spaces and punctuation;
  • sometimes drop very common stop words (“the”, “and”);
  • sometimes cut words to their stem (“charging” → “charg”), so “charge” and “charging” count together.

Everyday picture: a shopping receipt. It lists how many of each item you bought, but not the order you picked them off the shelves.

3. TF-IDF: weight the words that tell documents apart

Raw counts favour words like “the”, which are everywhere and tell you nothing. TF-IDF (term frequency times inverse document frequency) multiplies each count by a weight. The weight is large for rare words and zero for words found in every document.

  • TF (term frequency): how often the word appears in this document.
  • IDF (inverse document frequency): ln⁡(N/df). Here N is the number of documents, df is how many of them contain the word, and ln is the natural logarithm.

Watch it happen for the word “great” in the first of four reviews:

Count it. Review 1 is "the battery is great and the screen is great". "great" appears 2 times. That is its TF.

How many reviews use it? Of the 4 reviews, only review 1 contains "great", so df = 1.

Rarity weight. IDF = ln(4 ÷ 1) = ln 4 = 1.386. A word in only one review gets a big weight.

Multiply. TF-IDF = 2 × 1.386 = 2.77. "great" is the word that tells this review apart.

Compare with "the". "the" appears 2 times too, but it is in all 4 reviews: IDF = ln(4 ÷ 4) = 0, so its TF-IDF is 0. Common words vanish.

Try it: which words matter?
wordcountin reviewsIDFTF-IDF
most similar review (cosine of TF-IDF)
same, with plain counts
IDF = ln(4 ÷ number of reviews with the word). "the", "is" and "and" appear in all four reviews, so their IDF is ln 1 = 0 and they drop out, however often they occur. Words found in only one review get the biggest weight. Each review becomes a row of numbers, one per word in the vocabulary, and two reviews are similar when their rows point the same way (cosine similarity). Word order is lost: "the battery died" and "died the battery" get the same numbers.
count(w, d)
how many times word w appears in document d (the TF)
N
how many documents there are in the whole collection
df(w)
how many documents contain word w at least once
ln(N / df)
the rarity weight (IDF): 0 for a word in every document, bigger the rarer the word is
Physical meaning
TF-IDF is a highlighter pen. It marks the words that make one ticket, review or email different from the rest, and leaves the filler words blank. A search box uses it to rank the documents where your search words stand out.

Two documents are compared by the angle between their rows: cosine similarity (Lesson 1.4). A long review and a short review with the same mix of words point the same way, so length does not matter. Cosine runs from 0 (no weighted words in common) to 1 (the same mix).

Maths and principle

Principle: a word matters to a document if it is frequent there and rare elsewhere.

TF-IDF of word w in document d, with N documents of which df(w) contain w:

tfidf(w,d)=count(w,d)×ln⁡Ndf(w)

Cosine similarity of rows 𝐚 and 𝐛: the dot product (multiply place by place and add), divided by the two lengths.

cos⁡θ=𝐚⋅𝐛‖𝐚‖‖𝐛‖

Libraries add small tweaks (adding 1 inside the logarithm, dividing each row by its length), but the idea is the same.

Analogy: in a library, “the” is printed in every book and tells you nothing; “photosynthesis” points straight at the biology shelf.

Try it: 10 documents; “refund” is in 2 of them and appears 3 times in one. Its TF-IDF there? (3×ln⁡(10/2)=3×1.609=4.83.)

4. Worked example

The four reviews in the simulation. Review 1 is “the battery is great and the screen is great”.

word count in reviews IDF = ln(4/df) TF-IDF
great 2 1 1.386 2.77
screen 1 2 0.693 0.69
battery 1 3 0.288 0.29
the, is, and 2, 2, 1 4 0 0

“great” carries the review; “the”, “is” and “and” vanish. With plain counts, review 1 looks most like review 3 (cosine 0.72), mostly because both are full of “the” and “is”. With TF-IDF the similarity drops to 0.07: the only weighted word they share is “screen”.

“Bag of words understands what the text means.” It only counts. “not good” and “good” share the word “good”, so they look alike. “cheap” and “inexpensive” share no letters, so they look unrelated. Word order and meaning are both lost. Embeddings (Lesson 10.2) fix the second problem, and attention (Lesson 10.4) the first.

5. Practice

  1. In the simulation, which review is most similar to review 2, and which word links them?
  2. Why do “the”, “is” and “and” get TF-IDF 0 here, even though they are the most common words?
  3. Name one thing bag of words cannot capture.
Answers
  1. Review 4, through “charger”, “slow” and “died”: the same complaint in a different order.
  2. They are in all four reviews, so IDF = ln(4/4) = 0, and 0 times any count is 0.
  3. Word order and meaning: “not good” and “good” share a word; “cheap” and “inexpensive” share none. Embeddings (Lesson 10.2) address the second.

Try it on your own data: the text cleaner does the cleaning steps (lower case, punctuation, stop words) on your own text, and the topic modelling (LDA) finds themes in many documents from their word counts.

New words: vocabulary, bag of words, term frequency, inverse document frequency, TF-IDF, stop words, stemming, cosine similarity.

  • Bag of words turns each document into a row of word counts, one column per vocabulary word; word order is lost.
  • TF-IDF multiplies each count by ln(N / df), so rare words get weight and words in every document get 0.
  • Cosine similarity compares two rows by their angle, from 0 (nothing shared) to 1 (same mix), whatever their length.
  • These counts can’t see meaning or order: “cheap” and “inexpensive” look unrelated, which embeddings fix.

Check yourself

1A word appears in every document of a collection. Its IDF is
2In a bag-of-words model, each document becomes:
3Cosine similarity between two documents' TF-IDF rows is 0. They

Sources

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.