Module 12: Private document search (RAG), fully offline · Lesson 12.5 · about 35 min
Finding, re-ranking and citing sources
Ask questions of your own documents and get answers with numbered sources. Add keyword search for names and codes, and an optional re-ranker for sharper results, all offline.
Module 12 · about 35 minutes · read first: 12.4. You’ll be able to ask questions of your indexed documents and get answers with numbered, checkable sources.
1. Why this matters
An answer you cannot check is a liability at work. Citations turn the model into a fast research assistant: it points at the page, and you (or your manager) can verify in ten seconds.
2. Ask your documents
Save as ask.py in askdocs, next to index.db:
import sqlite3
import sys
import numpy as np
import ollama
EMBED, CHAT, TOP_K = "embeddinggemma", "gemma4:e4b", 5
db = sqlite3.connect("index.db")
rows = db.execute("SELECT file, page, text, vec FROM chunks").fetchall()
matrix = np.array([np.frombuffer(r[3], dtype=np.float32) for r in rows])
matrix /= np.linalg.norm(matrix, axis=1, keepdims=True) # make every arrow length 1
def search(question, k=TOP_K):
q = np.array(ollama.embed(model=EMBED, input=question)["embeddings"][0], dtype=np.float32)
scores = matrix @ (q / np.linalg.norm(q)) # cosine similarity with every chunk at once
best = np.argsort(-scores)[:k]
return [(rows[i][0], rows[i][1], rows[i][2], float(scores[i])) for i in best]
def answer(question):
hits = search(question)
sources = "\n\n".join(f"[{n}] ({file}, page {page})\n{text}"
for n, (file, page, text, _) in enumerate(hits, 1))
prompt = ("Answer the question using only the sources below. After each fact, put the source "
"number in brackets, like [2]. If the sources do not contain the answer, say "
"'I could not find that in your documents.'\n\n"
f"Sources:\n{sources}\n\nQuestion: {question}")
reply = ollama.chat(model=CHAT, messages=[{"role": "user", "content": prompt}],
options={"temperature": 0})
return reply["message"]["content"], hits
if __name__ == "__main__":
text, hits = answer(" ".join(sys.argv[1:]))
print(text, "\n")
for n, (file, page, _, score) in enumerate(hits, 1):
print(f"[{n}] {file}, page {page} (score {score:.2f})")
Run python ask.py How many days of leave do new staff get?. The answer comes first, then the list of sources with their similarity scores. Temperature 0 keeps answers steady (Lesson 10.1).
3. Add keyword search (hybrid)
Embeddings are weak on exact codes, names and numbers: to a meaning map, “INV-20931” and “INV-20932” look almost the same. Old-fashioned keyword search is strong exactly there. Hybrid search runs both and merges the two ranked lists with reciprocal rank fusion (RRF): each list gives a chunk 1 ÷ (60 + its rank), and the scores are added. SQLite has a built-in keyword search (FTS5). Save as hybrid.py and call hybrid_search instead of search when your questions often contain codes. It returns (file, page, text) without a score, so in answer unpack three values instead of four:
import re
from ask import db, rows, search
# Build a keyword index once (SQLite's FTS5 comes with Python on most systems).
db.execute("CREATE VIRTUAL TABLE IF NOT EXISTS words USING fts5(text)")
if db.execute("SELECT COUNT(*) FROM words").fetchone()[0] == 0:
db.executemany("INSERT INTO words (rowid, text) VALUES (?, ?)",
[(i, r[2]) for i, r in enumerate(rows)])
db.commit()
def keyword_search(question, k=20):
terms = " OR ".join(f'"{w}"' for w in re.findall(r"\w+", question))
found = db.execute("SELECT rowid FROM words WHERE words MATCH ? ORDER BY rank LIMIT ?",
(terms, k)).fetchall()
return [rows[i][:3] for (i,) in found]
def hybrid_search(question, k=5):
"""Reciprocal rank fusion: a chunk scores 1/(60 + rank) in each list it appears in."""
score = {}
for ranked in (search(question, k=20), keyword_search(question)):
for rank, hit in enumerate(ranked, 1):
key = hit[:3]
score[key] = score.get(key, 0) + 1 / (60 + rank)
return sorted(score, key=score.get, reverse=True)[:k]
If you rebuild the index, delete the words table too (DROP TABLE IF EXISTS words in index_docs.py) so it is rebuilt to match.
The question. "What is the cancellation fee in the 2026 price list?" Three chunks are in the running: A (the 2026 price list), B (a guide to cancelling appointments) and C (a 2024 price list).
Meaning search. Vector search ranks B 1st, C 2nd, A 3rd: all three are "about cancelling and fees", and the exact year does not count for much.
Keyword search. FTS5 ranks A 1st (it has "2026", "fee" and "cancellation"), C 2nd, and does not find B at all.
Fuse the ranks. A: 1/63 + 1/61 ≈ 0.03227. C: 1/62 + 1/62 ≈ 0.03226, a whisker behind. B: 1/61 ≈ 0.0164. A comes first, and B, liked by only one list, drops to last.
Answer with numbers. The top chunks go into the prompt as [1], [2], [3]. The model writes "CA$40 if cancelled less than 24 hours ahead [1]", and the script prints "[1] policies/price-list-2026.pdf, page 2".
4. Optional: a re-ranker
A re-ranker reads the question and one chunk together and scores the pair. It is more accurate than vector search but too slow to run on everything, so take the top 20 and re-score them:
from sentence_transformers import CrossEncoder # pip install sentence-transformers
reranker = CrossEncoder("BAAI/bge-reranker-base") # downloads once (about 1 GB), then works offline
candidates = search(question, k=20)
scores = reranker.predict([(question, text) for _, _, text, _ in candidates])
best = [c for _, c in sorted(zip(scores, candidates), key=lambda p: -p[0])][:5]
Maths and principle
Principle: two weak, different rankings fused together beat either alone; rank fusion needs no tuning because it uses positions, not raw scores.
Equation:
Analogy: two judges at a cooking contest, one tasting (meaning), one checking the recipe was followed (exact words). A dish both rank in the top three wins over one only a single judge loved. The 60 stops a single first place from dominating. Where it breaks: if one list is junk, fusion drags good results down; check each list on its own first (Lesson 12.7).
Try it: chunk A is 1st in vectors and 3rd in keywords; chunk B is 2nd in vectors and absent from keywords. Scores? (A: 1/61 + 1/63 ≈ 0.0323; B: 1/62 ≈ 0.0161. A wins.)
- rank
- a chunk's position in one search's list: 1 for the best match, 2 for the next
- 60
- a fixed cushion from the original paper, so a single first place cannot outweigh agreement between lists
- Σ over lists
- add the scores from every list the chunk appears in; a list that missed it adds nothing
- citation [n]
- the number of the source the fact came from, printed with its file and page so a person can check
- Physical meaning
- Two judges with different tastes, one for meaning and one for exact words, each make a top list. A passage both of them rate highly is a safer bet than one only a single judge loved. The numbered citations let anyone walk to the shelf and check the page in ten seconds.
“If the answer has a citation, it must be right.” A citation only says which passage the model looked at. It can still misread it or mix two sources. Click through for anything that matters, and test with known questions (Lesson 12.7).
5. Worked example
Made-up numbers. A clinic manager asks “What is the cancellation fee in the 2026 price list?”. Vector search alone puts the price list 4th, behind three chunks about cancelling appointments in general. Hybrid search lifts it to 1st because “2026” and “fee” match exactly. The answer: “CA$40 if cancelled less than 24 hours ahead [1]”, with “[1] policies/price-list-2026.pdf, page 2” underneath.
6. Practice
- Ask a question your documents cannot answer. Does the model admit it?
- Ask about an exact code or name with
search, then withhybrid_search. Compare the order. - Make
ask.pyprint only sources whose score is above 0.3.
Answers
- It should say “I could not find that in your documents”. If it invents an answer, try a larger model (Gemma 4 12B) or make the instruction the first line of the prompt.
- The chunk with the exact code usually moves up.
hits = [h for h in search(question) if h[3] > 0.3]. The right threshold depends on the embedding model; check it with your test questions.
New words: hybrid search (meaning search plus keyword search), reciprocal rank fusion (combining rankings by position), re-ranker or cross-encoder (a model that scores a question and a passage together), citation (a pointer to the source of a fact).
ask.pyembeds the question, takes the top 5 chunks, and asks the model to answer only from them with numbered citations.- Tell the model to say “I could not find that in your documents” when the sources lack the answer; temperature 0 keeps answers steady.
- Keyword search (SQLite FTS5) catches exact codes, names and numbers that embeddings miss.
- Reciprocal rank fusion adds 1 ÷ (60 + rank) from each list, so chunks both searches like rise to the top.
- An optional re-ranker reads the question and each of the top 20 chunks together for a sharper order.
Check yourself
Sources
- Cormack, Clarke and Büttcher (2009), Reciprocal rank fusion outperforms Condorcet and individual rank learning methods, https://dl.acm.org/doi/10.1145/1571941.1572114
- SQLite FTS5 full-text search, https://sqlite.org/fts5.html
- Sentence Transformers CrossEncoder (re-rankers), https://sbert.net/docs/cross_encoder/usage/usage.html
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Prices and model names change fast: check the official source before you buy.