Home / Learn / Data & AI / Offline AI / Lesson 13.5
All lessons

Module 13: Agents and automation offline · Lesson 13.5 · about 25 min

Reading images and screenshots locally

Send photos, receipts, charts and screenshots to a local vision model and get descriptions or clean JSON back. Know what vision models read well, what they get wrong, and what an image costs in tokens.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 13 · about 25 minutes · read first: 11.3. You’ll be able to send images to a local model and get descriptions or structured data back.

A vision model cuts a picture into small squares and turns them into a few hundred tokens that sit next to your words, so it can read and describe images but loses detail finer than that budget.

1. Why this matters

A lot of work information is in pictures: receipts, whiteboard photos, scanned forms, screenshots of error messages, charts in reports. A local vision model reads them without uploading anything. That matters for ID cards, medical images and internal dashboards.

2. Which models can see

Gemma 4 models take images as well as text, so the model you installed in Module 9 may already do this. Other families have vision versions too (look for “VL” or “vision” in the name in the Ollama library). Bigger is better for small print and dense documents: E4B reads a clear receipt; 12B and up handle forms and charts far better.

How the model sees. A part called the image encoder shrinks the picture to a fixed size, cuts it into small squares called patches, and turns each patch into numbers the model can read, much like text tokens. Many models then merge neighbouring patches to save room. The numbers below follow one common design (an 896-pixel square, 14-pixel patches, 4 × 4 merging); your model may differ, but the idea is the same.

The photo. A receipt photo of 3,000 × 4,000 pixels. Far too many to read one by one.

Resize. The encoder squeezes it to a fixed 896 × 896 square, whatever its original size. Tiny print may already blur here.

Cut into patches. 896 ÷ 14 = 64 patches across and 64 down: 64 × 64 = 4,096 patches.

Merge. Each block of 4 × 4 = 16 neighbouring patches becomes one token: 4,096 ÷ 16 = 256 image tokens.

Read with your words. The 256 image tokens go into the context next to your prompt ("Read this receipt..."), and the model answers in text, or in JSON if you ask (section 4).

3. Describe an image

import ollama

reply = ollama.chat(model="gemma4:12b", messages=[{
    "role": "user",
    "content": "What does this screenshot show? If it is an error, explain it and suggest a fix.",
    "images": ["error.png"],
}])
print(reply["message"]["content"])

The images list takes file paths (or raw bytes). In the terminal, ollama run gemma4:12b also accepts a path typed in the prompt, and chat apps such as LM Studio and Open WebUI let you drag an image in.

4. Turn receipts into data

Combine vision with a JSON schema (Lesson 11.3):

from pydantic import BaseModel
import ollama

class Receipt(BaseModel):
    shop: str
    date: str
    total: float
    currency: str

reply = ollama.chat(
    model="gemma4:12b",
    messages=[{"role": "user", "content": "Read this receipt and fill in the fields. Date as YYYY-MM-DD.",
               "images": ["receipt.jpg"]}],
    format=Receipt.model_json_schema(),
    options={"temperature": 0},
)
print(Receipt.model_validate_json(reply["message"]["content"]))

Loop over a folder of receipt photos and write each to a CSV row, as in Lesson 11.2, and you have a private expense tracker.

Maths and principle

Principle: an image becomes a fixed budget of tokens; detail beyond what those tokens can hold is lost.

Equation: image tokens ≈ (width ÷ patch size) × (height ÷ patch size) ÷ pooling. Many models resize every image to a fixed grid and spend a few hundred tokens per image, whatever its original size.

Analogy: describing a painting to someone over the phone in exactly 250 words. A portrait fits easily; a page of small print does not, so you summarise and details vanish. Where it breaks: some engines tile large images into several pieces, which keeps detail but multiplies tokens and time.

Try it: a model spends 256 tokens per image and your context is 8,192 tokens. Leaving 2,000 tokens for text, how many images fit? ((8,192 − 2,000) ÷ 256 ≈ 24.)

width, height
the size of the picture after the encoder resizes it, in pixels
patch size
the side of each small square the picture is cut into (often 14 or 16 pixels)
pooling
how many neighbouring patches are merged into one token
image tokens
what the picture costs in the context window, just like words do
Physical meaning
Describing a painting over the phone in exactly 250 words. A face fits easily; a page of small print does not, so details vanish. That is why a vision model reads a clear receipt well but misses tiny print, and why each image eats into the room left for text.

“If the model read a number off the receipt, the number must be there.” Unlike ordinary OCR, which garbles unclear digits, a vision model can write a neat, believable number that is not on the page, especially from a blurry or faded photo. Spot-check totals against another record, such as the bank statement.

5. Worked example

Made-up numbers. A small builder photographs 140 supplier receipts a month. The receipt script with Gemma 4 12B on a 16 GB graphics card takes about 6 seconds a receipt, 14 minutes in all. Checking the totals against the bank statement finds 3 misreads (two blurry photos, one faded thermal print), which are fixed by hand. Before, the job took an evening.

6. Practice

  1. Take a photo of any printed page and ask the model to transcribe it. Compare with OCRmyPDF from Lesson 12.6.
  2. Run the receipt script on three receipts. Did any field come back wrong?
  3. Screenshot a chart and ask “what is the highest value and when?”. Check it yourself.
Answers
  1. On clean print, OCR is usually more exact; on photos at an angle or with handwriting, the vision model often does better.
  2. Dates in other formats and faded totals are the usual mistakes. Saying “YYYY-MM-DD” in the prompt helps.
  3. Models read labels well but can misjudge values between gridlines. Never use a vision model’s reading of a chart as data without checking.

New words: vision model or multimodal model (takes images as well as text), image tokens (an image turned into tokens), image encoder (the part that turns pixels into tokens), patch (one small square of the picture).

  • Gemma 4 and other “vision” or “VL” models take images as well as text; pass file paths in the images list.
  • The image encoder resizes the picture, cuts it into patches and turns them into a fixed budget of image tokens, often a few hundred.
  • Image tokens use context space like text, so many images in one request leave less room for words.
  • Combine vision with a JSON schema to turn receipts and forms into clean data.
  • Vision models read and describe well but miscount, misjudge chart values and can invent digits: check numbers.

Check yourself

1How does a vision model "see" an image?
2Which job suits a local vision model least?
3Why check numbers that a vision model reads from a receipt?

Sources

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Prices and model names change fast: check the official source before you buy.