Home / Learn / Data & AI / Offline AI / Lesson 11.3
All lessons

Module 11: The local API · Lesson 11.3 · about 30 min

Streaming, JSON output and tool calling

Show answers as they are written, get clean JSON fields your code can use, and let the model call your own Python functions, all on your machine.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 11 · about 30 minutes · read first: 11.2. You’ll be able to stream answers, get structured JSON back, and let the model call your own functions.

Streaming shows the answer as it is written, a schema makes the model fill in exact fields, and tool calling lets it ask your code to do a job, while your code stays in charge.

1. Why this matters

The folder script in 11.2 returns free text. Real workflows need more:

  • a progress display while long answers are written (streaming);
  • fields you can put straight into a spreadsheet column (JSON with a schema);
  • a model that can ask your code to look something up (tool calling).

These three features are what make a pipeline dependable.

2. Streaming

Without streaming, you wait for the whole answer, like waiting for a letter. With streaming, each token is sent the moment it is written, like hearing someone speak. The answer and the total time are the same; you just start reading sooner.

import ollama

for chunk in ollama.chat(
    model="gemma4:e4b",
    messages=[{"role": "user", "content": "Explain what a context window is."}],
    stream=True,
):
    print(chunk["message"]["content"], end="", flush=True)
print()

3. JSON you can rely on

Describe the fields with Pydantic (installed with the ollama library) and pass the schema as format. A schema is a description of the fields and their types: “summary is text, action_needed is true or false, keywords is a list of words”. The engine then only produces text that fits it, like a paper form with labelled boxes.

from pydantic import BaseModel
import ollama

class Note(BaseModel):
    summary: str
    action_needed: bool
    keywords: list[str]

email = "Hi team, the client moved the launch to 3 March. Please update the plan by Friday. Thanks, Sara"

reply = ollama.chat(
    model="gemma4:e4b",
    messages=[{"role": "user", "content": "Read this email and fill in the fields.\n\n" + email}],
    format=Note.model_json_schema(),
    options={"temperature": 0},
)
note = Note.model_validate_json(reply["message"]["content"])
print(note.summary, note.action_needed, note.keywords)

Swap this into the folder script from 11.2 and each field becomes its own spreadsheet column.

4. Tool calling

Give the model a list of your functions. When it decides one would help, it replies with the function’s name and arguments instead of text. Your script runs it and sends the result back; the model then writes the final answer.

It works like a manager and an assistant. The model is the manager who says “please count the words in this”. Your script is the assistant who decides whether to do it, does it, and reports back.

The question. The user asks: "How many words are in: 'one two three'? Use the tool." The script sends it with the list of tools, here just word_count.

The model asks for a call. Instead of text, it replies with tool_calls: name word_count, arguments {"text": "one two three"}. It has not run anything.

Your script decides and runs it. The name is in TOOLS, so the script calls word_count("one two three") and gets 3. This is where you put safety checks.

The result goes back. The script adds a message with role tool and content "3" to the conversation and asks the model again.

The final answer. The model now writes plain text: "There are 3 words." Two requests, one function run, and your code was in control the whole time.

import ollama

def word_count(text: str) -> int:
    """Count the words in a piece of text.

    Args:
      text: the text to count
    """
    return len(text.split())

TOOLS = {"word_count": word_count}
messages = [{"role": "user", "content": "How many words are in: 'one two three'? Use the tool."}]

reply = ollama.chat(model="gemma4:e4b", messages=messages, tools=[word_count])
calls = reply["message"].get("tool_calls") or []
if calls:
    messages.append(reply["message"])
    for call in calls:
        name, args = call["function"]["name"], call["function"]["arguments"]
        result = TOOLS[name](**args) if name in TOOLS else "unknown tool"
        messages.append({"role": "tool", "content": str(result), "tool_name": name})
    reply = ollama.chat(model="gemma4:e4b", messages=messages)
print(reply["message"]["content"])

The model never runs anything itself. Your code decides, which is where you put safety checks (Lesson 13.8). Small models sometimes skip a tool or fill arguments badly; larger ones are more reliable, and Module 13 builds on this.

Maths and principle

Principle: a schema shrinks the set of allowed next tokens to those that keep the output valid.

Equation: at each step,

allowed tokens=all tokens∩tokens that keep the text matching the schema

(∩ means “in both lists”).

Analogy: filling in a paper form with boxes. You can still write what you like inside a box, but you cannot write outside it or skip a box.

Try it: a field allows only true or false. How many different values can come back? (2. That is why booleans from a schema need no cleaning.)

streaming
sending each token as soon as it is written instead of the whole answer at the end
schema
a list of fields and their types that the output must match
allowed tokens
at every step the engine blocks any token that would break the schema, so only valid text can be written
tool call
a reply that names one of your functions and its arguments, for your code to run or refuse
Physical meaning
Streaming is hearing a voice instead of waiting for a letter. A schema is a form with boxes: the model can write what it likes inside a box but cannot skip one or scribble outside. Tool calling is a manager asking an assistant to fetch something: the assistant, your code, decides whether to do it.

“With tool calling, the model runs code on my computer.” The model only writes a request: a function name and some arguments. Your script chooses whether to run it. If the name is not on your list, or the arguments look wrong, your code can simply refuse.

5. Worked example

Made-up numbers. A clinic manager runs the JSON version over 400 appointment emails. Each comes back as summary, action_needed and keywords. She filters action_needed = true in the spreadsheet: 37 emails need a reply today. Nothing left the clinic’s laptop.

6. Practice

  1. Run the streaming example and watch the words appear.
  2. Add a field urgency: int (1 to 5) to Note and run it on three emails.
  3. Add a second tool, today(), that returns today’s date, and ask “What is today’s date?”.
Answers
  1. If nothing prints until the end, check flush=True is there.
  2. Add urgency: int to the class. Say “1 to 5” in the prompt too: the schema fixes the type, the prompt gives the meaning.
  3. def today() -> str: return __import__("datetime").date.today().isoformat() and add it to both TOOLS and tools=[...].

New words: streaming (sending an answer piece by piece), schema (a description of the fields and their types), tool calling or function calling (the model asks your code to run a function).

  • Streaming (stream=True) shows tokens as they are written; the total time and the answer stay the same.
  • Pass a Pydantic schema as format and use temperature 0 to get exact fields your code can trust.
  • In tool calling the model only names a function and its arguments; your script runs it and sends back the result.
  • Your code is the place for safety checks: refuse unknown tools and bad arguments.
  • Small models sometimes skip tools or fill arguments badly; check results and use larger models for tool-heavy jobs.

Check yourself

1Why ask for JSON with a schema instead of asking politely for JSON in the prompt?
2In tool calling, who actually runs the function?
3What does streaming change?

Sources

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Prices and model names change fast: check the official source before you buy.