Module 10: Settings and prompting · Lesson 10.3 · about 15 min
Context length settings and their memory cost
Why local engines start with a short context, how to raise it in Ollama, LM Studio and llama.cpp, and how to pick a length your memory can hold.
Module 10 · about 15 minutes · read first: 3.3 and 8.4. You’ll be able to set a context length that holds your documents and still fits in memory.
1. Why this matters
A model card may say “128K context”, but your engine may start at 4,096 tokens to save memory. Long documents then get cut off without any warning. Setting the right length is one of the most common fixes for “it ignored my file”.
2. Defaults
- Ollama: depends on graphics memory. A late-2026 guide reports 4,096 tokens under 24 GB, 32,768 for 24 to 48 GB, and 256,000 at 48 GB or more. Check your version’s documentation.
- LM Studio: shows a context length slider when you load a model, with an estimate of the memory it will use.
- llama.cpp:
-c 16384sets it;-c 0uses the model’s maximum.
The context is like a desk. A bigger desk holds more papers at once, but it takes up more of the room (memory). Whatever does not fit on the desk falls off, usually from the oldest end.
The document. A contract of 7,500 words is about 7,500 ÷ 0.75 = 10,000 tokens. Add a 200-token system prompt.
The default is too small. The engine is set to 4,096 tokens. Only the last part fits; roughly the first 6,000 tokens never reach the model. It summarises the second half as if it were the whole contract.
Work out what you need. 10,000 (contract) + 200 (system) + 1,000 (room for the answer) = 11,200 tokens.
Pick a setting. The next common size above 11,200 is 16,384. Set num_ctx to 16384.
Check the memory. The KV cache grows with the context, roughly four times bigger at 16,384 than at 4,096. Use the calculator below: model + cache + 1 GB must fit (Lesson 4.1).
3. How to change it
- Ollama, whole server: set
OLLAMA_CONTEXT_LENGTH=16384before starting it, or use the context slider in the desktop app’s settings. - Ollama, one chat:
/set parameter num_ctx 16384. - Ollama, from code:
options={"num_ctx": 16384}. - Ollama, saved model:
PARAMETER num_ctx 16384in a Modelfile. - LM Studio: move the slider before loading.
4. Picking a length
- Estimate your longest input: words ÷ 0.75 = tokens (Lesson 3.1), plus room for the answer.
- Use the calculator to check the KV cache for that length.
- Check that model + cache + 1 GB fits (Lesson 4.1).
“The model card says 128K, so my model reads 128K.” The card gives the most the model can handle. Your engine decides how much it actually reserves, and it often starts much lower to save memory. Check the setting, not the card.
Maths and principle
Principle: reserve the context you need, not the most the model allows.
Equation:
Analogy: booking a hall: book for the guests you expect, not the building’s fire limit.
Try it: a 9,000-word report, a 1,000-token answer and a 200-token system prompt? (9,000 ÷ 0.75 = 12,000 + 1,000 + 200 = 13,200 tokens, so 16,384 is a good setting.)
- context length (num_ctx)
- the number of tokens the engine reserves room for: prompt, document, chat so far and the answer together
- words ÷ 0.75
- turns an English word count into an approximate token count
- answer tokens
- space you leave for the model to write its reply; it shares the same budget
- truncation
- what happens when the text is longer than the setting: part of it is cut and the model never sees it
- Physical meaning
- The context is the size of the model's desk. Make it big enough for today's papers plus a notepad for the answer, but every extra inch costs memory, and if the desk no longer fits in the room the model will not load or will crawl.
5. Worked example
Made-up numbers. Ritu’s summaries miss the first half of long contracts. Her Ollama default is 4,096 tokens; contracts run to 10,000. She sets num_ctx to 16,384 in her script. The KV cache adds about 1 GB for her model, which still fits on her 16 GB card.
6. Practice
- Find your engine’s current context length.
- Work out the tokens of your longest regular document.
- Set a context length with room to spare and test a long document.
Answers
- Ollama:
ollama psshows loaded models and their context in recent versions; LM Studio shows it on the model load panel. - Words ÷ 0.75.
- If the answer now mentions the start of the document, it worked.
New words: num_ctx, context slider, truncation (cutting off text that does not fit).
- The model card gives the most a model can read; the engine’s setting decides what it actually reads.
- Engines often start at a few thousand tokens, and longer input is cut off without warning.
- Context needed ≈ input words ÷ 0.75 + answer tokens + system prompt; round up to the next common size.
- In Ollama set
num_ctx(per chat, per request or in a Modelfile) orOLLAMA_CONTEXT_LENGTHfor the whole server. - A longer context means a bigger KV cache: check that model + cache + 1 GB still fits.
Check yourself
Sources
- Ollama default context by graphics memory (4,096 tokens under 24 GiB, 32,768 for 24 to 48 GiB, 256,000 at 48 GiB or more) and how to change it: Fast.io, last reviewed 31 October 2026, https://fast.io/resources/ollama-context-length/
- Ollama FAQ: context window size (official page), https://docs.ollama.com/faq
- llama.cpp server options (-c, --ctx-size), project documentation, https://github.com/ggml-org/llama.cpp/tree/master/tools/server
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Prices and model names change fast: check the official source before you buy.