Module 9: Installing your first local AI · Lesson 9.3 · about 25 min
llama.cpp: the engine underneath
Run a GGUF model file directly with llama.cpp, start its built-in web chat and server, and learn the settings that matter most.
Module 9 · about 25 minutes · read first: 9.2. You’ll be able to run a model file with llama.cpp, open its web chat, and set the three options that matter most.
1. Why this matters
Most desktop apps, and Ollama itself in parts, are built on llama.cpp, an open-source engine. Using it directly means no hidden defaults: you choose exactly how much goes on the graphics card, how long the context is and which port it serves on. It is also how you run models on a server with no screen.
If a desktop app is a car with an automatic gearbox, llama.cpp is the same engine with a manual one.
2. Install
- Windows:
winget install llama.cpp, or download a ready-made build from the project’s GitHub Releases page (pick the one for your graphics card: CUDA for NVIDIA, Vulkan for most others, CPU if unsure). - macOS:
brew install llama.cpp(with Homebrew installed). - Linux: a package from your distribution, a release download, or build it from source.
Check it: llama-cli --version.
3. Get a model file
llama.cpp runs GGUF files. Two ways to get one:
- Reuse a file you already downloaded in an app (Lesson 9.1 shows where the app keeps them).
- Let llama.cpp download one from Hugging Face with
-hf <publisher>/<model>-GGUF. Search Hugging Face for “gemma 4 gguf” and pick a repository from Google or a well-known publisher (Lesson 7.6 covers safe downloads). Copy its exact name.
4. Run it
Chat in the terminal:
llama-cli -m gemma-4-e4b-Q4_K_M.gguf -c 8192 -ngl 99
Start the server, which also gives you a chat page in the browser:
llama-server -m gemma-4-e4b-Q4_K_M.gguf -c 8192 -ngl 99 --port 8080
Then open http://localhost:8080 in your browser. The same address answers programs (Module 11). Your file name will differ: use the one you downloaded.
localhost means “this same computer”. The port (8080 here) is a number that picks which program on the computer answers, like a flat number in a building.
| Setting | What it does | Start with |
|---|---|---|
-m |
The model file | your GGUF file |
-c |
Context length in tokens | 8192 |
-ngl |
Layers on the graphics card | 99 (as many as fit) |
--port |
The address number for the server | 8080 |
-t |
Processor threads | leave out at first |
5. How -ngl splits the model
A model is a stack of layers, slices that each do part of the work, one after another. -ngl (number of GPU layers) says how many of them go on the graphics card. The rest stay with the processor.
Watch it happen: placing 40 layers on a 6 GB card
The file. An 8 GB GGUF with 40 layers. Each layer is about 8 ÷ 40 = 0.2 GB.
The card. 6 GB of graphics memory. Keep about 1 GB for the context (8,192 tokens), so 5 GB is left for layers.
Fill it. 5 ÷ 0.2 = 25 layers fit on the card. They are read at graphics-card speed.
The rest. The other 15 layers (3 GB) run on the processor, more slowly. Every token passes through all 40, so the slow ones hold things up (Lesson 9.2).
The setting. -ngl 25 does exactly this. -ngl 99 means "as many as fit", and llama.cpp works it out for you.
Maths and principle
Principle: a model is a stack of layers; each layer you move to the graphics card is read at graphics-card speed.
Equation: layers that fit ≈ (graphics memory − room for context) ÷ (file size ÷ number of layers).
Analogy: loading boxes into a fast van and a slow trailer. Every box in the van arrives sooner; you load the van first and only use the trailer for what is left.
Try it: an 8 GB file with 40 layers, 6 GB of graphics memory and 1 GB set aside for context? (Each layer is 0.2 GB; 5 ÷ 0.2 = 25 layers on the graphics card, 15 on the processor.)
- F
- the size of the GGUF file in GB
- N
- how many layers the model has (it is in the model card, and llama.cpp prints it when loading)
- V
- your graphics card's memory in GB
- R
- memory kept for the context's KV cache and the engine
- ⌊ ⌋
- round down: only whole layers can move
- Physical meaning
- The number of layers on the card is the number of boxes in the fast van. Every layer you can add there makes each token a little quicker; when they all fit, you get the card's full speed.
“-ngl 99 means the model needs 99 layers” or “it forces everything onto the card.” 99 is just a number bigger than most models’ layer count, so it means “put every layer you can on the card”. If they do not all fit, llama.cpp puts as many as it can and leaves the rest on the processor.
6. Worked example
Made-up numbers. Sam’s office has an old Linux server with a 12 GB graphics card and no monitor. He copies a 7.7 GB Gemma 4 12B file to it over the network, runs llama-server with -ngl 99 -c 8192 --port 8080, and opens http://<server-address>:8080 from his laptop. The whole office can now try it in a browser (Module 15 shows how to do that safely).
7. Practice
- Install llama.cpp and run
llama-cli --version. - Start
llama-serverwith a model you have and open the web chat. - Halve
-cand compare the memory used in your system monitor.
Answers
- If Windows says the command is not found, open a new terminal after installing.
- If the page does not open, check the terminal for the line saying the server is listening, and the port.
- Memory use drops a little: the context cache shrinks. The model itself stays the same size.
New words: GGUF (the model file format llama.cpp uses), layer (one slice of the model’s stack), port (the number that picks which program on a machine answers), localhost (this same computer).
- llama.cpp is the open-source engine inside most local AI apps; it runs GGUF files.
llama-clichats in the terminal;llama-servergives a browser chat at localhost:8080 and an API for programs.- The three settings that matter:
-m(file),-c(context in tokens),-ngl(layers on the graphics card). - Layers that fit ≈ (card memory − context room) ÷ (file size ÷ layers), rounded down.
Check yourself
Sources
- llama.cpp (open source, MIT licence), https://github.com/ggml-org/llama.cpp
- llama.cpp server: built-in web chat and OpenAI-style API, project documentation (official page), https://github.com/ggml-org/llama.cpp/tree/master/tools/server
- llama.cpp build guide: CUDA, Vulkan, Metal and CPU builds (official page), https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Prices and model names change fast: check the official source before you buy.