Home / Learn / Data & AI / Offline AI / Lesson 15.2
All lessons

Module 15: Scaling up: from one laptop to a team server · Lesson 15.2 · about 25 min

Serving engines built for many users

Ollama, LM Studio, llama.cpp's server, vLLM and SGLang compared, with what each is good at and a starting command for each.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 15 · about 25 minutes · read first: 15.1 and 7.5. You’ll be able to choose a serving engine for your group and start it with one command.

The serving engine is the program that loads the model and answers requests; with many users it decides how many people one card can serve, and because they all speak the same API you can start simple and switch later.

1. Why this matters

The engine is the program that loads the model and answers requests. On a laptop, any engine will do. With many users, the engine decides how many people you can serve on the same hardware, sometimes by a factor of five or more.

Think of a restaurant. The model is the kitchen. The engine is the front-of-house team: how well they seat guests, share tables and keep orders moving decides how many people eat each evening, with the same kitchen.

2. The five engines, side by side

Engine Best for Runs on Model files Batching
LM Studio One person, or a few, with a friendly app Windows, Mac, Linux GGUF, MLX on Mac Basic
Ollama Simple shared server, small teams Windows, Mac, Linux GGUF (its own library) A few parallel requests
llama.cpp llama-server Full control, any hardware, small to medium teams Almost anything, including CPU only GGUF Continuous, set slots with -np
vLLM Many users on NVIDIA (and some AMD) cards Linux servers Hugging Face folders, AWQ, GPTQ, FP8 Continuous, PagedAttention
SGLang Many users, agents and chat with shared prompts Linux servers Hugging Face folders Continuous, shares repeated prompt starts

All five offer the OpenAI-style API from Lesson 11.1, so you can start simple and move up without rewriting anything.

3. Starting each one

The model names below are examples; use the ones you picked in Module 7.

# Ollama: allow 4 requests at once, listen on the network
OLLAMA_NUM_PARALLEL=4 OLLAMA_HOST=0.0.0.0 ollama serve

# llama.cpp: 4 slots sharing a 32,768-token context, metrics on
llama-server -m gemma-4-12b-Q4_K_M.gguf -c 32768 -np 4 --host 0.0.0.0 --port 8080 --metrics

# vLLM (Linux, NVIDIA): the model's name exactly as on its Hugging Face page, or a local folder
vllm serve <model-name-or-folder> --max-model-len 16384

# SGLang (Linux, NVIDIA)
python -m sglang.launch_server --model-path <model-name-or-folder> --port 30000

Each prints the address it is listening on. Point Open WebUI (Lesson 9.5) or your scripts at it.

4. Choosing

  • Up to about 10 light users, or a Mac: Ollama or LM Studio. Least setup, good enough.
  • 10 to 50 users, or agents that send many requests, on any hardware: llama-server with 4 to 16 slots.
  • 50 or more users, or heavy agents, on NVIDIA Linux servers: vLLM or SGLang. They are harder to set up (drivers, Python environments, matching CUDA versions) but serve several times more people from the same card.

The memory rule from Module 4 still holds: the model plus every user’s KV cache must fit. vLLM grabs most of the card’s memory at start (about 90% by default) and hands it out as KV cache pages; that is normal.

How many conversations fit on one card? Walk through it with the numbers from the box below:

The card. 32 GB of graphics memory.

Take off the model. The weights take 8 GB and the engine needs about 2 GB of working space: 32 − 8 − 2 = 22 GB left for conversations.

One conversation's memory. This model's KV cache uses 0.125 MB per token. A full 8,192-token chat needs 0.125 × 8,192 = 1,024 MB, which is 1 GB.

Divide. 22 GB ÷ 1 GB = 22 conversations at full length at the same moment.

Why paging helps. Most chats are short. vLLM's PagedAttention hands out memory in small pages only as a chat grows, so in practice many more than 22 people can be in a batch at once.

Maths and principle

Principle: memory left after the weights decides how many conversations fit at once.

Equation: conversations at once ≈ (card memory − weights − working space) ÷ (KV cache per token × tokens per conversation).

Analogy: a restaurant: the kitchen is the model, the tables are KV cache. A bigger kitchen does not seat more guests; more floor space does.

Try it: 32 GB card, 8 GB weights, 2 GB working space, 0.125 MB per token (Lesson 8.4’s 8B example), 8,192 tokens per chat? (22 GB ÷ (0.125 MB × 8,192 = 1 GB) = 22 conversations at full length.)

card memory
the graphics card's total memory in GB
weights
the model file loaded into that memory
working space
room the engine needs for its own sums, about 1 to 2 GB
KV cache per token
memory each conversation uses for every token it holds (Lesson 8.4)
tokens per conversation
how long each chat may grow (its context)
Physical meaning
The kitchen (weights) takes a fixed part of the building; the rest is tables (KV cache). A bigger kitchen does not seat more guests, more floor space does. Shorter chats, or a smaller quant, leave room for more tables.

“vLLM is the fastest, so it is always the right choice.” vLLM shines with many users on NVIDIA Linux servers. On a Mac, or for a handful of people, it is the wrong tool: harder to set up, and it does not target Apple chips. Start with Ollama or LM Studio, and move up only when the queue says so; your scripts will not need to change.

5. Worked example

Made-up numbers. A college computer lab has a Linux server with two 32 GB cards and 60 students. They first try Ollama: at the start of class, answers take a minute to begin. They switch to vLLM with the same 12B model and see the first token in under 2 seconds for most students, because vLLM keeps 30 or more conversations in one batch.

6. Practice

  1. Which engine would you pick for a 3-person office with a Mac Studio, and why?
  2. What does -np 4 with -c 16384 give each user in llama-server?
  3. Why can you change engines without changing your Python scripts?
Answers
  1. Ollama or LM Studio: few users, a Mac (vLLM targets Linux GPU servers), least setup.
  2. Four slots of 4,096 tokens each.
  3. They all serve the same OpenAI-style API; only the base address and the model name change (Lesson 11.4).

New words: serving engine, slot, PagedAttention, safetensors, AWQ and GPTQ (quantised formats used by vLLM).

  • LM Studio and Ollama suit one person or a small team; llama-server suits 10 to 50 users on any hardware; vLLM and SGLang suit many users on NVIDIA Linux servers.
  • All of them speak the OpenAI-style API, so you can switch engines without rewriting scripts.
  • Conversations at once ≈ (card memory − weights − working space) ÷ (KV cache per token × tokens per conversation).
  • vLLM loads Hugging Face folders (and AWQ, GPTQ, FP8); llama.cpp and Ollama load GGUF.
  • vLLM taking about 90% of the card at start is normal: it is reserving room for KV cache pages.

Check yourself

1You have one Mac mini and five colleagues. Which engine is the sensible first choice?
2What do all these engines have in common that makes switching between them easy?
3Which file format does vLLM usually load?

Sources

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Prices and model names change fast: check the official source before you buy.