Home / Learn / Data & AI / Offline AI / Lesson 7.4
All lessons

Module 7: Getting and choosing models · Lesson 7.4 · about 20 min

Model cards, benchmarks and why leaderboards mislead

How to read a model card, what popular benchmarks measure, why leaderboard ranks can mislead, and how to test a model on your own work in an hour.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 7 · about 20 minutes · read first: 7.1 and 7.3. You’ll be able to choose between models with evidence from your own work.

A benchmark is someone else's exam; the only score that predicts how a model will do your job is its score on your own tasks.

1. Why this matters

Every week a new model claims to beat the rest. Leaderboards help you make a shortlist, but they cannot tell you which model is best for your documents, your language or your code. One hour of your own testing can.

2. Reading a model card

The model card is the model’s front page (Lesson 7.1). Look for:

  • Sizes and context length, and whether it reads images or audio.
  • Intended use and limits: what the makers say it is not for.
  • Licence (Lesson 7.2).
  • Benchmark table, and which version was tested (often the full-precision one, not the small copy you will run).
  • Recommended settings (temperature and so on, Lesson 10.1) and the chat template (Lesson 10.2).

3. What benchmarks measure

A benchmark is a fixed set of test questions with known answers. A leaderboard ranks models by their scores. Common kinds:

  • Knowledge tests (multiple-choice exams across subjects).
  • Hard science questions written by experts.
  • Maths contest problems.
  • Coding: writing functions, or fixing real bugs in real projects.
  • Crowd votes (LMArena): people compare two anonymous answers and pick the better one; a rating comes out.

4. Why leaderboards mislead

  1. Contamination: test questions leak onto the web and into training data. The model can then remember answers instead of working them out, like a student who saw the exam paper.
  2. Mismatch: an exam score says little about summarising your contracts in Hindi.
  3. Style over substance: crowd votes reward long, confident, well-formatted answers.
  4. Different settings: makers test with the best prompt, long thinking, or many tries.

5. Your own one-hour test

  1. Collect 20 real tasks (prompts and inputs) from your work. Keep them private.
  2. Write down what a good answer must contain for each.
  3. Run each model on all 20 with the same settings. Save the answers.
  4. Score without looking at which model wrote which (ask a colleague to shuffle them). This is blind scoring: you cannot favour a name you like.
  5. Pick the smallest model that scores well enough: it will be faster and cheaper (Lesson 4.2).

Watch it happen: a blind test of two models

Twenty tasks. Imran copies 20 real support emails, and for each writes one line: "a good reply must give the refund date and the form link".

Run both. Model A (top of a leaderboard) and model B (smaller) answer all 20 with the same settings. He saves the 40 replies.

Shuffle and hide names. A colleague labels the replies 1 to 40 in random order. Imran marks each one good or not, without knowing which model wrote it.

Unblind and count.

ModelGood repliesScore
A14 of 2070%
B16 of 2080%

Is the gap real? Two tasks out of 20 is close to the noise line (look for 4 or more). But B also writes twice as fast on his laptop, so B wins on evidence that matters to him.

Maths and principle

Principle: a benchmark predicts your results only as far as its tasks look like yours.

Equation: your score = correct answers on your tasks ÷ number of tasks. With 20 tasks, a difference of 1 or 2 is noise; look for 4 or more.

Analogy: a driving test passed in a quiet town says little about driving in Mumbai traffic. Test drive the route you will use.

Try it: model A gets 15 of 20 of your tasks right, model B 16 of 20. Is B clearly better? (No: one task is within the noise. Choose on speed, memory or licence, or test 20 more.)

Score
good answers ÷ all tasks, as a percentage
Tasks
real jobs from your own work, each with a short note of what a good answer must contain
Noise
the gap you could get by luck; with 20 tasks, a gap of 1 or 2 means little
Contamination
test questions that leaked into training, so the model remembers answers instead of working them out
Physical meaning
A leaderboard is a newspaper review of a restaurant; your own test is eating there. Only the second tells you if you like the food.

“The model at the top of the leaderboard is the best model for me.” It is the best at that leaderboard’s questions, with the maker’s best settings, often at full precision. Your work, your language and your 4-bit copy can give a different order.

6. Worked example

Made-up numbers. Imran runs his 20 support emails through two models. The leaderboard leader gets 14 right; a smaller model gets 16 right and writes twice as fast on his laptop. He picks the smaller one.

7. Practice

  1. Open the model card of a model you use. Which version were its scores measured on?
  2. Write five of your own test tasks today.
  3. Why keep your tasks private?
Answers
  1. Often the full-precision model with long thinking turned on.
  2. Real work beats invented puzzles.
  3. If they get published, future models might train on them and the test stops measuring skill.

New words: model card, benchmark, contamination, leaderboard, blind scoring.

  • Model cards list sizes, limits, licence, benchmark scores and recommended settings; check which version the scores are for.
  • Leaderboards mislead through contamination, mismatch with your work, style over substance and best-case settings.
  • Your own test: 20 private real tasks, the same settings, blind scoring.
  • Score = good answers ÷ tasks. With 20 tasks, a gap of 1 or 2 is noise.
  • Pick the smallest model that scores well enough.

Check yourself

1Why can a high benchmark score mislead?
2What is the most reliable test for your job?
3A model card says the scores are for the full 16-bit model. You will run a 3-bit copy. What should you expect?

Sources

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Prices and model names change fast: check the official source before you buy.