llm/curriculum
Module 14 · ready

Evaluation

How do I tell if a model is actually good at my work?

methods and benchmarksyour own practical evals

Two different questions

Evaluation answers one of two questions, and mixing them up causes most confusion:

  1. Is this copy the same as the original? Fidelity: perplexity, KL divergence, top-1 agreement (module 07). That’s what ViC305’s numbers measure. They tell you a quant didn’t break the model, not whether the model is good.
  2. Is this model good at my work? Capability: benchmarks and your own task tests. That’s this module.

What benchmarks measure

type examples what passes
knowledge, multiple choice MMLU-Pro, GPQA Diamond picking the right letter
math AIME, MATH-500 the final number matches
code, single function HumanEval, LiveCodeBench hidden unit tests pass
real repo bug fixing SWE-bench Verified (500 human-checked GitHub issues) the repo’s own tests go from fail to pass
terminal agent tasks Terminal-Bench a checker script verifies the final state of the container
tool / function calling BFCL calls have the right function, arguments and format
chat preference LMArena humans vote between two anonymous answers

Agent benchmarks score the model plus its harness (scaffold): the prompt, the tools, the retry logic, the context management. The same model can score very differently in Terminus-2, Claude Code or OpenCode, so a score only means something with the harness named. A published example: the same 2.6B model solved 62% of tasks in Mini-SWE-Agent and 33% in Claude Code (module 11).

Contamination

If benchmark questions (or their answers) were in the training data, the score measures memory, not skill. Benchmarks are public and the web gets scraped, so this happens easily. Defences:

  • Fresh problems: LiveCodeBench only uses problems published after a model’s training cutoff (Jain et al., 2024).
  • Private or held-out sets that never get published.
  • Rephrased checks: a big drop on a reworded copy of a benchmark is a red flag.
  • Your own tasks: the strongest defence. Nobody trained on your pagoda prompt.

Your own practical evals

Your current method (one visual task like the voxel pagoda, one product task like the Apple-style todo app, then reviewing the whole session) is a legitimate eval design. It catches things benchmarks don’t: taste, finishing a multi-step build, recovering from errors. To make it comparable across models:

  • Fix everything except the model: same prompt, same harness version, same quant format, same sampling settings, same hardware.
  • Run each task several times (3+). One run is an anecdote.
  • Write the rubric before looking at results: does it build? does it run without console errors? does it match the spec? visual quality 1-5.
  • Log the session, not just the result: number of tool calls, failed calls (bad JSON, wrong tool name, wrong arguments), retries, loops, time to first token, tokens per second, total tokens.
  • Screenshots and video of the final build, so judgement can be checked later.
Why tool-call failures matter so much

Module 07’s quants had 96-99% top-1 agreement at 8-bit. Over a 40,000-token agent session, a 1% chance of a different token at each step means hundreds of different tokens. Most are harmless, but one wrong token inside a JSON tool call breaks the call. That’s why low-bit quants often look fine in chat and fall apart in long agent sessions, and why you count failed calls.

LLM-as-judge

Using a strong model to grade answers scales well but has known biases: it prefers longer answers, the first answer shown, and its own style (Zheng et al., 2023). Mitigate by grading against a written rubric, swapping answer order, using a different model family as judge, and spot-checking by hand. For visual work, a vision model judging screenshots against the rubric works, but always keep the screenshots for your own check.

Statistics: pass@k and error bars

Models are random (sampling), so one run per task hides a lot of luck. Two numbers matter:

  • pass@1: the chance a single attempt succeeds. The honest number for “will it work when I use it”.
  • pass@k: the chance at least one of k attempts succeeds. Useful when you can verify and retry automatically. It’s always ≥ pass@1, so be suspicious of comparing pass@5 against someone’s pass@1.

And benchmarks are small. A model at 50% on 89 tasks has a 95% interval of roughly ±10 points, so “47% vs 52%” is noise:

pass@k and error bars, computed livehow much luck is hiding in a benchmark score
Left: pass@k = 1 − C(n−c, k) / C(n, k), the unbiased estimator from the Codex paper (Chen et al., 2021): the chance that at least one of k attempts passes, estimated from n attempts with c passes. Right: a 95% interval for a pass rate over T independent tasks, using the normal approximation ±1.96·√(p(1−p)/T). Two models whose intervals overlap heavily are not clearly different.

Ways to get tighter numbers: more tasks, several runs per task and averaging them, and comparing two models on the same tasks (paired comparison: count the tasks where one passes and the other fails).

Where to go deeper