Building an evaluation harness

Chapter 17 · turn "better" into a number

The hour

• Cases + scorer + pass rate
• Code checks for exact answers
• LLM judge for open-ended ones
• Judges are biased — validate them

The whole harness

def run_eval(system):
    passed = 0
    for c in CASES:
        out = ask(system, c["q"])
        passed += c["expect"].lower() in out.lower()
    return passed / len(CASES)     # the number you compare

That rate is the point.

Write cases that catch regressions

{"q": "Capital of Australia?", "expect": "Canberra"}

Not Sydney. The traps are what a "more confident" prompt fails — include them on purpose.

When code can't score it

Summaries, tone, explanations → promote a model to judge:

"Score 1-5 on accuracy and clarity.
 Reply 'score: N - reason'."

Narrow job, fixed rubric, structured output.

Judges are biased

• Prefer longer answers and their own style
• Can be lenient

Pin them down, one dimension at a time, and spot-check against human ratings first.

Put it to work — three apps

• Eval runner — edit the prompt, watch the rate
• LLM judge — grade a draft on a rubric
• Prompt A/B — two prompts, one set, a winner

Same three pieces, arranged three ways.

Takeaway

Cases, a scorer, a pass rate — the line between engineering and vibes. Write the traps in; validate your judge. Build it before you need it. Next: the harness's most important job — safety.