Building an evaluation harness

What this session covers

Every change you make to a prompt, a model, or a pipeline is a bet that it got better. With no way to measure, you're betting on a feeling — and a feeling is exactly what a confident, wrong answer is built to exploit. An evaluation harness turns "better" into a number: a fixed set of test cases, a scorer, and a pass rate you compare before and after. Once you have one, you stop arguing about whether the new prompt helped and just run it.

The harness itself is almost embarrassingly simple: cases, a check function, a loop, a count. The judgment lives in two places — writing test cases that include the traps that actually catch regressions, and picking the right scorer. Some outputs have a correct answer you can check in code; others are open-ended and need a model to score them against a rubric. You'll use both.

OpenAI

Start with the cheapest scorer there is: an exact-or-contains check against an expected answer. Stock the case list with the ones that catch regressions — the capital of Australia is Canberra, and a prompt that "improved" by growing more confident will cheerfully answer Sydney.

# The test set. A good one includes the obvious cases AND the traps — Canberra,
# not Sydney, is exactly the kind of case that catches a regression.
CASES = [
    {"q": "What is the capital of France?", "expect": "Paris"},
    {"q": "What is the capital of Japan?", "expect": "Tokyo"},
    {"q": "What is the capital of Australia?", "expect": "Canberra"},
    {"q": "What is 2 + 2?", "expect": "4"},
]

The harness runs each case, scores it, and reports a rate. That rate is the whole point — one comparable number that tells you whether a change helped.

# Run every case through the model, score with a check function, report a rate.
# This "exact/contains" check is the cheapest scorer; use it whenever the right
# answer is unambiguous. The score is the number you compare across changes.
def check(output: str, expect: str) -> bool:
    return expect.lower() in output.lower()


def run_eval(system: str) -> float:
    passed = 0
    for c in CASES:
        out = ask(system, c["q"])
        ok = check(out, c["expect"])
        passed += ok
        print(f"  [{'PASS' if ok else 'FAIL'}] {c['q']} -> {out[:30]!r} (want {c['expect']})")
    rate = passed / len(CASES)
    print(f"score: {passed}/{len(CASES)} = {rate:.0%}")
    return rate

When there's no fixed right answer — a summary, a tone, an explanation — code has nothing to compare against, so you promote a model to judge. Give that judge a narrow job, a clear rubric, and a structured output.

# When the answer isn't a fixed string — a summary, a tone, an explanation — a
# code check can't score it. Use an LLM as the judge, with a rubric and a scale.
# Keep the judge's job narrow and its output structured so the score is usable.
def judge(question: str, answer: str) -> str:
    return ask(
        "You are a strict grader. Score the ANSWER to the QUESTION from 1-5 on "
        "accuracy and clarity. Reply as 'score: N - reason'.",
        f"QUESTION: {question}\nANSWER: {answer}",
    )

Gemini

The harness is provider-agnostic — cases, check, rate — so the Gemini version changes only the ask call. That's the point of building it this way: one test set, run across providers, which is how you'd actually decide between OpenAI and Gemini instead of guessing.

def check(output: str, expect: str) -> bool:
    return expect.lower() in output.lower()


def run_eval(system: str) -> float:
    passed = 0
    for c in CASES:
        out = ask(system, c["q"])
        ok = check(out, c["expect"])
        passed += ok
        print(f"  [{'PASS' if ok else 'FAIL'}] {c['q']} -> {out[:30]!r} (want {c['expect']})")
    rate = passed / len(CASES)
    print(f"score: {passed}/{len(CASES)} = {rate:.0%}")
    return rate

A few honest cautions about LLM judges, because they're easy to reach for and easy to misuse. Judges are biased. They lean toward longer answers and toward their own style, and they grade soft. Pin one down: a specific rubric, a fixed scale, one dimension at a time, and spot-check it against human ratings before you trust a single score. A judge you haven't validated is just another opinion with a number stapled to it.

Put it to work

Three docker-compose apps under code/showcase/<slug>/, one idea each. Same drill: bash bootstrap-secrets.sh, docker compose up --build, http://localhost:3000.

Showcase 1 — Eval runner

A fixed test set and an editable system prompt. Change the prompt, run it, watch the pass rate move. This is prompt engineering done honestly: the question stops being "does this look better" and becomes "did the score go up."

Showcase 2 — LLM judge

Paste a support-reply draft and get it graded on tone, clarity, and completeness, with a reason for each. This is the scorer for everything a string match can't touch, and a worked example of keeping a judge narrow and structured.

Showcase 3 — Prompt A/B

Two system prompts, one test set, two pass rates, a winner. This is the artifact that ends the argument: paste the current prompt and the proposed one, and let the number settle it.

All three are the same three pieces — cases, a scorer, a rate — arranged three ways. Build one for anything you plan to ship.

Run it

The README in this folder lists the Python version, the install line, the two environment variables, and the exact commands. Keys come from the untracked .env at the course root.

Takeaways

An evaluation harness is what separates engineering from vibes: a fixed test set, a scorer, a pass rate you compare across every change. It's trivial to build — cases, a check, a loop — and the craft sits in the test cases (load them with the traps that catch regressions) and the scorer (code for exact answers, an LLM judge for the open-ended ones). Keep the judge under suspicion until you've validated it against human ratings, because a biased judge only launders a guess into a score. Build the harness before you need it and "is the new version better?" turns into a command you run instead of an argument you keep having. Next week the harness takes on its most important job: measuring safety, moderation and guardrails.