Showcase — Eval runner

A fixed test set with expected answers. Edit the system prompt; the runner scores every case and reports a pass rate. Change the instruction, watch the number move — eval-driven prompt tuning.

Run

bash bootstrap-secrets.sh              # reads ../../../../.env, writes secrets/
docker compose up --build              # default: PROVIDER=openai

Open http://localhost:3000. Gemini: PROVIDER=gemini docker compose up --build.

What's where

• backend/ai_openai.py / backend/ai_gemini.py — the cases, a contains-check, and the pass rate.
• frontend/app/page.tsx — system-prompt box, per-case results + score.

Stop

docker compose down

unzip eval-runner.zip
cd eval-runner
bash bootstrap-secrets.sh   # one-time: pulls API keys into ./secrets
docker compose up --build   # default provider: openai
# or:  PROVIDER=gemini docker compose up --build

The same modules the Run button hits. The whole project (frontend, Dockerfile, compose) is in the ZIP under README.

backend/ai_openai.py

"""Showcase 1 (OpenAI): an eval runner you steer with the system prompt.

A fixed test set with expected answers. You edit the system prompt; the runner
scores every case and reports a pass rate. This is eval-driven prompt tuning:
change the instruction, watch the number move, keep what wins. Try a vague prompt
versus a precise one and compare.
"""
from openai import OpenAI

_client = OpenAI()

_MODEL = "gpt-5.4-nano"

_CASES = [
    {"q": "What is the capital of France?", "expect": "Paris"},
    {"q": "What is the capital of Japan?", "expect": "Tokyo"},
    {"q": "What is the capital of Australia?", "expect": "Canberra"},
    {"q": "What is 2 + 2?", "expect": "4"},
    {"q": "Who wrote Romeo and Juliet?", "expect": "Shakespeare"},
]


def _ask(system: str, q: str) -> str:
    return _client.responses.create(model=_MODEL, instructions=system, input=[{"role": "user", "content": q}]).output_text


def run(system: str) -> str:
    sys_prompt = system.strip() or "Answer the question."
    rows, passed = [], 0
    for c in _CASES:
        out = _ask(sys_prompt, c["q"])
        ok = c["expect"].lower() in out.lower()
        passed += ok
        rows.append(f"[{'PASS' if ok else 'FAIL'}] {c['q']} -> {out[:35]!r} (want {c['expect']})")
    rate = passed / len(_CASES)
    return (f"system prompt tested:\n  {sys_prompt}\n\n" + "\n".join(rows)
            + f"\n\nscore: {passed}/{len(_CASES)} = {rate:.0%}")

backend/ai_gemini.py

"""Showcase 1 (Gemini): an eval runner you steer with the system prompt.

Same fixed test set and pass-rate scoring, on Gemini.
"""
import os

from google import genai
from google.genai import types

_client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])

_MODEL = "gemini-3.1-flash-lite"

_CASES = [
    {"q": "What is the capital of France?", "expect": "Paris"},
    {"q": "What is the capital of Japan?", "expect": "Tokyo"},
    {"q": "What is the capital of Australia?", "expect": "Canberra"},
    {"q": "What is 2 + 2?", "expect": "4"},
    {"q": "Who wrote Romeo and Juliet?", "expect": "Shakespeare"},
]


def _ask(system: str, q: str) -> str:
    r = _client.models.generate_content(
        model=_MODEL,
        contents=[types.Content(role="user", parts=[types.Part(text=q)])],
        config=types.GenerateContentConfig(system_instruction=system),
    )
    return r.text or ""


def run(system: str) -> str:
    sys_prompt = system.strip() or "Answer the question."
    rows, passed = [], 0
    for c in _CASES:
        out = _ask(sys_prompt, c["q"])
        ok = c["expect"].lower() in out.lower()
        passed += ok
        rows.append(f"[{'PASS' if ok else 'FAIL'}] {c['q']} -> {out[:35]!r} (want {c['expect']})")
    rate = passed / len(_CASES)
    return (f"system prompt tested:\n  {sys_prompt}\n\n" + "\n".join(rows)
            + f"\n\nscore: {passed}/{len(_CASES)} = {rate:.0%}")

Project files

• .gitignore
• README.es.md
• README.md
• backend/Dockerfile
• backend/ai_gemini.py
• backend/ai_openai.py
• backend/main.py
• backend/requirements.txt
• bootstrap-secrets.sh
• docker-compose.yml
• frontend/Dockerfile
• frontend/app/layout.tsx
• frontend/app/page.tsx
• frontend/next.config.ts
• frontend/package.json
• frontend/tsconfig.json