Corredor de evals

Un conjunto de pruebas fijo con respuestas esperadas. Edita el system prompt; el runner califica cada caso y reporta una tasa de aprobación. Cambia la instruction, observa moverse el número — tuning de prompts guiado por evals.

README Pruébalo Código

Showcase — Corredor de evals

Córrelo

bash bootstrap-secrets.sh              # reads ../../../../.env, writes secrets/
docker compose up --build              # default: PROVIDER=openai

Abre http://localhost:3000. Gemini: PROVIDER=gemini docker compose up --build.

Qué hay aquí

backend/ai_openai.py / backend/ai_gemini.py — los casos, un contains-check y la tasa de aprobación.
frontend/app/page.tsx — caja de system prompt, resultados por caso + score.

Detenlo

docker compose down

Ejecútalo en tu máquina

Descarga el proyecto como ZIP y córrelo con Docker. Levanta un backend FastAPI y un frontend Next.js en localhost:3000.

Descargar eval-runner.zip

unzip eval-runner.zip
cd eval-runner
bash bootstrap-secrets.sh   # one-time: pulls API keys into ./secrets
docker compose up --build   # default provider: openai
# or:  PROVIDER=gemini docker compose up --build

Escribe algo, elige un proveedor y ejecuta el mismo código de Código contra la API real. Requiere iniciar sesión.

Entrada

Proveedor

Output

Los mismos módulos que ejecuta el botón Run. El proyecto completo (frontend, Dockerfile, compose) está en el ZIP, pestaña README.

`backend/ai_openai.py`

"""Showcase 1 (OpenAI): an eval runner you steer with the system prompt.

A fixed test set with expected answers. You edit the system prompt; the runner
scores every case and reports a pass rate. This is eval-driven prompt tuning:
change the instruction, watch the number move, keep what wins. Try a vague prompt
versus a precise one and compare.
"""
from openai import OpenAI

_client = OpenAI()

_MODEL = "gpt-5.4-nano"

_CASES = [
    {"q": "What is the capital of France?", "expect": "Paris"},
    {"q": "What is the capital of Japan?", "expect": "Tokyo"},
    {"q": "What is the capital of Australia?", "expect": "Canberra"},
    {"q": "What is 2 + 2?", "expect": "4"},
    {"q": "Who wrote Romeo and Juliet?", "expect": "Shakespeare"},
]


def _ask(system: str, q: str) -> str:
    return _client.responses.create(model=_MODEL, instructions=system, input=[{"role": "user", "content": q}]).output_text


def run(system: str) -> str:
    sys_prompt = system.strip() or "Answer the question."
    rows, passed = [], 0
    for c in _CASES:
        out = _ask(sys_prompt, c["q"])
        ok = c["expect"].lower() in out.lower()
        passed += ok
        rows.append(f"[{'PASS' if ok else 'FAIL'}] {c['q']} -> {out[:35]!r} (want {c['expect']})")
    rate = passed / len(_CASES)
    return (f"system prompt tested:\n  {sys_prompt}\n\n" + "\n".join(rows)
            + f"\n\nscore: {passed}/{len(_CASES)} = {rate:.0%}")

`backend/ai_gemini.py`

"""Showcase 1 (Gemini): an eval runner you steer with the system prompt.

Same fixed test set and pass-rate scoring, on Gemini.
"""
import os

from google import genai
from google.genai import types

_client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])

_MODEL = "gemini-3.1-flash-lite"

_CASES = [
    {"q": "What is the capital of France?", "expect": "Paris"},
    {"q": "What is the capital of Japan?", "expect": "Tokyo"},
    {"q": "What is the capital of Australia?", "expect": "Canberra"},
    {"q": "What is 2 + 2?", "expect": "4"},
    {"q": "Who wrote Romeo and Juliet?", "expect": "Shakespeare"},
]


def _ask(system: str, q: str) -> str:
    r = _client.models.generate_content(
        model=_MODEL,
        contents=[types.Content(role="user", parts=[types.Part(text=q)])],
        config=types.GenerateContentConfig(system_instruction=system),
    )
    return r.text or ""


def run(system: str) -> str:
    sys_prompt = system.strip() or "Answer the question."
    rows, passed = [], 0
    for c in _CASES:
        out = _ask(sys_prompt, c["q"])
        ok = c["expect"].lower() in out.lower()
        passed += ok
        rows.append(f"[{'PASS' if ok else 'FAIL'}] {c['q']} -> {out[:35]!r} (want {c['expect']})")
    rate = passed / len(_CASES)
    return (f"system prompt tested:\n  {sys_prompt}\n\n" + "\n".join(rows)
            + f"\n\nscore: {passed}/{len(_CASES)} = {rate:.0%}")