A/B de prompts

Pega dos system prompts separados por una línea con `===`. Ambos corren contra los mismos casos fijos y recibes cada tasa de aprobación más un ganador — el mismo conjunto de pruebas, dos variantes, una comparación.

README Pruébalo Código

Showcase — A/B de prompts

Pega dos system prompts separados por una línea con ===. Ambos corren contra los mismos casos fijos y recibes cada tasa de aprobación más un ganador — el mismo conjunto de pruebas, dos variantes, una comparación.

Córrelo

bash bootstrap-secrets.sh              # reads ../../../../.env, writes secrets/
docker compose up --build              # default: PROVIDER=openai

Abre http://localhost:3000. Gemini: PROVIDER=gemini docker compose up --build.

Qué hay aquí

backend/ai_openai.py / backend/ai_gemini.py — califican cada prompt sobre los casos compartidos.
frontend/app/page.tsx — caja de dos prompts, scores A vs B + ganador.

Detenlo

docker compose down

Ejecútalo en tu máquina

Descarga el proyecto como ZIP y córrelo con Docker. Levanta un backend FastAPI y un frontend Next.js en localhost:3000.

Descargar prompt-ab.zip

unzip prompt-ab.zip
cd prompt-ab
bash bootstrap-secrets.sh   # one-time: pulls API keys into ./secrets
docker compose up --build   # default provider: openai
# or:  PROVIDER=gemini docker compose up --build

Escribe algo, elige un proveedor y ejecuta el mismo código de Código contra la API real. Requiere iniciar sesión.

Entrada

Proveedor

Output

Los mismos módulos que ejecuta el botón Run. El proyecto completo (frontend, Dockerfile, compose) está en el ZIP, pestaña README.

`backend/ai_openai.py`

"""Showcase 3 (OpenAI): A/B two prompts on the same test set.

Paste two system prompts separated by a line with '==='. Both run against the
same fixed cases and you get each one's pass rate and a winner. This is how you
replace "the new prompt feels better" with a number — the same test set, two
variants, one comparison.
"""
from openai import OpenAI

_client = OpenAI()

_MODEL = "gpt-5.4-nano"

_CASES = [
    {"q": "What is the capital of France?", "expect": "Paris"},
    {"q": "What is the capital of Japan?", "expect": "Tokyo"},
    {"q": "What is the capital of Australia?", "expect": "Canberra"},
    {"q": "What is 2 + 2?", "expect": "4"},
    {"q": "Who wrote Romeo and Juliet?", "expect": "Shakespeare"},
]


def _score(system: str) -> int:
    passed = 0
    for c in _CASES:
        out = _client.responses.create(model=_MODEL, instructions=system, input=[{"role": "user", "content": c["q"]}]).output_text
        passed += c["expect"].lower() in out.lower()
    return passed


def run(text: str) -> str:
    parts = [p.strip() for p in text.split("===")]
    if len(parts) < 2 or not parts[0] or not parts[1]:
        return "Provide two system prompts separated by a line containing '==='."
    a, b = parts[0], parts[1]
    sa, sb, n = _score(a), _score(b), len(_CASES)
    winner = "A" if sa > sb else ("B" if sb > sa else "tie")
    return (f"A: {sa}/{n} = {sa/n:.0%}\n   {a}\n\n"
            f"B: {sb}/{n} = {sb/n:.0%}\n   {b}\n\n"
            f"winner: {winner}")

`backend/ai_gemini.py`

"""Showcase 3 (Gemini): A/B two prompts on the same test set.

Same two-prompt comparison over the fixed cases, on Gemini.
"""
import os

from google import genai
from google.genai import types

_client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])

_MODEL = "gemini-3.1-flash-lite"

_CASES = [
    {"q": "What is the capital of France?", "expect": "Paris"},
    {"q": "What is the capital of Japan?", "expect": "Tokyo"},
    {"q": "What is the capital of Australia?", "expect": "Canberra"},
    {"q": "What is 2 + 2?", "expect": "4"},
    {"q": "Who wrote Romeo and Juliet?", "expect": "Shakespeare"},
]


def _score(system: str) -> int:
    passed = 0
    for c in _CASES:
        r = _client.models.generate_content(
            model=_MODEL,
            contents=[types.Content(role="user", parts=[types.Part(text=c["q"])])],
            config=types.GenerateContentConfig(system_instruction=system),
        )
        passed += c["expect"].lower() in (r.text or "").lower()
    return passed


def run(text: str) -> str:
    parts = [p.strip() for p in text.split("===")]
    if len(parts) < 2 or not parts[0] or not parts[1]:
        return "Provide two system prompts separated by a line containing '==='."
    a, b = parts[0], parts[1]
    sa, sb, n = _score(a), _score(b), len(_CASES)
    winner = "A" if sa > sb else ("B" if sb > sa else "tie")
    return (f"A: {sa}/{n} = {sa/n:.0%}\n   {a}\n\n"
            f"B: {sb}/{n} = {sb/n:.0%}\n   {b}\n\n"
            f"winner: {winner}")