Prompt A/B

Paste two system prompts separated by a line with `===`. Both run against the same fixed cases and you get each pass rate plus a winner — the same test set, two variants, one comparison.

README Try it Source

Showcase — Prompt A/B

Paste two system prompts separated by a line with ===. Both run against the same fixed cases and you get each pass rate plus a winner — the same test set, two variants, one comparison.

Run

bash bootstrap-secrets.sh              # reads ../../../../.env, writes secrets/
docker compose up --build              # default: PROVIDER=openai

Open http://localhost:3000. Gemini: PROVIDER=gemini docker compose up --build.

What's where

backend/ai_openai.py / backend/ai_gemini.py — score each prompt on the shared cases.
frontend/app/page.tsx — two-prompt box, A vs B scores + winner.

Stop

docker compose down

Run locally

Download the project as a ZIP and run it with Docker. Brings up a FastAPI backend + Next.js frontend on localhost:3000.

Download prompt-ab.zip

unzip prompt-ab.zip
cd prompt-ab
bash bootstrap-secrets.sh   # one-time: pulls API keys into ./secrets
docker compose up --build   # default provider: openai
# or:  PROVIDER=gemini docker compose up --build

Type some input, pick a provider, and run the same code shown in Source against the live API. Sign-in required.

Input

Provider

Output

The same modules the Run button hits. The whole project (frontend, Dockerfile, compose) is in the ZIP under README.

`backend/ai_openai.py`

"""Showcase 3 (OpenAI): A/B two prompts on the same test set.

Paste two system prompts separated by a line with '==='. Both run against the
same fixed cases and you get each one's pass rate and a winner. This is how you
replace "the new prompt feels better" with a number — the same test set, two
variants, one comparison.
"""
from openai import OpenAI

_client = OpenAI()

_MODEL = "gpt-5.4-nano"

_CASES = [
    {"q": "What is the capital of France?", "expect": "Paris"},
    {"q": "What is the capital of Japan?", "expect": "Tokyo"},
    {"q": "What is the capital of Australia?", "expect": "Canberra"},
    {"q": "What is 2 + 2?", "expect": "4"},
    {"q": "Who wrote Romeo and Juliet?", "expect": "Shakespeare"},
]


def _score(system: str) -> int:
    passed = 0
    for c in _CASES:
        out = _client.responses.create(model=_MODEL, instructions=system, input=[{"role": "user", "content": c["q"]}]).output_text
        passed += c["expect"].lower() in out.lower()
    return passed


def run(text: str) -> str:
    parts = [p.strip() for p in text.split("===")]
    if len(parts) < 2 or not parts[0] or not parts[1]:
        return "Provide two system prompts separated by a line containing '==='."
    a, b = parts[0], parts[1]
    sa, sb, n = _score(a), _score(b), len(_CASES)
    winner = "A" if sa > sb else ("B" if sb > sa else "tie")
    return (f"A: {sa}/{n} = {sa/n:.0%}\n   {a}\n\n"
            f"B: {sb}/{n} = {sb/n:.0%}\n   {b}\n\n"
            f"winner: {winner}")

`backend/ai_gemini.py`

"""Showcase 3 (Gemini): A/B two prompts on the same test set.

Same two-prompt comparison over the fixed cases, on Gemini.
"""
import os

from google import genai
from google.genai import types

_client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])

_MODEL = "gemini-3.1-flash-lite"

_CASES = [
    {"q": "What is the capital of France?", "expect": "Paris"},
    {"q": "What is the capital of Japan?", "expect": "Tokyo"},
    {"q": "What is the capital of Australia?", "expect": "Canberra"},
    {"q": "What is 2 + 2?", "expect": "4"},
    {"q": "Who wrote Romeo and Juliet?", "expect": "Shakespeare"},
]


def _score(system: str) -> int:
    passed = 0
    for c in _CASES:
        r = _client.models.generate_content(
            model=_MODEL,
            contents=[types.Content(role="user", parts=[types.Part(text=c["q"])])],
            config=types.GenerateContentConfig(system_instruction=system),
        )
        passed += c["expect"].lower() in (r.text or "").lower()
    return passed


def run(text: str) -> str:
    parts = [p.strip() for p in text.split("===")]
    if len(parts) < 2 or not parts[0] or not parts[1]:
        return "Provide two system prompts separated by a line containing '==='."
    a, b = parts[0], parts[1]
    sa, sb, n = _score(a), _score(b), len(_CASES)
    winner = "A" if sa > sb else ("B" if sb > sa else "tie")
    return (f"A: {sa}/{n} = {sa/n:.0%}\n   {a}\n\n"
            f"B: {sb}/{n} = {sb/n:.0%}\n   {b}\n\n"
            f"winner: {winner}")