LLM judge

Paste a customer-support reply draft; a judge model grades it 1-5 on tone, clarity, and completeness with a reason per dimension. Narrow job, fixed rubric, structured output — what makes an LLM judge trustworthy.

README Try it Source

Showcase — LLM judge

Run

bash bootstrap-secrets.sh              # reads ../../../../.env, writes secrets/
docker compose up --build              # default: PROVIDER=openai

Open http://localhost:3000. Gemini: PROVIDER=gemini docker compose up --build.

What's where

backend/ai_openai.py / backend/ai_gemini.py — the grading rubric.
frontend/app/page.tsx — draft box, scored dimensions + verdict.

Stop

docker compose down

Run locally

Download the project as a ZIP and run it with Docker. Brings up a FastAPI backend + Next.js frontend on localhost:3000.

Download llm-judge.zip

unzip llm-judge.zip
cd llm-judge
bash bootstrap-secrets.sh   # one-time: pulls API keys into ./secrets
docker compose up --build   # default provider: openai
# or:  PROVIDER=gemini docker compose up --build

Type some input, pick a provider, and run the same code shown in Source against the live API. Sign-in required.

Input

Provider

Output

The same modules the Run button hits. The whole project (frontend, Dockerfile, compose) is in the ZIP under README.

`backend/ai_openai.py`

"""Showcase 2 (OpenAI): an LLM judge with a rubric.

When outputs are open-ended — a support reply, a summary, an explanation — no
code check scores them. Paste a customer-support reply draft and a judge model
grades it on tone, clarity, and completeness, with a reason per dimension. Narrow
job, fixed rubric, structured output: that's what makes an LLM judge trustworthy.
"""
from openai import OpenAI

_client = OpenAI()

_MODEL = "gpt-5.4-nano"

_RUBRIC = (
    "You grade customer-support reply drafts. Score the draft the user provides "
    "from 1-5 on each of: tone (warm, professional), clarity (easy to follow), and "
    "completeness (actually resolves the issue). Output one line per dimension as "
    "'tone: N - reason', then a final 'overall: N - one-line verdict'."
)


def run(text: str) -> str:
    draft = text.strip()
    if not draft:
        return "Paste a customer-support reply draft to grade."
    response = _client.responses.create(
        model=_MODEL, instructions=_RUBRIC,
        input=[{"role": "user", "content": draft}],
    )
    return response.output_text

`backend/ai_gemini.py`

"""Showcase 2 (Gemini): an LLM judge with a rubric.

Same rubric grading, on Gemini.
"""
import os

from google import genai
from google.genai import types

_client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])

_MODEL = "gemini-3.1-flash-lite"

_RUBRIC = (
    "You grade customer-support reply drafts. Score the draft the user provides "
    "from 1-5 on each of: tone (warm, professional), clarity (easy to follow), and "
    "completeness (actually resolves the issue). Output one line per dimension as "
    "'tone: N - reason', then a final 'overall: N - one-line verdict'."
)


def run(text: str) -> str:
    draft = text.strip()
    if not draft:
        return "Paste a customer-support reply draft to grade."
    response = _client.models.generate_content(
        model=_MODEL,
        contents=[types.Content(role="user", parts=[types.Part(text=draft)])],
        config=types.GenerateContentConfig(system_instruction=_RUBRIC),
    )
    return response.text or ""