Juez LLM

Pega un borrador de respuesta de soporte al cliente; un modelo juez lo califica del 1 al 5 en tono, claridad y completitud con una razón por dimensión. Trabajo estrecho, rúbrica fija, output estructurado — lo que vuelve confiable a un juez LLM.

README Pruébalo Código

Showcase — Juez LLM

Córrelo

bash bootstrap-secrets.sh              # reads ../../../../.env, writes secrets/
docker compose up --build              # default: PROVIDER=openai

Abre http://localhost:3000. Gemini: PROVIDER=gemini docker compose up --build.

Qué hay aquí

backend/ai_openai.py / backend/ai_gemini.py — la rúbrica de calificación.
frontend/app/page.tsx — caja de borrador, dimensiones calificadas + veredicto.

Detenlo

docker compose down

Ejecútalo en tu máquina

Descarga el proyecto como ZIP y córrelo con Docker. Levanta un backend FastAPI y un frontend Next.js en localhost:3000.

Descargar llm-judge.zip

unzip llm-judge.zip
cd llm-judge
bash bootstrap-secrets.sh   # one-time: pulls API keys into ./secrets
docker compose up --build   # default provider: openai
# or:  PROVIDER=gemini docker compose up --build

Escribe algo, elige un proveedor y ejecuta el mismo código de Código contra la API real. Requiere iniciar sesión.

Entrada

Proveedor

Output

Los mismos módulos que ejecuta el botón Run. El proyecto completo (frontend, Dockerfile, compose) está en el ZIP, pestaña README.

`backend/ai_openai.py`

"""Showcase 2 (OpenAI): an LLM judge with a rubric.

When outputs are open-ended — a support reply, a summary, an explanation — no
code check scores them. Paste a customer-support reply draft and a judge model
grades it on tone, clarity, and completeness, with a reason per dimension. Narrow
job, fixed rubric, structured output: that's what makes an LLM judge trustworthy.
"""
from openai import OpenAI

_client = OpenAI()

_MODEL = "gpt-5.4-nano"

_RUBRIC = (
    "You grade customer-support reply drafts. Score the draft the user provides "
    "from 1-5 on each of: tone (warm, professional), clarity (easy to follow), and "
    "completeness (actually resolves the issue). Output one line per dimension as "
    "'tone: N - reason', then a final 'overall: N - one-line verdict'."
)


def run(text: str) -> str:
    draft = text.strip()
    if not draft:
        return "Paste a customer-support reply draft to grade."
    response = _client.responses.create(
        model=_MODEL, instructions=_RUBRIC,
        input=[{"role": "user", "content": draft}],
    )
    return response.output_text

`backend/ai_gemini.py`

"""Showcase 2 (Gemini): an LLM judge with a rubric.

Same rubric grading, on Gemini.
"""
import os

from google import genai
from google.genai import types

_client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])

_MODEL = "gemini-3.1-flash-lite"

_RUBRIC = (
    "You grade customer-support reply drafts. Score the draft the user provides "
    "from 1-5 on each of: tone (warm, professional), clarity (easy to follow), and "
    "completeness (actually resolves the issue). Output one line per dimension as "
    "'tone: N - reason', then a final 'overall: N - one-line verdict'."
)


def run(text: str) -> str:
    draft = text.strip()
    if not draft:
        return "Paste a customer-support reply draft to grade."
    response = _client.models.generate_content(
        model=_MODEL,
        contents=[types.Content(role="user", parts=[types.Part(text=draft)])],
        config=types.GenerateContentConfig(system_instruction=_RUBRIC),
    )
    return response.text or ""