Dataset builder

Turn examples into a clean training file — one line as `user prompt => desired answer`, out comes valid JSONL (OpenAI chat format, or Gemini input/output pairs). No model call; this is pure data plumbing, the real bulk of fine-tuning work.

README Try it Source

Showcase — Dataset builder

Turn examples into a clean training file — one line as user prompt => desired answer, out comes valid JSONL (OpenAI chat format, or Gemini input/output pairs). No model call; this is pure data plumbing, the real bulk of fine-tuning work.

Run

bash bootstrap-secrets.sh              # reads ../../../../.env, writes secrets/
docker compose up --build              # default: PROVIDER=openai

Open http://localhost:3000. Gemini: PROVIDER=gemini docker compose up --build.

What's where

backend/ai_openai.py — emits OpenAI chat JSONL.
backend/ai_gemini.py — emits Gemini tuning JSONL (input/output).
frontend/app/page.tsx — examples in, formatted dataset out.

Stop

docker compose down

Run locally

Download the project as a ZIP and run it with Docker. Brings up a FastAPI backend + Next.js frontend on localhost:3000.

Download dataset-builder.zip

unzip dataset-builder.zip
cd dataset-builder
bash bootstrap-secrets.sh   # one-time: pulls API keys into ./secrets
docker compose up --build   # default provider: openai
# or:  PROVIDER=gemini docker compose up --build

Type some input, pick a provider, and run the same code shown in Source against the live API. Sign-in required.

Input

Provider

Output

The same modules the Run button hits. The whole project (frontend, Dockerfile, compose) is in the ZIP under README.

`backend/ai_openai.py`

"""Showcase 1 (OpenAI): build a fine-tune dataset.

The unglamorous 80% of fine-tuning: turning examples into a clean training file.
Paste one example per line as `user prompt => desired answer` and get valid
OpenAI chat JSONL back, with a count and a nudge if you're short on examples. No
model call — this is pure data plumbing, which is exactly the point.
"""
import json

_SEPARATORS = ("=>", "|", "\t")


def _split(line: str):
    for sep in _SEPARATORS:
        if sep in line:
            u, a = line.split(sep, 1)
            return u.strip(), a.strip()
    return None


def run(text: str) -> str:
    pairs = [p for p in (_split(l.strip()) for l in text.strip().splitlines() if l.strip()) if p]
    if not pairs:
        return "Enter one example per line as:  user prompt => desired answer"
    lines = [
        json.dumps({"messages": [
            {"role": "user", "content": u},
            {"role": "assistant", "content": a},
        ]})
        for u, a in pairs
    ]
    jsonl = "\n".join(lines)
    note = f"\n\n--- {len(pairs)} example(s), valid OpenAI chat JSONL ---"
    if len(pairs) < 10:
        note += "\n(real fine-tunes want 50+ examples; this validates the format)"
    return jsonl + note

`backend/ai_gemini.py`

"""Showcase 1 (Gemini): build a fine-tune dataset.

Same input, Gemini's tuning format — simple input/output pairs as JSONL rather
than full chat messages. Seeing both formats side by side is the lesson: the data
shape is provider-specific, the discipline (consistent examples) is not.
"""
import json

_SEPARATORS = ("=>", "|", "\t")


def _split(line: str):
    for sep in _SEPARATORS:
        if sep in line:
            u, a = line.split(sep, 1)
            return u.strip(), a.strip()
    return None


def run(text: str) -> str:
    pairs = [p for p in (_split(l.strip()) for l in text.strip().splitlines() if l.strip()) if p]
    if not pairs:
        return "Enter one example per line as:  user prompt => desired answer"
    lines = [json.dumps({"text_input": u, "output": a}) for u, a in pairs]
    jsonl = "\n".join(lines)
    note = f"\n\n--- {len(pairs)} example(s), Gemini tuning format ---"
    if len(pairs) < 10:
        note += "\n(real tunes want hundreds of examples; this validates the format)"
    return jsonl + note