Receipt parser

Vision meets structured output. Paste a photo URL of a receipt or invoice and

README Try it Source

Showcase — Receipt parser

Vision meets structured output. Paste a photo URL of a receipt or invoice and get back clean JSON — merchant, date, line items, total — the kind of record you insert straight into a database. The model reads the pixels; the prompt pins the shape (Gemini also uses JSON response mode).

Run

bash bootstrap-secrets.sh              # reads ../../../../.env, writes secrets/
docker compose up --build              # default: PROVIDER=openai

Open http://localhost:3000. Gemini: PROVIDER=gemini docker compose up --build.

What's where

backend/ai_openai.py / backend/ai_gemini.py — a JSON-extraction prompt over the receipt image.
frontend/app/page.tsx — URL box + extracted JSON.

Stop

docker compose down

Run locally

Download the project as a ZIP and run it with Docker. Brings up a FastAPI backend + Next.js frontend on localhost:3000.

Download receipt-parser.zip

unzip receipt-parser.zip
cd receipt-parser
bash bootstrap-secrets.sh   # one-time: pulls API keys into ./secrets
docker compose up --build   # default provider: openai
# or:  PROVIDER=gemini docker compose up --build

Type some input, pick a provider, and run the same code shown in Source against the live API. Sign-in required.

Input

Provider

Output

The same modules the Run button hits. The whole project (frontend, Dockerfile, compose) is in the ZIP under README.

`backend/ai_openai.py`

"""Showcase 3 (OpenAI): reading a receipt into structured JSON.

Vision meets structured output (week 4). Paste a photo URL of a receipt or
invoice and get back clean JSON — merchant, date, line items, total — the kind
of thing you'd insert straight into a database. The model reads the pixels; the
prompt pins the shape.
"""
from openai import OpenAI

_client = OpenAI()

_MODEL = "gpt-5.4-nano"

_PROMPT = (
    "Read this receipt or invoice image and extract it as JSON with exactly these "
    "keys: merchant (string), date (string or null), items (list of objects with "
    "name and price), total (number or null). Output ONLY the JSON, no prose. Use "
    "null for anything you can't read."
)


def run(image_url: str) -> str:
    url = image_url.strip()
    if not url.startswith("http"):
        return "Paste a public image URL of a receipt or invoice (http/https)."
    response = _client.responses.create(
        model=_MODEL,
        input=[{
            "role": "user",
            "content": [
                {"type": "input_text", "text": _PROMPT},
                {"type": "input_image", "image_url": url},
            ],
        }],
    )
    return response.output_text

`backend/ai_gemini.py`

"""Showcase 3 (Gemini): reading a receipt into structured JSON.

Same extraction schema; Gemini takes the image as bytes and is asked for JSON.
"""
import os
import urllib.request

from google import genai
from google.genai import types

_client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])

_MODEL = "gemini-3.1-flash-lite"

_PROMPT = (
    "Read this receipt or invoice image and extract it as JSON with exactly these "
    "keys: merchant (string), date (string or null), items (list of objects with "
    "name and price), total (number or null). Output ONLY the JSON, no prose. Use "
    "null for anything you can't read."
)


def _mime(url: str) -> str:
    u = url.lower()
    if u.endswith(".png"):
        return "image/png"
    if u.endswith(".webp"):
        return "image/webp"
    return "image/jpeg"


def run(image_url: str) -> str:
    url = image_url.strip()
    if not url.startswith("http"):
        return "Paste a public image URL of a receipt or invoice (http/https)."
    image_bytes = urllib.request.urlopen(urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})).read()
    response = _client.models.generate_content(
        model=_MODEL,
        contents=[types.Part.from_bytes(data=image_bytes, mime_type=_mime(url)), _PROMPT],
        config=types.GenerateContentConfig(response_mime_type="application/json"),
    )
    return response.text or ""