Dedup finder

Paste a list — support tickets, survey answers, product names — and find the

README Try it Source

Showcase — Dedup finder

Paste a list — support tickets, survey answers, product names — and find the pairs that mean the same thing even when they're worded differently. Every line is embedded once, all pairs are compared by cosine similarity, and the closest ones surface at the top. Exact-match or fuzzy-string dedup can't see past the wording; embeddings compare meaning.

Run

bash bootstrap-secrets.sh              # reads ../../../../.env, writes secrets/
docker compose up --build              # default: PROVIDER=openai

Open http://localhost:3000.

To run against Gemini instead:

PROVIDER=gemini docker compose up --build

What's where

backend/ai_openai.py / backend/ai_gemini.py — embed every line, compare all pairs by cosine similarity, rank the closest.
backend/main.py — identical FastAPI loader; reads PROVIDER and dispatches.
frontend/app/page.tsx — a text box (one item per line) + ranked pairs.

All-pairs comparison is O(n²) — fine for a text box, not for a million rows. That's the problem a vector database solves, next chapter.

Stop

docker compose down

Run locally

Download the project as a ZIP and run it with Docker. Brings up a FastAPI backend + Next.js frontend on localhost:3000.

Download dedup-finder.zip

unzip dedup-finder.zip
cd dedup-finder
bash bootstrap-secrets.sh   # one-time: pulls API keys into ./secrets
docker compose up --build   # default provider: openai
# or:  PROVIDER=gemini docker compose up --build

Type some input, pick a provider, and run the same code shown in Source against the live API. Sign-in required.

Input

Provider

Output

The same modules the Run button hits. The whole project (frontend, Dockerfile, compose) is in the ZIP under README.

`backend/ai_openai.py`

"""Showcase 3 (OpenAI): near-duplicate detection with embeddings.

Paste a list of lines — support tickets, survey answers, product names — and
find the pairs that MEAN the same thing even when they're worded differently.
Embed every line once, compare all pairs by cosine similarity, and surface the
closest ones. Exact-match dedup can't do this; embeddings can.
"""
import math

from openai import OpenAI

_client = OpenAI()

_MODEL = "text-embedding-3-small"


def _embed(texts: list[str]) -> list[list[float]]:
    return [item.embedding for item in _client.embeddings.create(model=_MODEL, input=texts).data]


def _cosine(a: list[float], b: list[float]) -> float:
    dot = sum(x * y for x, y in zip(a, b))
    na = math.sqrt(sum(x * x for x in a))
    nb = math.sqrt(sum(y * y for y in b))
    return dot / (na * nb) if na and nb else 0.0


def run(text: str) -> str:
    items = [line.strip() for line in text.splitlines() if line.strip()]
    if len(items) < 2:
        return "Enter at least two lines (one item per line)."
    vecs = _embed(items)
    pairs = []
    for i in range(len(items)):
        for j in range(i + 1, len(items)):
            pairs.append((_cosine(vecs[i], vecs[j]), items[i], items[j]))
    pairs.sort(reverse=True)
    rows = [f"{score:.3f}  [{a}]  ≈  [{b}]" for score, a, b in pairs[:8]]
    return "Most similar pairs (likely duplicates at the top):\n\n" + "\n".join(rows)

`backend/ai_gemini.py`

"""Showcase 3 (Gemini): near-duplicate detection with embeddings.

Same all-pairs cosine comparison, Gemini's embedding model.
"""
import math
import os

from google import genai

_client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])

_MODEL = "gemini-embedding-001"


def _embed(texts: list[str]) -> list[list[float]]:
    return [e.values for e in _client.models.embed_content(model=_MODEL, contents=texts).embeddings]


def _cosine(a: list[float], b: list[float]) -> float:
    dot = sum(x * y for x, y in zip(a, b))
    na = math.sqrt(sum(x * x for x in a))
    nb = math.sqrt(sum(y * y for y in b))
    return dot / (na * nb) if na and nb else 0.0


def run(text: str) -> str:
    items = [line.strip() for line in text.splitlines() if line.strip()]
    if len(items) < 2:
        return "Enter at least two lines (one item per line)."
    vecs = _embed(items)
    pairs = []
    for i in range(len(items)):
        for j in range(i + 1, len(items)):
            pairs.append((_cosine(vecs[i], vecs[j]), items[i], items[j]))
    pairs.sort(reverse=True)
    rows = [f"{score:.3f}  [{a}]  ≈  [{b}]" for score, a, b in pairs[:8]]
    return "Most similar pairs (likely duplicates at the top):\n\n" + "\n".join(rows)