Vector store and retrieval

What this session covers

Last week's search was a for loop over a Python list. It worked, and for a few dozen documents it'll keep working forever. Two things break as you scale, though. You recompute embeddings you already have, and every query rescans every document. A vector store fixes both. It's the loop given a home — an object you add documents to, embedding once and keeping the vector next to the text, then query by meaning. It also leaves room for the one piece a teaching example can't carry: an index that finds neighbors without scanning everything.

We build the smallest honest version, add and search, still a linear scan underneath but the exact interface a real database exposes. Then the part that actually matters: knowing where the toy ends. When you outgrow it you reach for pgvector, Qdrant, Pinecone, or something like them. They swap the scan for approximate-nearest-neighbor search and add persistence, and your code still calls add and search.

OpenAI

The store keeps a list of items, each carrying its text, optional metadata, and vector. add embeds a batch and appends. search embeds the query, optionally filters by metadata, and ranks the rest by cosine.

# A minimal vector store. add() embeds and remembers text + metadata + vector;
# search() ranks the store by cosine and returns the top k, optionally filtered
# by a metadata predicate first. A production database (pgvector, Pinecone,
# Qdrant) swaps the linear scan for an approximate-nearest-neighbor index — but
# the surface you code against is exactly add() and search().
class VectorStore:
    def __init__(self, embed_fn):
        self._embed = embed_fn
        self._items: list[dict] = []

    def add(self, docs: list[dict]) -> None:  # each doc: {id, text, meta?}
        vectors = self._embed([d["text"] for d in docs])
        for doc, vec in zip(docs, vectors):
            self._items.append({**doc, "vec": vec})

    def search(self, query: str, k: int = 3, where: dict | None = None) -> list[tuple]:
        qv = self._embed([query])[0]
        pool = [
            it for it in self._items
            if not where or all(it.get("meta", {}).get(f) == v for f, v in where.items())
        ]
        scored = sorted((( cosine(qv, it["vec"]), it) for it in pool), key=lambda t: t[0], reverse=True)
        return scored[:k]

Two design choices matter more than they look. First, the store takes an embed function instead of calling a provider itself, which is why the same class works for OpenAI and Gemini unchanged. Second, search filters by metadata before ranking. That filtering is what turns a similarity toy into something you'd actually ship: "find docs like this query, but only in the billing category, only from this customer, only since March." Structure you already know about shouldn't be left for the embeddings to rediscover the hard way.

Gemini

Same store, handed Gemini's embed function. Nothing else moves, which is the whole reason embed is a parameter and not a hardcoded call.

class VectorStore:
    def __init__(self, embed_fn):
        self._embed = embed_fn
        self._items: list[dict] = []

    def add(self, docs: list[dict]) -> None:
        vectors = self._embed([d["text"] for d in docs])
        for doc, vec in zip(docs, vectors):
            self._items.append({**doc, "vec": vec})

    def search(self, query: str, k: int = 3, where: dict | None = None) -> list[tuple]:
        qv = self._embed([query])[0]
        pool = [
            it for it in self._items
            if not where or all(it.get("meta", {}).get(f) == v for f, v in where.items())
        ]
        scored = sorted(((cosine(qv, it["vec"]), it) for it in pool), key=lambda t: t[0], reverse=True)
        return scored[:k]

The moment persistence and scale enter the picture, last week's caveat grows teeth. The vectors in your store are tied to the embedding model that produced them. Switch models and every stored vector is garbage; you re-embed the entire corpus and rebuild the index, no shortcuts. I treat the embedding model the way I treat a column type in a schema: it's part of the contract, and you version it.

Put it to work

Three docker-compose apps under code/showcase/<slug>/, each built on one shared store.py. Same drill: bash bootstrap-secrets.sh, docker compose up --build, http://localhost:3000. Each one hits the store from a different angle.

Showcase 1 — Document retriever

Index a chunked knowledge base, return the top passages for a question with their source ids. This is the retrieval step RAG is built on — finding the right context — with the generation left out on purpose so you can watch retrieval by itself. Next chapter feeds these passages to a model.

Showcase 2 — Metadata filter

The same store with topic tags. Prefix a query with billing: or account: and the store keeps only that topic before it ranks by meaning. This is the line between a real vector store and a similarity loop, and it's why "just cosine everything" stops being enough the moment you're in production.

Showcase 3 — Recommender

Read the store backwards: hand it an item, get its nearest neighbors. Describe an app and get similar ones, with the seed excluded. Content-based recommendation falls straight out of the same search call. No ratings, no user history, just meaning.

All three share store.py and differ only in what they put in it and how they query it. That's the lesson. The store is generic; the application is the corpus, the metadata, and the shape of the query.

Run it

The README in this folder has the Python version, the install line, the two environment variables, and the exact commands. Keys come from the untracked .env at the course root. The store is pure standard library, so the only things you install are the SDKs. And this blog runs its own retrieval on pgvector, if you want to see the production version of the same idea.

Takeaways

A vector store is add and search with the embeddings kept next to the text. Everything a hosted vector database piles on — an ANN index, persistence, metadata filters at scale — sits behind that same two-method interface, so the code you wrote against the toy is the code you keep. Two rules carry over from embeddings and bite harder here. The embedding model is part of your schema, so change it and you re-embed everything. And metadata filtering isn't optional polish; it's what makes similarity search usable on real data. You now have the piece RAG needs. Next week we bolt on the other half: hand the retrieved passages to a model and make it answer from them, honestly and with citations.