RAG end to end

What this session covers

You have both halves now. Embeddings turn text into vectors; a vector store retrieves the passages closest to a question. Retrieval-augmented generation is the join. Retrieve the relevant chunks, augment the prompt by pasting them in as context, and generate an answer the model has to draw from that context — citing what it used and admitting when the answer isn't there. That's the whole technique. It's how you get a model to answer questions about your documents, your data, your product — knowledge it was never trained on — without fine-tuning and without it inventing things.

Three letters, three steps, and the third step is where the discipline lives. A model handed context will cheerfully ignore it and answer from memory unless you tell it not to. The prompt is where you enforce "only from the context," "cite your sources," and "say you don't know." Those three rules are the whole difference between a useful assistant and a confident liar.

OpenAI

Retrieval first: the vector search from last week, inlined so the whole loop fits in one file. Embed the knowledge base once, rank it against the question, take the top few.

# Embed the KB once, then for a question return the top-k chunks by cosine.
# This is the whole "retrieval" step — the vector store from last week, inlined.
KB_VECS = embed([c["text"] for c in KB])


def retrieve(question: str, k: int = 3) -> list[dict]:
    qv = embed([question])[0]
    scored = sorted(zip(KB, KB_VECS), key=lambda cv: cosine(qv, cv[1]), reverse=True)
    return [chunk for chunk, _ in scored[:k]]

Then augment and generate. The retrieved chunks become a numbered context block, and the instructions carry the weight: answer only from the context, cite the [id]s, refuse when the context comes up short.

# Augment + generate. The retrieved chunks become a numbered context block; the
# instructions force the model to answer ONLY from it, cite the chunk ids, and
# say so when the context doesn't cover the question. Grounding lives in the
# prompt, not in hope.
def answer(question: str) -> str:
    chunks = retrieve(question)
    context = "\n".join(f"[{c['id']}] {c['text']}" for c in chunks)
    response = client.responses.create(
        model=_CHAT_MODEL,
        instructions=(
            "Answer the question using ONLY the context below. Cite the sources "
            "you use by their [id]. If the context does not contain the answer, "
            "say 'I don't have that in the docs.' Do not use outside knowledge.\n\n"
            f"Context:\n{context}"
        ),
        input=[{"role": "user", "content": question}],
    )
    return response.output_text

The demo asks three questions on purpose. Two are answerable from the KB and get cited answers. The third, "what's your CEO's name?", isn't in the docs, and the instructions make the model say so instead of inventing a name. Strip that last rule out sometime and watch it guess. That's the failure mode RAG exists to prevent, and the prompt is the only thing standing between you and it.

Gemini

Identical pipeline on Gemini's models. Retrieval is byte-for-byte the same; only the embed call and the chat call change hands.

def answer(question: str) -> str:
    chunks = retrieve(question)
    context = "\n".join(f"[{c['id']}] {c['text']}" for c in chunks)
    response = client.models.generate_content(
        model=_CHAT_MODEL,
        contents=[types.Content(role="user", parts=[types.Part(text=question)])],
        config=types.GenerateContentConfig(
            system_instruction=(
                "Answer the question using ONLY the context below. Cite the sources "
                "you use by their [id]. If the context does not contain the answer, "
                "say 'I don't have that in the docs.' Do not use outside knowledge.\n\n"
                f"Context:\n{context}"
            ),
        ),
    )
    return response.text or ""

One thing worth naming out loud: RAG quality is retrieval quality. If the right chunk isn't in the top-k, no amount of prompting saves the answer, because the model can only work with what you hand it. When a RAG system answers wrong, look at what got retrieved before you blame the model. Nine times out of ten the chunk it needed never made it into the context.

Put it to work

Three docker-compose apps under code/showcase/<slug>/, each the same retrieve-augment-generate loop pointed at a different job. Same drill: bash bootstrap-secrets.sh, docker compose up --build, then http://localhost:3000.

Showcase 1 — Document Q&A

The canonical loop over a product's docs. Answers cite their chunk [id]s, and the retrieved ids sit right beneath the answer so you can check the citations against what was actually fetched. Traceable by construction, not by promise.

Showcase 2 — Grounded refuse

The same loop with a narrow HR-policy corpus and a hard refusal rule. In-scope questions get cited answers; out-of-scope ones ("what's the stock price?") get a flat "that isn't covered" instead of a hallucination. This is the behavior that makes RAG trustworthy, and in my experience it's the one people forget to test for.

Showcase 3 — Cited summary

RAG past plain question-answering: give it a topic, it retrieves several relevant notes and synthesizes a short summary that cites every claim. Retrieval plus synthesis, with sources attached. That's the shape of a research assistant.

All three retrieve, then ground, then generate, and all three make the model show its sources. The corpus and the instruction change from one to the next. The loop doesn't.

Run it

The README in this folder has the Python version, the install line, the two environment variables, and the exact commands. Keys come from the untracked .env at the course root. The retrieval is pure standard library; only the SDK calls need keys. This blog's own chat answers from its articles this exact way, running on pgvector.

Takeaways

RAG is retrieve, augment, generate, and the generate step is where you win or lose, because a model answers from memory unless the prompt forces it onto the context. Three rules are non-negotiable: answer only from what's retrieved, cite the sources, refuse when the context doesn't cover the question. And keep in mind that RAG's ceiling is retrieval's ceiling. A perfect prompt can't rescue a chunk that never got fetched, so when the answers are wrong, inspect the retrieval first. That closes the arc: embeddings, a store, and now grounded generation over your own knowledge. Next week we widen what the model can take in at all — vision and multimodal input, where the "document" is an image.