Long context and document Q&A

What this session covers

Three weeks ago you reached for RAG because a document wouldn't fit in a prompt. Modern context windows change that math. When a model swallows hundreds of thousands of tokens — and some take a million — plenty of documents just fit, and for those you can drop retrieval entirely. No chunking, no embeddings, no vector store, no gamble that the one paragraph you needed missed the top-k. You paste the whole thing in and ask.

Long context doesn't beat RAG. The question worth asking is which tool the situation calls for. Long context is simpler and can't miss a chunk, but it gets slower and more expensive the more you stuff in, and it won't scale to a corpus that changes constantly or dwarfs any window. Learning the capability takes a minute; the judgment about when to use it is the part that matters.

OpenAI

There's barely any new API here. The whole document goes into the instructions and the model answers from it. What changed is the scale of what you're allowed to put there.

# The entire document goes into the prompt as context. No retrieval step — the
# model reads all of it and answers from the relevant part.
def ask(document: str, question: str) -> str:
    response = client.responses.create(
        model=_MODEL,
        instructions=f"Answer the question using this document.\n\n{document}",
        input=[{"role": "user", "content": question}],
    )
    return response.output_text

The demo buries a specific fact — a calibration code — in one section and asks for it back. The model finds it because the whole manual is sitting right there. That's the appeal: with the entire document in context, there's no retrieval step to fumble. When it fails, the cause is a full window rather than a chunk that never got retrieved. That's a far easier failure to reason about.

Gemini

Same approach. Gemini's models carry especially large windows, which is exactly what whole-document work wants.

def ask(document: str, question: str) -> str:
    response = client.models.generate_content(
        model=_MODEL,
        contents=[types.Content(role="user", parts=[types.Part(text=question)])],
        config=types.GenerateContentConfig(
            system_instruction=f"Answer the question using this document.\n\n{document}",
        ),
    )
    return response.text or ""

Two things to keep in mind. First, "fits in the window" and "free" are not the same thing. You pay for every token in the prompt on every call, so re-sending a 100k-token document across twenty questions costs twenty times what retrieving the relevant 2k would. (Next week's prompt caching softens exactly this.) Second, models still lose things in the middle of a very long context; they're sharpest at the start and the end, so put the question and the material that matters where the model will actually look.

Put it to work

Three docker-compose apps under code/showcase/<slug>/. Same drill: bash bootstrap-secrets.sh, docker compose up --build, http://localhost:3000.

Showcase 1 — Whole-doc Q&A

Paste a document, put your question on a Q: line, and get an answer drawn from the full text — RAG minus the retrieval. Run it beside week 9's doc-qa and you'll feel the tradeoff: this one can't miss a chunk, but it re-reads the whole document every single time.

Showcase 2 — Long summarize

Paste an article or transcript, get a TL;DR and the key points. Summarization is where long context wins most easily — the task genuinely needs to see everything, and now it can, without the map-reduce gymnastics we used to bolt on.

Showcase 3 — Needle finder

A wall of release and ops notes sits in the prompt already; ask for one buried detail and it comes back exact. This is the "needle in a haystack" test the labs benchmark long-context models on. Try a few — the on-call name, the rate limit, a telemetry flag — and you'll see it doesn't matter where in the document the answer lives.

All three drop a whole document into context and ask a different kind of question of it. No embeddings, no store, just a big prompt and clear instructions.

Run it

The README in this folder lists the Python version, the install line, the two environment variables, and the exact commands. Keys come from the untracked .env at the course root.

Takeaways

Long context is the anti-RAG. When a document fits the window, put the whole thing in and skip retrieval — no chunking, no embeddings, nothing to miss. Reach for it when the source is one document that fits and doesn't change often; reach for RAG when the corpus is large, moves constantly, or runs far bigger than any window. And respect the two costs long context hides. You pay for every token on every call (caching helps, and that's next week), and models pay closest attention to the start and end of a huge prompt, so put the question and the material that matters there. Next week we make the money and the speed explicit: prompt caching, cost, and latency.