Long context and document Q&A

Chapter 15 · when the whole document just fits

The hour

Windows of 100k–1M tokens change the math
Document fits? Skip retrieval — paste it all
Long context vs RAG: know which tool
The hidden costs: tokens, and the middle

The anti-RAG

RAG existed because documents didn't fit. Now many do.

No chunking
No embeddings
No "the chunk I needed wasn't retrieved"

Barely any new API

def ask(document, question):
    r = client.responses.create(
        model="gpt-5.4-nano",
        instructions=f"Answer using this document.\n\n{document}",
        input=[{"role": "user", "content": question}])
    return r.output_text

What's new is the scale of what goes in instructions.

Long context vs RAG

Long context	RAG
Fits, can't miss	Scales past any window
Re-reads all each call	Fetches only what's needed
Single, stable doc	Large, changing corpus

Not "better" — situational.

Two hidden costs

You pay for every token, every call — 20 questions = 20× the document (caching helps, next week)
Models attend best to the start and end — put the question + key material there

Put it to work — three apps

Whole-doc Q&A — paste doc + Q:, answer from all of it
Long summarize — TL;DR + key points, no map-reduce
Needle finder — one buried fact from a wall of text

Takeaway

Fits the window? Paste it and skip retrieval. Big or changing corpus? RAG. Mind the token bill and the lost middle. Next: prompt caching, cost, and latency — the money made explicit.