Prompt caching, cost and latency

What this session covers

Everything you've built works. Now the job is making it cheap enough and fast enough to ship. Three numbers decide that — tokens, dollars, milliseconds — and every response hands you all three if you bother to read them. The most useful habit I know in production LLM work is checking response.usage on every call. Do it and the monthly bill stops ambushing you, because you watched each call's cost land in real time.

Then two levers. Prompt caching takes a large, repeated prefix — a system prompt, a long document, a set of examples — and makes it nearly free after the first call. The provider keeps that prefix warm, the way Postgres keeps hot pages pinned in the buffer cache, and bills the cached portion at a fraction of the going rate. The second lever is model choice, a straight trade of quality for speed and cost: a small fast model handles most prompts well at a tenth of the price and a tenth of the wait, so the skill is spotting the minority of tasks that genuinely need the big model.

OpenAI

Read the numbers first. response.usage carries input and output token counts, and input_tokens_details.cached_tokens tells you how much of the input came from cache. Multiply by the rates and you have the cost.

# Read tokens off response.usage and price them. cached_tokens are input tokens
# served from cache — usually billed far cheaper, so a big cached prefix is
# nearly free to re-send.
def report(response, seconds: float) -> None:
    u = response.usage
    cached = getattr(getattr(u, "input_tokens_details", None), "cached_tokens", 0) or 0
    p_in, p_out = _PRICES[_MODEL]
    cost = (u.input_tokens * p_in + u.output_tokens * p_out) / 1_000_000
    print(f"  in={u.input_tokens} (cached {cached}) out={u.output_tokens} "
          f"cost=${cost:.6f} latency={seconds:.2f}s")

Caching kicks in automatically once a prompt is large enough. Send the same big prefix twice and the second call's cached_tokens jumps. The design rule that follows is simple: stable material goes at the front of the prompt, variable material at the back. A cache hit needs the prefix to match byte for byte — the same reason a query with inlined literals never reuses a cached plan while a parameterized one does. A fixed system prompt and a static document cache cleanly; a timestamp at the top busts the cache on every single call.

def ask(question: str):
    start = time.time()
    response = client.responses.create(
        model=_MODEL, instructions=CONTEXT,
        input=[{"role": "user", "content": question}],
    )
    report(response, time.time() - start)
    return response.output_text

Gemini

Same three numbers, this time from usage_metadata — prompt_token_count, candidates_token_count, and cached_content_token_count. The caching-capable models report cached tokens the same way.

def report(response, seconds: float) -> None:
    u = response.usage_metadata
    cached = getattr(u, "cached_content_token_count", 0) or 0
    out = u.candidates_token_count or 0
    p_in, p_out = _PRICES[_MODEL]
    cost = (u.prompt_token_count * p_in + out * p_out) / 1_000_000
    print(f"  in={u.prompt_token_count} (cached {cached}) out={out} "
          f"cost=${cost:.6f} latency={seconds:.2f}s")

One habit that pays for itself: log usage on every call in production, not just in the demo. Cost creeps in from places you won't guess — a retry loop, a prompt that quietly grew, a document that got bigger — and per-call usage logs turn "why did the bill double?" into a query instead of an investigation, the same way per-statement logging turns a mystery slowdown into a sorted list. The token count is ground truth. Everything else is an estimate.

Put it to work

Three docker-compose apps under code/showcase/<slug>/, each one putting a different number on screen. Same drill: bash bootstrap-secrets.sh, docker compose up --build, http://localhost:3000.

Showcase 1 — Cost meter

Every answer arrives with its input tokens, output tokens, dollar cost, and latency attached. Run a few prompts and watch output length drive the cost — the lever you control most directly is how much you ask the model to write.

Showcase 2 — Cache saver

A big fixed prefix sits in front of every question. Ask once, ask again, and the cached-token count leaps on the repeat. This is the lever that makes a long-context or heavy-system-prompt app economical: you pay for the fixed part once and coast on it after that.

Showcase 3 — Speed vs quality

The same prompt sent to a small fast model and a large one, latencies side by side. The fast model handles most prompts fine; the exercise is building a feel for the ones where the quality gap earns back the wait and the cost.

All three read the same usage fields you'll read in production. Keep the habit; the apps are just where you practice it.

Run it

The README in this folder lists the Python version, the install line, the two environment variables, and the exact commands. Keys come from the untracked .env at the course root. The prices baked into the code are illustrative — check current pricing before you trust any dollar figure.

Takeaways

Three numbers run production LLM work — tokens, cost, latency — and every response carries all three, so read usage out of habit and log it always. The token count is ground truth, and it turns bill surprises into queries. Two levers carry most of the savings. Prompt caching makes a repeated prefix nearly free, so keep stable material at the front and variable material at the back and the cache stays warm. Model choice trades quality for speed and money, so default to the small fast model and escalate only the tasks that measurably need more. Next week we make quality itself measurable: an evaluation harness, so "is the new prompt better?" has an answer instead of a vibe.