Prompt caching, cost and latency

Chapter 16 · make it affordable and fast

The hour

• Three numbers: tokens, dollars, milliseconds
• Read them off every response — usage
• Caching: a repeated prefix is nearly free
• Model choice: small + fast is usually enough

Read the numbers

u = response.usage
cached = u.input_tokens_details.cached_tokens
cost = (u.input_tokens*p_in + u.output_tokens*p_out) / 1e6

Check usage on every call and the bill stops surprising you.

Prompt caching

Send the same big prefix twice → the second call's cached_tokens jumps, billed at a fraction of the rate.

Design rule: stable material at the FRONT, variable material at the BACK. A timestamp up top busts the cache every call.

Model choice

Small fast model: ~1/10 the price and the wait, good enough for most prompts.

The skill is spotting the minority of tasks that actually need the big model.

Put it to work — three apps

• Cost meter — tokens + $ + latency per answer
• Cache saver — watch cached tokens leap on repeat
• Speed vs quality — fast vs big model, timed

All read the same usage fields you'll read in prod.

Log it always

Costs creep in: a retry loop, a prompt that grew, a document that got bigger.

Per-call usage logs turn "why did the bill double?" from an investigation into a query.

Takeaway

Tokens, cost, latency — on every response. Read and log usage. Cache the stable prefix; default to the small model. Next: make quality itself measurable — an evaluation harness.