Streaming responses

Week 3 · Chapter 4 · Basic

The problem

• One-shot responses block until the whole answer is ready
• That works for one sentence
• For a paragraph, the reader watches nothing happen
• Same latency, terrible feel

What changes

• Server ships tokens as it generates them
• Client prints them as they arrive
• Same model, same prompt, same total latency
• Different felt latency — that's most of what users notice

OpenAI: one keyword

stream = client.responses.create(
    model="gpt-5.4-nano",
    input=prompt,
    stream=True,
)
for event in stream:
    if event.type == "response.output_text.delta":
        print(event.delta, end="", flush=True)

• stream=True switches the return type to typed events
• Filter on response.output_text.delta
• flush=True is non-negotiable; Python buffers stdout by default

Gemini: separate method

stream = client.models.generate_content_stream(
    model="gemini-3.1-flash-lite",
    contents=prompt,
)
for chunk in stream:
    if chunk.text:
        print(chunk.text, end="", flush=True)

• generate_content_stream is the streaming twin
• Each chunk carries a .text delta or empty
• Same UX, different SDK ergonomics

When streaming pays off

• Long output (paragraphs, translations, rewrites)
• Chat-like UX where waiting is the worst part
• Skip it for one-line answers — the overhead isn't worth it

Three showcases this week

• Live explainer — Q&A; watch the answer take shape
• Stream translate — paste text + target language; sentence by sentence
• Live rewrite — paste a draft; a tighter version streams in

Each one is its own docker-compose project under code/showcase/<slug>/.

The two-callable pattern

def run_stream(input: str) -> Iterator[str]:
    stream = client.responses.create(..., stream=True)
    for event in stream:
        if event.type == "response.output_text.delta":
            yield event.delta

def run(input: str) -> str:
    return "".join(run_stream(input))

• run_stream is the lesson
• run is a one-liner accumulator
• Local UI uses the stream; website try-it uses the final string

Backend has two endpoints

• POST /api/ai → final string (non-streaming)
• POST /api/ai/stream → SSE feed of {delta} events
• One shared dispatch, two surfaces

Takeaway

Streaming is one keyword argument away from a better-feeling app. The model never got faster. The experience of waiting did. Reach for it whenever the output is long enough that a reader would tab away.