Vision and multimodal input

What this session covers

The models you've been feeding text can take images too. It's not a separate vision model bolted on. It's the same chat model, handed a message whose content is a list — some text parts, some image parts, read together in one pass. That one change opens up a lot of work you couldn't do before: caption an image, answer a question about a photo, turn a receipt or a chart or a screenshot into structured data. The interface hardly moves. What the model can see is what changes.

There's almost no new syntax this week. You build the multimodal message once, then point it at three different jobs. The only real wrinkle is plumbing: OpenAI takes an image URL directly, Gemini wants the raw bytes. Once that's sorted, the whole job is "add an image part and ask."

OpenAI

The message content becomes a list of parts. OpenAI fetches the image URL for you, so you hand it the URL straight through, next to the text.

# The message content is a list: a text part and an image part. OpenAI accepts
# an image URL directly (it fetches it), so no downloading on our side. Add more
# image parts for multi-image reasoning ("what changed between these two?").
def describe(image_url: str, question: str) -> str:
    response = client.responses.create(
        model=_MODEL,
        input=[{
            "role": "user",
            "content": [
                {"type": "input_text", "text": question},
                {"type": "input_image", "image_url": image_url},
            ],
        }],
    )
    return response.output_text

Add a second input_image part and you can ask "what changed between these two photos?" — the model sees both. The text part still steers everything. Send the same image with "caption this," then "read the price tag," then "is this safe to eat?" and you get answers that have nothing to do with each other. Vision doesn't replace prompting. It gives prompting more to work with.

Gemini

One wiring difference. Gemini wants the image as inline bytes rather than a URL, so you download it first and pass a bytes part. After that the shape is identical: image part, text part, one call.

# Download the image and pass it as an inline bytes part alongside the question.
# The contents list mixes a Part and a plain string — Gemini treats the string
# as a text part.
def describe(image_url: str, question: str) -> str:
    image_bytes = urllib.request.urlopen(urllib.request.Request(image_url, headers={"User-Agent": "Mozilla/5.0"})).read()
    response = client.models.generate_content(
        model=_MODEL,
        contents=[
            types.Part.from_bytes(data=image_bytes, mime_type=_mime(image_url)),
            question,
        ],
    )
    return response.text or ""

Two practical notes that save you a confused afternoon. Images cost tokens, and a high-resolution photo costs a lot of them, so downscale before sending when you don't need the detail. And these models read and describe well but get shaky on precise spatial questions like "is the cup exactly 3cm left of the plate?" Trust them on content. Verify them on geometry.

Put it to work

Three docker-compose apps under code/showcase/<slug>/. Each one takes a public image URL and returns text, so they run in the same text box as every other week. Same drill: bash bootstrap-secrets.sh, docker compose up --build, http://localhost:3000.

Showcase 1 — Image caption

A URL in, an alt-text caption and a short description out. It's the simplest multimodal call there is, and one I'd actually use: automatic alt text is the accessibility work most sites never get around to.

Showcase 2 — Visual Q&A

First line a URL, the rest a question. Ask how many people are in a photo, or what a sign says, or whether the sky is overcast. The image is the context and your question is the prompt. It's retrieval-augmented generation where the "document" happens to be a picture.

Showcase 3 — Receipt parser

Vision plus the structured output from week 4. A photo of a receipt comes back as clean JSON: merchant, date, line items, total. This is the one that pays for itself. Turning documents-as-images into database rows used to mean a bespoke OCR pipeline; now it takes a prompt.

All three build the same multimodal message. What changes is the instruction and how you handle the result — a caption, an answer, or structured JSON.

Run it

The README in this folder lists the Python version, the install line, the two environment variables, and the exact commands. Keys come from the untracked .env at the course root. Images get fetched with the standard library; only the SDK calls need keys.

Takeaways

Multimodal input is the same models with a fuller message: content becomes a list of text and image parts, read together. So everything you already know — prompting, structured output, grounding — still applies, now over pixels. Plumbing is the only real gotcha: URL for OpenAI, bytes for Gemini. Two habits keep you out of trouble. Downscale images you don't need at full resolution, because they cost real tokens. And trust vision on content while you verify it on geometry. Next week the model stops reading images and starts making them: image generation.