Vision and multimodal input

Chapter 12 · the model can see now

The hour

• Same chat model, image + text in one message
• Content becomes a list of parts
• Caption, visual Q&A, read a receipt
• URL for OpenAI, bytes for Gemini

The one idea

Message content is a list:

[ text part, image part ]

Reasoned over in one pass. Everything you know about prompting still applies — now over pixels.

OpenAI — pass the URL

client.responses.create(model="gpt-5.4-nano", input=[{
  "role": "user",
  "content": [
    {"type": "input_text",  "text": question},
    {"type": "input_image", "image_url": url},
  ],
}])

OpenAI fetches the URL for you.

Gemini — pass the bytes

img = urllib.request.urlopen(url).read()
client.models.generate_content(
  model="gemini-3.1-flash-lite",
  contents=[
    types.Part.from_bytes(data=img, mime_type="image/jpeg"),
    question,
  ])

Put it to work — three apps

• Image caption — URL → alt text + description
• Visual Q&A — URL + question → grounded answer
• Receipt parser — receipt photo → JSON (vision + structured output)

Same multimodal message; instruction + output handling change.

Two practical notes

• Images cost tokens — downscale what you don't need in full res
• Strong at content, weaker at precise geometry — trust the description, verify the measurements

Takeaway

Multimodal input = the same models with a richer message. Prompting, structured output, grounding all still apply, over pixels. Next: the model stops reading images and starts making them.