The leap from text-only to multimodal models — systems that natively understand images, audio, and sometimes video alongside text — is one of the most visible capability shifts in AI. But the interesting part isn't just "now it can see." It's how it sees, and what that architecture enables and prevents.

Native vs. bolted-on

The old way to make a text model "see" was a pipeline: a separate image-captioning model turned a picture into text, then the text model read the caption. This works for simple cases but loses enormous information — a caption can't capture everything in an image, and the text model never actually engages with the visual content.

Native multimodal models work differently. They encode images (and audio) into the same representational space as text and process everything together. The model isn't reading a description of your chart; it's attending to the chart itself alongside your question. That's what enables genuine cross-modal reasoning — answering a question that requires jointly understanding a picture and a paragraph, rather than handling each separately and hoping they line up.

What it unlocks in practice

The practical workflows this opens are broad and genuinely useful:

Document understanding. Hand the model a scanned invoice, a form, or a slide and ask questions about it. It reads layout, not just linearised text, which matters for tables and structured documents.

Chart and diagram reasoning. Show it a graph and ask what the trend is, or hand it an architecture diagram and ask how data flows. It engages with the visual structure.

Screenshot understanding. Critical for agents — a model that can look at a screen and understand what's on it can operate software, which is the foundation of computer-use agents.

Voice interaction. Native audio understanding (and generation) makes natural spoken conversation possible without a brittle speech-to-text-to-speech chain, preserving tone and reducing latency.

Visual Q&A in the wild. Point a camera at something and ask about it — a plant, a appliance, a piece of equipment, a sign in another language.

Where it still falls short

Multimodal models are impressive but uneven, and knowing the weak spots prevents disappointment:

Fine detail. Models often miss small or subtle visual elements. They get the gist of an image reliably but can overlook a detail in a corner or misread a small label.

Precise spatial reasoning. "Is object A to the left of object B?" and counting many similar objects remain surprisingly error-prone. The models understand scenes holistically better than they reason about exact positions.

Text in images. Reading text embedded in an image (dense documents, small fonts, stylised text) is much better than it used to be but still less reliable than the models' confidence suggests. For high-stakes text extraction, dedicated OCR plus verification is still wise.

Hallucinated detail. Asked to describe an image, a model may confidently mention things that aren't there, especially when the question presupposes them. "What colour is the dog?" can produce a colour even when there's no dog.

Building with multimodal

A few practical principles:

• Don't assume the model saw everything. For critical detail, ask targeted questions rather than open ones, and consider showing higher-resolution crops of the important region.
• Verify extracted text and numbers. Treat visual extraction as a draft to confirm, not ground truth, where it matters.
• Lean on its strengths. It's excellent at holistic understanding, summarising visual content, and connecting an image to a question. It's weaker at precision tasks.
• Watch the cost. Images consume substantial tokens; high-resolution images more so. Multimodal requests are often pricier than they look.

Why it matters strategically

Multimodality isn't a feature; it's a precondition for whole categories of AI product. Agents that operate real software need to see screens. Assistants that help in the physical world need to see what the user sees. Document and data workflows need to read layout, not just text. The shift from text-only to natively multimodal models is what moves AI from "chatbot in a box" toward something that engages with the actual, messy, visual world — with all the capability and all the rough edges that implies.