"Multimodal" is one of those AI words that gets used constantly and explained rarely. It sounds like a feature — the model does text and images now — but it points at something more structural: how a system takes in different kinds of information and, increasingly, handles them as one thing rather than as separate bolt-on skills.

Here is the plain version.

A "modality" is a type of input or output

Text, images, audio, video, and things like depth maps or sensor readings are each a modality — a distinct form the information comes in. A model that only reads and writes text is unimodal. A multimodal model works across more than one: it might look at a photo and answer questions about it, transcribe and reason about speech, describe a video, or generate an image from a written prompt.

The interesting part is not that it can touch several modalities. It is how.

The old way: separate models, stitched together

The first systems that appeared "multimodal" were often a pipeline of specialists. One model turned speech into text, that text went to a language model, and a third model turned the answer back into speech. Each step worked, but information leaked at every seam: tone of voice, a hesitation, the exact thing a person was pointing at in an image — all flattened into text before the next model saw it. The system could describe a picture, but it wasn't really reasoning over the picture. It was reasoning over a caption of it.

The new way: one shared representation

Modern multimodal models do something different. They convert each modality into the same internal form — a shared representation, sequences of numbers ("embeddings") that live in one common space. A patch of an image and a word of text both become vectors the model can attend to side by side, in the same reasoning process.

The workhorse for getting there is an encoder per modality: a vision encoder turns an image into tokens the model handles much like word-tokens; an audio encoder does the same for sound. A widely used training trick — contrastive learning, popularised by systems like CLIP — teaches the model to place an image and its matching description close together in that shared space, so "a photo of a dog" and an actual photo of a dog end up near each other. Once everything lives in one space, a single model can reason across it: it can weigh what's in the picture against what the question asked, because both are just tokens in the same stream.

That is why a good multimodal model can do things the pipeline never could — read a chart and factor the trend into its answer, notice that a caption contradicts the image, or follow a spoken instruction that refers to something on screen.

Why it matters beyond the demo

Two reasons this is more than a spec-sheet bullet.

It changes what the model can be wrong about — and right about. Grounding a model in images and audio, not just text scraped from the web, gives it a handle on the physical world that text alone withholds. It is a big part of why AI is moving off the screen and into robotics, weather, medical imaging and science.

It changes the risk surface too. More input types means more ways to confuse or attack a model — a malicious instruction hidden in an image, for instance, rather than in text. Multimodal capability and multimodal safety have to advance together.

Where it still falls short

Multimodal is not solved. Models still hallucinate about images — confidently describing objects that aren't there. Fine-grained visual detail (exact counts, precise spatial relationships, reading dense text in a photo) remains shaky. Video, which is images plus time (and usually sound), is harder again, and true real-time understanding of a live scene is still early. And "multimodal" on a release page can mean anything from deep native integration to a modality bolted on for the announcement — the word alone doesn't tell you which.

The short version: multimodal means a model that turns different kinds of input into one shared language it can reason over together, instead of passing captions between specialists. When it works, the model stops describing the world secondhand and starts working with it directly. When it doesn't, it does so with great confidence — which is exactly why knowing what the word actually means is worth the five minutes.