A large language model writes the way you might read a sentence aloud while covering it with your hand: one word at a time, each new word chosen only after the last one is fixed. This is what "autoregressive" means, and it is the single biggest reason a chatbot's reply streams onto your screen in a steady trickle rather than appearing all at once.

So here is a puzzle. If a model can only produce one token — roughly one word-piece — per pass through its enormous stack of weights, why doesn't a long answer take far longer than it does? Part of the answer, increasingly, is a technique called speculative decoding: a way of getting several tokens out for close to the price of one, without changing a single word of the final result.

The real bottleneck isn't thinking — it's fetching

To see why the trick works, you have to know what actually slows a model down when it generates text. It is not, mostly, the arithmetic. Producing one token requires reading the model's entire set of weights — tens or hundreds of billions of numbers — out of memory and into the chip that does the maths. For a single token, that is a huge amount of data movement to support a comparatively tiny amount of computation. The expensive chip spends most of its time waiting for memory, not calculating. Engineers call this being memory-bandwidth-bound.

The crucial consequence: checking many tokens at once is almost as cheap as checking one. The model can score a whole sequence of positions in a single parallel pass — that is exactly what happens during training. It is only generation that is stuck going one step at a time, because each new token depends on the one before it. Speculative decoding is built entirely around exploiting that gap.

A fast assistant guesses; the slow expert checks

The idea, introduced independently in 2022 and 2023 by research teams at Google and at DeepMind, is to pair the big, slow, accurate model with a small, fast, rougher one — a "draft" model.

It works in a loop:

1. Draft. The small model quickly guesses the next handful of tokens — say the next four or five. Because it is small, this is cheap.
2. Verify. The big model takes those guesses and checks them all in one parallel pass — the kind of pass it does cheaply. For each guessed position it asks: is this the token I would have produced here?
3. Accept or correct. It keeps every guess from the start that it agrees with. At the first one it would not have chosen, it throws away that guess and the rest, substitutes its own token, and the loop repeats.

If the draft model guessed well, the big model has just confirmed several tokens in the time it would normally take to produce one. If the draft guessed badly, almost nothing is lost — you still get at least one correct token from the verification pass, exactly as you would have without the trick. The draft is a bet that fails cheaply and pays well.

The part that matters: it doesn't cut corners

It is tempting to assume a shortcut like this trades quality for speed — that the small model's guesses must be "good enough" approximations. They are not. With the acceptance rule described in the original papers, the output is mathematically identical in distribution to what the big model would have produced on its own. The draft model only ever proposes; the big model is still the sole authority on every token that survives. Speculative decoding is, in its standard form, a pure latency win, not a fidelity compromise. That is what makes it unusual, and so widely adopted.

How big a win? In practice, reported speedups tend to land in the region of two to three times, though the exact figure depends heavily on how often the draft and the main model agree — predictable, boilerplate text accelerates more than surprising text.

Many drafts, one idea

The "small separate model" is only the first version. The same insight now comes in several shapes. Some systems let the big model draft for itself by bolting on extra lightweight prediction heads — approaches with names like Medusa and EAGLE — so there is no second model to host. Multi-token prediction, where a model is trained to propose several future tokens at once, is a close cousin; it is the lineage behind the efficiency gains DeepSeek reported in late June 2026 with its DSpark work, which leans on exactly this family of techniques. The common thread never changes: guess ahead cheaply, verify in parallel, keep only what the real model endorses.

The honest caveats

Speculative decoding is not free. You pay in memory and complexity — often you are now hosting two models, and the orchestration between them is fiddly. The benefit is conditional: when the draft model is poorly matched to the task and its guesses are frequently rejected, the speedup shrinks, and in the worst case the extra drafting work is close to wasted. And a caution worth flagging: the lossless guarantee holds for the exact acceptance rule in the research; some production systems deliberately relax that rule to squeeze out more speed, which can nudge the output away from the original distribution. When a vendor claims a speculative-decoding speedup, the fair question is whether it is the lossless kind or a relaxed one.

But the core idea is one of the most elegant in modern AI systems work — and a good reminder that a lot of the progress you feel as a user is not the model getting smarter, but the engineering around it getting cleverer about the one word at a time.