You have probably felt it without knowing the cause: the longer a chat with an AI goes on, or the bigger the document you paste in, the more sluggish and the more expensive it seems to get. That isn't your imagination, and it isn't really the model "thinking harder." It's a specific, unglamorous piece of plumbing called the KV cache — the trick that makes AI chat fast in the first place, and the reason long context is never quite free.

Why generation would be agonisingly slow without it

A language model writes one token at a time, and to choose each new token it uses attention: every new token looks back at every token that came before it and weighs how relevant each one is. That looking-back is the heart of how these models work — and, done naively, it is brutally wasteful.

Here's the waste. To produce the 1,000th token, the model needs certain intermediate values for all 999 earlier tokens. To produce the 1,001st, it needs them for all 1,000. If it recomputed those from scratch every single step, the work would pile up roughly with the square of the conversation length — generating a long answer would crawl.

The saving grace is that those intermediate values don't change. In the attention machinery, each token produces two vectors — a Key and a Value (the K and V) — and once a token is in the past, its K and V are fixed forever. So rather than recompute them every step, the model computes them once and stores them. That store is the KV cache. When the next token arrives, the model computes K and V for just that one new token and reuses the cache for everything before it. A step that would have scaled with the whole history now scales with a single token. That is, in large part, why a chatbot can stream a reply at a readable pace at all.

The catch: the cache grows, and you pay for it

Nothing is free, and here the price is memory. The KV cache holds a Key and a Value vector for every token, in every layer, for every attention head. It grows steadily as the conversation lengthens — and for long contexts it can swell to many gigabytes, often becoming the single biggest consumer of GPU memory during inference, larger even than the model's own weights.

That growing cache is exactly why long chats get slower and pricier, in two compounding ways:

• Memory. A bigger cache means each conversation occupies more of the GPU. Fewer conversations fit on one chip at once, so the cost of serving each one rises. This is a real part of why providers charge by the token and why very long contexts command premium pricing.
• Bandwidth. At every generation step, the model has to read that entire cache back out of memory. The longer the context, the more data gets shuttled around per token — and since this kind of generation step is bottlenecked by memory speed, not arithmetic, a fatter cache directly means slower output. (It's the same memory-bandwidth wall that speculative decoding is designed to work around.)

So the context window isn't just an arbitrary limit someone picked. It's partly a budget for how big this cache is allowed to get.

How engineers fight back

Because the KV cache is such a dominant cost, a lot of cleverness goes into taming it.

The most common move is to make the cache smaller per token. Techniques with names like multi-query and grouped-query attention let many attention heads share the same Keys and Values instead of each keeping its own, shrinking the cache several-fold — often four- to eightfold — with little quality loss — which is why nearly every modern model uses them.

A second move is to reuse it. Prompt caching — increasingly offered by the big AI providers — saves the KV cache for a shared chunk of text (a long system prompt, a document everyone asks about) so it doesn't have to be rebuilt for each request. This is precisely why "cached" input tokens are billed at a steep discount: the expensive part was already done.

And there are more: serving systems that manage cache memory in neat pages to avoid waste, compressing the cached values, or simply forgetting the oldest tokens once a window fills up.

The takeaway

The KV cache is one of those invisible mechanisms that quietly shapes the experience of using AI. It's the reason replies stream smoothly rather than grinding slower with every word — and, at the same time, the reason a very long conversation or a giant pasted document costs more and runs slower. When you feel that drag, you're feeling the cache fill up. Context, it turns out, has a weight — and the KV cache is where it's stored.