The short version

In under five years, the amount of text an AI model can read at once has gone from about 2,000 tokens to a million or two in models you can actually use today, with one lab claiming a hundred million in the lab. That looks like an exponential march towards infinity. It isn't, quite. There's a hard piece of maths underneath that pushes back, and the honest answer to "can context be truly unlimited?" is not in the way the marketing implies but yes, in a different sense, if we stop thinking about one giant window and start thinking about memory as a system.

This piece is the journey and the destination: how we got here, why it's hard, the tricks that got us this far, and where the field is actually heading. (If you want the deep-dive on what a million-token window costs you in practice, we covered that separately in Long Context Is Not Free: What Million-Token Windows Really Buy You. This piece is the longer arc and the forward look.)

The journey: from 2K to a hundred million

The numbers are genuinely staggering when you line them up.

• GPT-3 (2020): ~2,048 tokens. A couple of thousand words. Enough for a long email, not a document.
• GPT-4 (2023): 8K, with a 32K variant. A real jump, enough for a short report.
• Claude (2023): 100K, then 200K. Anthropic's Claude was the first frontier model to reach a 100K-token window in May 2023, and Claude 2.1 doubled that to 200K that November. Suddenly you could drop in whole books or financial filings.
• GPT-4 Turbo (late 2023): 128K. OpenAI's answer, roughly a 300-page book.
• Gemini 1.5 (2024): 1M, then 2M. Google's Gemini 1.5 Pro shipped a one-million-token window and later opened up two million to developers, all of it multimodal, so the budget could be a mix of text, images, audio and video.
• Llama 4 Scout (2025): 10M advertised. Meta's Llama 4 Scout advertised a ten-million-token window, the largest on any widely-listed model.
• Magic.dev's LTM-2 (2024): 100M, as a claim. Magic.dev said its LTM-2-mini ran a hundred-million-token context, equivalent in their words to ten million lines of code or 750 novels. Worth flagging clearly: this is the lab's own claim with limited independent verification, not a product most people have put through its paces.

Where does 2026 sit? Comfortably in the million-plus era. More than a dozen hosted frontier models now ship one-million-token windows or larger, including the latest Claude, Gemini and GPT generations, with the giant figures (10M, 100M) still living more in research and marketing than in daily use.

So the trend is real and dramatic. But a window you're advertised and a window you can trust are two different things, and the gap between them is the whole story.

Why it's hard: the maths fights back

Here's the engine of the problem. The transformer, the architecture behind almost every model you've heard of, works through a mechanism called self-attention. To process a sequence, every token looks at every other token to decide what's relevant. That "everything looks at everything" is what makes transformers so good, and it's also the trap.

If you have n tokens, the number of those pairwise comparisons grows as n squared. Double the input and the work roughly quadruples. This is O(n²) complexity, and it hits both compute (how much calculation) and memory (how much you have to hold). Go from 1,000 tokens to a million and the naive attention cost rises by a factor of a million squared over the original. That's why long context was, for years, simply unaffordable.

There's a second, sneakier bottleneck: the KV cache. As a model generates, it stores a "key" and "value" for every token it has seen so it doesn't recompute the past from scratch. That cache grows linearly with context length, and it lives in expensive, finite GPU memory. Magic.dev's own illustration is vivid: running a conventional 405-billion-parameter model with a 100M-token context would need hundreds of top-end GPUs per user just to hold the cache. The KV cache, not the raw compute, is often the wall you hit first.

So the difficulty isn't a lack of cleverness. It's that the default design makes long context quadratically expensive in compute and linearly hungry in memory at the same time.

The techniques that got us this far

The last few years have been a sustained engineering assault on that wall. The important honest point: most of these make long context feasible and cheaper, but several do not repeal the underlying complexity.

• FlashAttention (Dao et al., 2022). An IO-aware rewrite of attention that is smarter about moving data between fast and slow GPU memory. Crucially, it computes exact attention, the same answer, far faster and with less memory traffic. It made long context practical, but the quadratic compute is still quadratic; FlashAttention just stops you wasting memory bandwidth on top of it.
• Sparse and sliding-window attention (Longformer, 2020; used in Mistral 7B). Instead of every token attending to every other, each token attends only to a local window of nearby tokens. That turns the cost from quadratic into something closer to linear in the sequence length. The trade is that distant tokens connect only indirectly, layer by layer.
• Ring Attention (Liu et al., 2023). Shards a very long sequence across many devices arranged in a ring, overlapping communication with computation so memory per device stops being the ceiling. It enables training on sequences in the hundreds of millions of tokens without approximating attention, by spreading the load rather than shrinking it.
• RoPE and position interpolation (RoPE, Su et al.; Position Interpolation; YaRN). Rotary position embeddings encode where a token sits. Interpolation methods stretch a model trained on a short window to handle a longer one with only light fine-tuning, which is how many "extended context" models are made cheaply.
• Retrieval (RAG). The pragmatic alternative: don't cram everything into context at all. Store your documents externally, fetch only the relevant chunks at query time, and feed those in. It sidesteps the quadratic problem by keeping the live context small, at the cost of needing a good retrieval step.

Together these are why a million-token window went from impossible to ordinary. None of them, on their own, delivers free infinite context.

The sub-quadratic bet: Mamba, RWKV and the fixed-size state

If attention's quadratic cost is the enemy, the boldest answer is to not use standard attention at all.

State-space models like Mamba (Gu and Dao, 2023) and recurrent-style hybrids like RWKV take a different shape. Instead of letting every token see every other token, they march through the sequence and keep a fixed-size summary state of everything seen so far. Cost scales linearly with length, and at inference the memory stays roughly constant no matter how long the input gets. In principle, you can stream a near-unlimited number of tokens through them.

That sounds like the whole problem solved. Here's the catch, and it's a fundamental one. A fixed-size state is a finite state. Everything the model has read has to be compressed into that bounded summary. So these architectures trade perfect recall for unlimited length: detail from far back can be smoothed away or overwritten, like trying to remember a 600-page book from a single page of notes. Transformers, by contrast, keep every token addressable (that's the KV cache) and pay for it in cost.

This is the tension in a nutshell. Either you keep everything addressable and pay a growing price, or you cap the price and accept lossy memory. You don't get both for free. Hybrid designs that mix attention layers with state-space layers are an active, promising attempt to get the best of each.

The catch: effective context vs advertised context

Even inside a single transformer window, a big number on the box doesn't mean reliable use of all of it.

The landmark finding here is "Lost in the Middle" (Liu et al., 2023, published in TACL). Researchers found that models reliably use information at the start and end of a long context but get noticeably worse at retrieving facts buried in the middle, even models explicitly built for long context. Performance traced a U-shape: strong at the edges, sagging in the centre.

The popular test for this is the needle in a haystack eval: hide one specific fact in a huge body of text and ask the model to find it. Some models, including Llama 4 Scout, report near-perfect needle retrieval across the full window. But finding a single planted sentence is the easy version. Real work needs reasoning across many scattered facts at once, and that is where advertised windows and real-world reliability tend to part company. A 1M-token window does not guarantee trustworthy use of all 1M tokens; it guarantees you're allowed to put them in.

This is the reason "effective context" has become its own field of study, and the reason we keep saying: treat the headline number as a ceiling, not a promise.

So, is truly unlimited context possible?

Here's the honest answer, and it's the spine of this whole piece.

There is a genuine, possibly fundamental, tension:

• Keep perfect recall (every token addressable, transformer-style) and you pay a cost that grows with length. That is not unlimited; at some point it's simply too expensive.
• Cap the cost (fixed-size state, Mamba/RWKV-style) and you get unlimited length but lossy memory. That is not total recall.

You can't escape both horns at once with a single trick. "Unlimited context" in the literal sense, an infinite window with perfect recall of every token at bounded cost, runs into this wall.

But here's the optimistic half. "Unlimited" in the sense that matters in practice, a system that can work over arbitrarily large bodies of information and bring the right piece to bear when needed, is very achievable, and largely already here. It just doesn't look like one enormous window. It looks like a hybrid: a generously long context for the active working set, retrieval to pull in relevant external knowledge on demand, agentic memory that reads and writes to a persistent store, and compression to keep the live state lean. Human memory works much the same way: we don't hold everything in active attention; we keep a small working memory and retrieve from a vast external store as needed.

Going forward: memory as a system, not a window

The most interesting shift in 2026 is one of framing. The race to ever-bigger window numbers is giving way to a smarter question: how do you give a model durable, reliable memory?

Watch four directions:

1. Sub-quadratic and hybrid architectures. Mamba/RWKV-style models and transformer hybrids that mix a few attention layers with cheap recurrent ones, chasing long-context ability without quadratic cost.
2. Hierarchical and agentic memory. Models that explicitly write notes to themselves and read them back, an external memory store the agent manages over time, rather than relying on one window holding everything.
3. Test-time KV-cache compression. Shrinking or pruning that growing cache on the fly, dropping or summarising tokens that no longer earn their place, to push effective context further without buying more hardware.
4. Better long-context evaluation. Honest benchmarks that test reasoning across scattered evidence, not just needle-finding, so the effective window stops hiding behind the advertised one.

The headline to take away: the question is moving from "how many tokens can it hold?" to "how well can it remember, retrieve and reason over what matters?" The first question has a ceiling set by maths. The second one doesn't, and that's where the real progress is. Truly unlimited context, in the only sense that's useful, is less a bigger box and more a better memory system, and we're building it now.