If you've ever had a long conversation with an AI and watched it forget something you told it near the start, you've bumped into the single most important number for understanding what these systems can and can't do: the context window. It's the AI's short-term memory — and, like yours, it's finite.

The model only sees what's in the window

Here's the thing most people don't realise: between one message and the next, a chatbot remembers nothing on its own. Everything it "knows" about your current conversation — your messages, its own replies, any document you pasted, the hidden system prompt that sets its behaviour — is fed back into it, every single turn, as one big block of text. That block is the context window. It is the entire world the model can see at the moment it answers. Nothing outside it exists as far as the model is concerned.

The window is measured in tokens — the word-fragments AI reads in, roughly three-quarters of a word each. Modern windows have grown startlingly fast: a few years ago 8,000 tokens was normal; today 128,000 is common and some models advertise a million or more — that top end is several novels' worth of text held in view at once.

Why it "forgets"

Once a conversation grows past the window's limit, something has to give. The oldest material falls out of view — dropped, or quietly summarised down — because there's simply no room for it. So when a long chat seems to forget what you said at the beginning, it usually hasn't decided to. That text is no longer in front of it. It's less like forgetting and more like a whiteboard that only holds so much before the earliest notes get wiped to make space.

This is also the place to clear up a common confusion. The context window is not the same as what the model learned in training. Its permanent knowledge — facts, language, skills — is baked into its weights and doesn't change from one chat to the next. The context window is the temporary working memory for this conversation only. Training knowledge is what it knows; the context window is what it's currently looking at.

Why not just make it infinite?

If a bigger window is better, why cap it at all? Cost. The machinery of attention, and the memory the model uses to track a conversation, both grow with the length of the context — which is exactly why long chats get slower and more expensive to run. A larger window means more memory, more compute, more money, and more latency per reply. That's why providers charge by the token and why every model has a ceiling. The context window is, in part, a budget.

Bigger isn't the same as "remembers everything perfectly"

And here's the caveat that the headline numbers hide. Even when information is inside the window, the model doesn't use all of it equally well. A well-documented effect researchers call "lost in the middle" shows that models tend to make better use of what's at the very start and the very end of a long context, and can genuinely overlook material buried in the middle. So a giant window is not a guarantee that the model has truly absorbed everything you put in it. Stuffing an entire 200-page document into the context and asking a question about page 90 is exactly the case most likely to disappoint.

That limitation is why so much practical AI engineering is really context management: retrieving only the handful of genuinely relevant passages rather than dumping everything in (the approach behind most "chat with your documents" tools), summarising older parts of a conversation to save room, and reusing cached chunks of text to cut cost. "Just paste the whole thing" is often not the best move — a smaller, sharper context frequently beats a huge, cluttered one.

The takeaway

The context window is three things at once: the size of the model's attention, its working memory for the conversation you're having, and a big part of the bill. It has grown enormously — from a few thousand tokens to a million-plus in a couple of years — and that genuinely matters. But bigger isn't free, and it isn't the same as flawless recall. When you understand the window, a lot of otherwise-baffling AI behaviour — the forgetting, the cost, the occasional blind spot in a long document — stops being mysterious and starts being predictable.