What is an AI model's context window?

The context window is a model's working memory, measured in tokens: the maximum amount of text, across prompt, attached documents and conversation history, that the model can "see" in a single request. What does not fit inside the window simply does not exist for the model at that moment: it is not ignored by choice, it is not there. This window's size has grown a lot in recent years, from a few thousand to hundreds of thousands of tokens, but a bigger window does not automatically mean better answers: filling it has a cost, in money and in latency, and models still tend to lose precision on information sitting in the middle of a very long context. For this reason the window should be treated as a resource to manage carefully, not a space to fill as much as possible.

Bigger is not automatically better

Context windows have grown enormously in recent years, from a few thousand to hundreds of thousands of tokens. But filling them is not free: more context means more cost per request (tokens are billed) and more response latency. There is also a documented phenomenon known as "lost in the middle": models tend to recall information at the start and end of a long context better, losing precision on what sits in the middle. A huge window full of irrelevant documents often produces worse answers than a small, well-curated one.

Why it matters in practice

It is why RAG stays relevant even with very large context windows: selecting the right chunks to place in context almost always beats "throwing everything in" and hoping the model finds what it needs. Designing what goes into the window, in what order and at what level of summarization, is the core of context engineering, and it should be managed as a scarce resource, not an infinite space to fill. An emerging approach like Recursive Language Models attacks the same constraint from another direction: instead of choosing what goes into the window, it treats context as an external environment to query with code, without having to ingest it whole.