What is just-in-time retrieval?

Just-in-time retrieval is the context engineering technique in which an AI agent retrieves documents, data, or historical preferences only at the moment they are needed during a task, instead of loading everything up front. The prompt keeps only a lightweight reference, a file path, a stored query, a link, an ID, and the agent uses a tool to fetch the actual content only when the current stage of the task requires it. It is the opposite of full preloading: instead of dumping every potentially relevant document into context from the start, the agent reasons one step at a time and decides, step by step, what it actually needs. The pattern is described by Anthropic in the post "Effective context engineering for AI agents" (September 29, 2025), which frames it as the emerging strategy for real agents as work shifts from static retrieval systems to agentic loops that decide for themselves when and what to retrieve.

Why it matters: the cost of filling context upfront

Preloading everything feels safe, but it carries a direct cost: the more tokens enter the context, the higher the risk of context rot, the accuracy decline that shows up well before the window's stated limit. Just-in-time retrieval is one of the concrete mitigations for that problem, because it keeps the active context small and relevant to the current step instead of saturating it with material that will, maybe, be needed three steps later. There is also a direct economic cost: every token in the prompt is paid for on every model call, so preloading a thousand pages of documentation for a task that will use three of them means paying for nine hundred and ninety-seven useless pages, on top of risking diluted model attention on the ones that actually matter.

An enterprise example

An agent handling technical support tickets for a small or midsize business faces thousands of documents: product manuals, changelogs, past tickets, warranty policies. With just-in-time retrieval, the agent does not load that entire archive for every ticket. It instead keeps a lightweight index (paths, titles, IDs) and, when a ticket names a specific product model, retrieves only that model's manual and the past tickets tied to that customer ID. If the ticket turns out to be more complex and also needs the warranty policy, the agent retrieves it at that point, not before. Context grows only as much as needed, one step at a time, instead of starting out already full.

The difference from RAG and from an agent memory layer

RAG is the technical retrieval mechanism: embeddings, vector search, reranking. Just-in-time retrieval is the strategy that decides when to invoke that mechanism, and it can very well use RAG as one of the tools available to the agent, while remaining an orthogonal layer to it: RAG answers "how do I retrieve", just-in-time retrieval answers "when do I retrieve". The difference from the agent memory layer comes down to time scope: the memory layer is about what an agent remembers across different sessions, over time. Just-in-time retrieval is about what an agent retrieves inside a single session, while working on a task, regardless of whether that task takes two minutes or two hours.

Why it matters for decision makers

For anyone evaluating or building AI agents in an organization, just-in-time retrieval is the difference between an agent that holds up on long, complex tasks over large document bases and one that bogs down after a few steps. It cuts cost per request because it does not pay for unused tokens, cuts latency because the model processes less context on every turn, and cuts the risk of confused answers on multi-step tasks because the active context stays relevant to the current step. It is a concrete technical question to ask anyone proposing an agent over a large document archive: does it preload everything, or retrieve only when needed.