Dynamic LLM Context Window Management for MLOps
Stuffing every retrieved chunk, chat turn and database dump into one prompt looks like the safe way to get accurate answers. It gets more expensive and often less accurate as the prompt grows. Context management means deciding what enters the window, what leaves it, and what the model should fetch later instead of carrying around. This guide covers the evidence on why long contexts degrade, a budget you can enforce, and the interactions with prompt caching that most pruning advice skips.
What the research says about long contexts
Liu et al.'s Lost in the Middle found that performance is often highest when relevant information sits at the beginning or end of the input and degrades significantly when it sits in the middle, even for explicitly long-context models. Take the finding at face value and do not overfit to it. Chroma's context rot report tested 18 models and found that performance degraded as input length grew even on deliberately simple tasks, while needle position made no notable difference in its setup. Two studies, two setups, two different position effects. Put the most important material at the start or end of the prompt as a cheap default, then check on your own task whether position matters before building machinery around it.
The Chroma report has a second finding that matters more for RAG. Topically related distractors hurt: even a single distractor lowered accuracy compared with the needle alone, and four lowered it further. The chunks your retriever returns at ranks 8 to 20 are exactly that kind of distractor, relevant-looking and wrong. On the LongMemEval benchmark, the same report found models did much better with a focused prompt of about 300 tokens than with the full prompt of about 113k tokens. So fill the window with the chunks that clear a relevance bar, and leave the rest of the budget empty. A larger window raises the ceiling and does nothing to make padding free.
Anthropic's context engineering guide puts the goal as finding the smallest set of high-signal tokens that makes the desired outcome likely, and treats context as a finite resource with diminishing returns.
Set a budget by section, not one limit
Divide the window into named sections with caps: system instructions, tool definitions, retrieved evidence, conversation history, tool results and an output reserve. Measure each against your real traffic, because the section that fills first is usually tool results or history, not retrieval. Stable material (instructions, tool schemas) goes first and changes rarely. Per-request material goes last. That ordering is also what makes caching work, as the next section explains.
Rolling windows fight prompt caching
The standard advice for chat history is a rolling window that drops the oldest turns. Anthropic's prompt caching documentation describes a hierarchy of tools, then system, then messages, where a change at one level invalidates that level and everything after it. Dropping the oldest message changes the start of the messages block, so the cached prefix no longer matches and each turn pays full input price for the whole history. Cached reads are billed at a fraction of the base input price, so a sliding window can cost more than the full history it replaced.
Prefer an append-only transcript that you compact in occasional steps. Anthropic describes two compaction forms: summarizing a conversation that nears the limit and restarting from the summary, and the lighter tool result clearing, which removes old raw tool outputs the agent no longer needs. Compacting rarely keeps the cache warm between compactions. When you do compact, put the summary at the start of a new prefix and expect one cache miss. Prompt caching also changes rate limits: on Anthropic's API, cache reads do not count toward input-tokens-per-minute limits for most models, so a stable prefix helps throughput as well as cost.
Fetch instead of carry
For agents, keep lightweight references in context (file paths, record IDs, saved queries) and load content when a step needs it. The Anthropic guide calls this just-in-time retrieval and also describes structured notes written outside the window and reloaded later for long tasks. A reference costs a few tokens and a loaded file costs thousands, so the agent that fetches carries a smaller window than the one that preloads.
Measure from the provider, and decide what happens at the limit
Estimating tokens with a local tokenizer drifts from the provider's count, and the count can move when you only change the model name. Anthropic's Opus 4.7 announcement says the new model uses an updated tokenizer and that "the same input can map to more tokens", roughly 1.0 to 1.35 times depending on content type. A budget of 150,000 tokens tuned on the old model therefore holds less text after an upgrade, and a stored per-document token count is stale. Read the prompt token count the provider returns on every call, trigger the forced final answer or compaction from that number, and recompute any cached counts when the model ID changes. Decide before launch what your system does at the budget. Options are a forced final answer, compaction, or a refusal that says what was cut. A silent truncation is the worst of these.
Pruning failures and fallbacks
Aggressive pruning fails on questions that need many sources. If the filter keeps five of fifty relevant documents, the answer will be confident and partial. Detect it instead of relying on a prompt line that asks the model to admit ignorance: track how many candidates cleared the relevance bar compared with how many were kept, and when most of them were cut, run a second pass (a map step over batches, or a larger budget on a more capable model) instead of answering from the first five. Log the discarded chunks, the final prompt and the response, so a reported hallucination can be traced to a retrieval miss, a pruning decision or a generation error.
Verify with a golden set
Build a set of queries that need facts from several sources, plus queries whose answer is absent. Run it at several budgets and pruning thresholds and record answer quality, citation coverage, tokens, cache-read share and latency. Include a long-history scenario to catch truncation that drops an instruction from turn two. Because the distractor result above is about near-misses, add queries where several documents are almost right and see whether a smaller context beats a larger one on your data. Re-run when the corpus or query mix shifts. Start by finding the endpoint that spends the most input tokens, put a section budget on it, and measure answer quality before and after.
References
- Liu et al., Lost in the Middle: How Language Models Use Long Contexts for the position effect in long inputs.
- Chroma, Context Rot for degradation with input length, distractor effects and the focused versus full prompt comparison.
- Anthropic, Effective context engineering for AI agents for compaction, tool result clearing and just-in-time retrieval.
- Anthropic prompt caching documentation for cache invalidation by prefix change.
- Anthropic, Introducing Claude Opus 4.7 for the updated tokenizer and its 1.0 to 1.35 times token increase.