Context Window Overflow and Silent Truncation in Agents

Context window overflow doesn't crash agents. Context window overflow doesn't crash agents, and that absence of a crash is the problem. When a prompt exceeds a model's limit, the most common outcome isn't an exception or a dropped connection: it's a fluent, confident, well-formatted answer built on evidence that never made it into the call. Evals miss this because they grade the output, not the input that produced it. Treating overflow as an observable, deterministic harness failure, rather than a fuzzy model-quality problem, is what actually lets an engineering team catch it before a customer does.

Engineers are trained to expect that dangerous failures announce themselves. A null pointer throws. A timeout returns a 400. A dropped connection appears in the logs within seconds. Context overflow breaks that pattern, and it breaks it in one of three ways depending on the provider: silent truncation, an explicit error, or degenerate output. Only the middle option behaves the way engineers expect failures to behave. The other two are worse, and silent truncation is the worst of all, because the model still returns clean, grammatically confident text. No stack trace. No empty string. No log line that says "evidence missing." What actually gets cut is usually the oldest turns or the middle of the prompt, which in practice means the system instructions, the earlier context, or the exact document the task was supposed to reference.

Picture a research-agent built on a ReAct loop that appends every tool output to the next prompt. Around step 80, the accumulated context crosses a 200K token ceiling. The catch handler logs "model error" and retries, three times, straight into the same overflow. Customers see "agent unavailable." Nothing about that log line points at prompt construction, so the postmortem takes a week to find what a token counter would have shown quickly. That asymmetry is the whole problem in miniature: a crash stops the run and tells you where it stopped. Silent truncation keeps the run alive and ships a worse answer, and the system consuming that answer has no way of knowing the evidence was incomplete.

Runaway cost is a related but distinct failure from overflow, and the two should be separated. Overflow is a structural, per-call question about what fits in a single context window. Runaway cost is unbounded token spend accumulating across many calls over time. They get conflated in postmortems because both involve "too many tokens," but the fix for one does nothing for the other. Capping per-call context doesn't stop a loop from calling the model 400 times, and rate-limiting the loop doesn't stop a single call from truncating the wrong document.

Why bigger context windows do not eliminate the problem

The obvious response is to point at window sizes and call the problem solved. Gemini 3 Pro supports up to 1 million tokens. Llama 4 Scout advertises up to 10 million. GPT-5.2 offers up to 400,000. Those numbers sound like they make overflow a non-issue, and that's exactly the mistake.

The arithmetic doesn't cooperate. System prompt overhead, RAG retrieval across dozens of chunks, conversation history that grows every single turn, and the completion budget itself all compete for the same finite space, and agent loops burn through it faster than a single-call chat application ever will, because agent loops keep appending. A million-token window fills up fast when every tool call's raw output gets tacked onto the next prompt without pruning.

Even before the hard limit hits, something researchers call context rot sets in: model accuracy degrades well before the window is technically full, which means the advertised limit and the effective limit are two different numbers. Liu and colleagues, in a 2024 study, found a U-shaped accuracy curve on multi-document QA and key-value retrieval tasks: models found information reliably at the start and end of a context window, but accuracy dropped by more than 30% when the relevant fact sat in the middle. That pattern held across six different model families, including GPT-3.5-Turbo, GPT-4, Claude 1.3, LongChat-13B, MPT-30B, and Cohere Command, which rules out the explanation that it's a quirk of one architecture.

There's a mechanical reason for this, not just an empirical one. Architectural properties of transformer attention tend to favor tokens near the start and end of a sequence, leaving tokens buried in the middle at a disadvantage before generation even starts.

The NoLiMa benchmark, published by Modarressi and colleagues in 2025, sharpened the picture further: at just 32K tokens, tested models showed sharp drops from their short-context baseline performance. GPT-4o's accuracy went from 99.3% down to 69.7% at that length, a drop that has nothing to do with the model running out of room and everything to do with it losing track of what's already there.

A study by Gamage quantified what that means for instructions specifically: a constraint issued at turn 3 that hasn't been touched again by turn 16 holds at roughly a 33% compliance rate, compared to substantially higher compliance for constraints issued and reinforced closer to the point of use. The context window is a fidelity-over-time constraint as well as a capacity constraint. It's a fidelity-over-time constraint, and the two require different fixes.

Given all of that, the practical response among teams that build agent harnesses is to distrust the advertised number. The advertised window is a marketing number. The effective window is the one that survives contact with a real retrieval task.

The four structural ways agents produce context-driven failures

Overflow is a family of four mechanically distinct failures. It's a family of four mechanically distinct failures, and each one needs its own instrumentation, because a fix aimed at one won't catch the others.

Tool output overflow happens when a tool call returns more data than the model can absorb, whether that's a server log dump, a database query result, or the full contents of a file, and the process doesn't crash. The scale of the problem here can be enormous: an IBM materials-science workflow from 2025 found that a traditional approach consumed a massive number of tokens and still failed, while the identical workflow rebuilt around memory pointers instead of raw output used 1,234 tokens and succeeded, a reduction north of 16,000x. For contrast, the loud version of this failure does exist: the loud version of this failure does exist when a provider enforces a hard limit and returns an explicit error code. That's the version everyone wishes for. The silent version looks different: 145KB of tool output gets injected into the conversation, the original question gets pushed out of the window entirely, and the agent answers a question it can no longer fully see, without ever flagging that anything is missing.

Workflow loops compound this problem structurally, because every step that appends full tool output to the next prompt makes the next step's overflow risk worse than the last one's. Step repetition is, in fact, the single most common failure category in the MAST taxonomy, showing up in 15.7% of traced failures, with reasoning-action mismatch close behind at 13.2% and failure to recognize termination conditions at 12.4%. The cost angle is real too: agent loops without cycle detection can burn unbounded cost over many calls, and per-call rate limits cannot stop failures that are structural in nature. Context exhaustion inside these loops appears as agents losing track of earlier requirements or contradicting themselves; one knowledge-graph pipeline deployment saw an agent carrying 180K tokens of accumulated context start hallucinating function signatures that didn't exist anywhere in the actual codebase.

Tool schema drift is a quieter failure. It happens when a tool's expected input shape changes, a renamed field, a new required property, a stricter validator, and nothing updates the agent's prompt or tool-calling code to match. A February 2026 bug report against n8n described exactly this: upgrading from version 2.4.7 to 2.6.3 caused the Vector Store Question Answer Tool to emit invalid JSON schemas for function calling, breaking both OpenAI and Anthropic integrations at the same time, with no mechanism in place to surface the schema change to anything consuming that tool. Schema drift rarely throws an error. It produces a malformed-but-plausible response that slides downstream until a human eventually notices the output is wrong, which is the same fluent-but-wrong shape as silent truncation, just triggered by a different mechanism. Related subtypes of schema drift include cases where an agent retries using the same failing input shape, calls a function that was never in the runtime's catalog, fabricates a plausible response when a tool returns an error, or continues passing tests against a stale mock while a third-party endpoint has already changed its schema in production.

Prompt specification failures round out the fourth category, and they're the largest by volume. The MAST taxonomy from UC Berkeley, published at NeurIPS 2025, analyzed 1,642 multi-agent execution traces across seven frameworks (MetaGPT, ChatDev, HyperAgent, OpenManus, AppWorld, Magentic, and AG2), built on annotation work involving 150 hand-labeled traces with an inter-annotator agreement of κ = 0.88, a strong number for this kind of qualitative coding. The results: specification failures account for 41.77% of traced failures, coordination failures 36.94%, and verification gaps 21.30%. Specification and coordination together make up 79% of everything that goes wrong, which means the model itself is rarely the culprit. Multi-agent systems in this study failed at rates between 41% and 86.7% depending on the benchmark, a range wide enough to show that harness design changes failure rates more than swapping the underlying model does. Constraint decay belongs in this category too: a rule stated early in a session loses compliance as the conversation grows and that rule drifts toward the middle of an increasingly crowded window, which is the U-shaped attention finding appearing as a behavioral symptom rather than a benchmark statistic.

Why root cause attribution is hard, and why it must still be done

The failure and its symptom rarely occur in the same place. That's the core difficulty in debugging agents: whatever went wrong upstream doesn't announce itself until several steps later, once its consequences have already propagated through the rest of the run.

One study quantified this gap across 78 behavioral failures and found the root cause preceded the visible symptom by a median of 4 steps, with some cases stretching to 26 steps, and the root cause was strictly upstream of the symptom in 62 of the 78 runs, or 79% of the time. A single root error tends not to travel alone either: it triggers an average of 3.2 further violated checks downstream, and 76% of failures end up violating more than one check by the time anyone notices.

AgenTracer's case study puts a concrete shape on this. AgenTracer-8B traced a failure back to Step 2, where a Web Surfer agent retrieved a file with the wrong date attached, an error that only became visible when someone examined the evidence at Step 11. Every step in between looked correct. Nothing in the intervening trace hinted that anything upstream had gone wrong.

Even the best available tooling for this leaves real uncertainty on the table. AgentDebugX's DeepDebug approach, which combines global trajectory understanding, structure-guided investigation, and cross-examination, achieves 63.6% strict attribution accuracy on the Who&When benchmark. That's the state of the art, and it still gets more than a third of cases wrong. The AgentFail benchmark shows that feeding a fine-grained failure taxonomy into the guidance an LLM uses for localization improves identification accuracy by roughly 15%, suggesting that how failure types get represented matters almost as much as the detection method itself. Emerging research directions aim to trace failures back to specific agents and actions more precisely. Those approaches are promising directionally, but are not yet something teams can treat as a production default.

For context overflow specifically, this attribution gap is the whole ballgame. Truncation symptoms almost never occur at the point of truncation. A run that gives a wrong answer at step 11 may have silently lost its required evidence at step 2, and without a trace that captures the resolved input, the exact string sent to the model, at every step, there's no way to reconstruct what happened. And this is precisely where model-as-judge approaches fall apart: most agent failures, stale caches, malformed JSON, empty tool results, overflowed context, are observable facts, not matters of subjective quality. Asking a judge model to detect them is circular, since judge and judged share the same substrate and the same blind spots, and it misses the deterministic signal sitting right there in the trace.

Treating context overflow as a deterministic harness failure

Overflow is a harness construction failure, since the model never saw the evidence to begin with. It's a harness construction failure: the prompt that got assembled did not contain what the task actually required, and no amount of model capability fixes an input the model never received.

This puts overflow in what one framing calls Tier 1 evidence: externally observable facts the agent cannot fabricate. Did the resolved input fit inside the model's window? Did the specific document ID the task depended on actually appear in the assembled prompt? Was the tool's returned output non-empty? These are facts about what the harness built, inspectable independent of anything the model says about its own reasoning. They're facts about what the harness built, inspectable independent of anything the model says about its own reasoning.

That independence is the whole point. The byte count of an assembled prompt, the presence or absence of a chunk ID, the embedding similarity score of a retrieved passage: none of these are things the agent gets to write itself, because the harness is what assembles them before the model ever sees the call. Compare that to Tier 3 evidence, model-as-judge evaluation, where one model judges what another model produced with no independent ground truth to anchor the judgment. That approach has a real place, for the genuinely subjective tail of failures where reasonable evaluators might disagree. It has no place evaluating whether a document fit inside a context window, because that question already has a deterministic answer sitting in the trace.

Overflow can originate in any of six harness layers: prompt construction, the tool interface, context and memory management, lifecycle and orchestration, observability, and verification. A RAG configuration with too permissive a top-K setting is a retrieval-layer failure. A missing summarization step between loop iterations is an orchestration-layer failure. A prompt assembly step with no logging at all is an observability-layer failure, and it's often the reason the other five layers stay invisible until something breaks in production.

Teams that label an overflow-driven wrong answer as "model hallucination" end up fixing the wrong layer. They retrain, fine-tune, or swap to a different model, none of which touches the actual defect, which sits in prompt construction, retrieval configuration, or the absence of a summarization pass. The MAST numbers back this up directly: 79% of multi-agent failures trace back to specification and coordination problems, not to anything about the model's underlying capability. Harness engineering is the lever that actually moves the needle here, and it's the lever most postmortems skip past on the way to blaming the model.

How to instrument resolved inputs so overflow becomes detectable

Most eval setups grade the output text and never look at the input that produced it. The question they ask is "did the answer look good?" The question that actually catches overflow is "did the evidence the agent needed make it into the call at all?" Those are different questions, and answering the first one tells nothing about the second.

Catching overflow requires capturing the fully-resolved prompt at the exact moment of the model call, not the template it was built from, but the assembled string as it actually went out, along with the token count at every span, the required chunk or document IDs the task depended on, and any provider error codes returned at the boundary. Without that resolved artifact sitting in the trace, none of the downstream attribution work described earlier is even possible.

A handful of OpenTelemetry signals do most of the work here: llm.token_count.prompt and llm.token_count.completion attached to every LLM span, provider error rate tracked specifically for context_length_exceeded responses, prompt-token counts at the p99 level broken out by route, and agent step count per session as a leading indicator that a loop is heading toward trouble before it gets there.

The gate that matters most is cheap and deterministic, and it should run before a single judge token gets spent. Two checks: does the token count of the resolved prompt fit inside the model's actual limit, and does every chunk ID the task required actually appear in the resolved prompt string. Fail either check, and the run is dead on arrival. Block it. Don't grade it, because grading a run that never had access to its own evidence produces a score with no meaning attached to it.

Both of those checks are arithmetic and string search against an artifact that's already been captured. They run in the hot path at close to zero added inference cost, which frees up model-as-judge evaluation for the roughly one-fifth of cases that are genuinely subjective and can't be settled by a deterministic check. That split matters: spending judge tokens on questions arithmetic can already answer is waste, and it dilutes the signal on the harder cases that actually need judgment.

One more detail matters here: the threshold for what counts as overflow risk has to be model-aware. A single global token ceiling will be wrong for most of a fleet running mixed models, since a threshold tuned to a 400K-token model says nothing useful about a 1-million-token one. The alert should be a function of the specific model's limit, not a fixed number pasted across every deployment. One workable pattern used in production sets the alarm when prompt-token p99 drifts past 80% of the model's actual limit, catching the approach to overflow while there's still room to intervene, rather than catching the moment after truncation has already happened and the evidence is already gone.