Retrieval Ranking Errors That Cause Agent Hallucination

Agent hallucination often starts before generation, when retrieval ranking gives the model the wrong evidence, evidence that is out of date, or weak evidence with a score that makes it look reliable. The engineering consequence is direct. Retrieval ranking must be treated as a distinct, diagnosable failure layer, so teams can fix the evidence supplied to the model rather than mistake every wrong answer for a model defect.

Why retrieval ranking deserves its own failure category

Whether the cause is a knowledge failure or an evidence-selection failure changes where debugging starts. The CIVI paper applies the ARISE framework to ten frontier models and attributes 72.1% of all observed failures to retrieval-bound causes. That finding concerns civic-information search agents, and retrieval-bound causes include more than ranking alone. It nevertheless establishes the central diagnostic problem. A model can give a wrong answer when search fails to find evidence that supports the answer, even if the model has the knowledge and ability to answer.

The agent hallucination survey likewise identifies retrieval and memory as a workflow stage with its own hallucination triggers, separate from generation. Reading the answer may establish that a claim is wrong, but it cannot establish whether the model received the right material and misunderstood it or received unsuitable material in the first place. Those cases call for different engineering responses. Retrieval ranking therefore needs to be inspected as a diagnosable layer in its own right.

The three ranking errors that hand the model bad evidence

The most visible ranking error is the wrong chunk. A ranker surfaces text that appears relevant but concerns an adjacent document or topic, and the model builds its answer on that text without better evidence. The agent hallucination survey finds that retrieved context that is off topic or misleading can directly trigger hallucination. ARISE names the broader pattern retrieval failure, where search was invoked and content was returned, but that content did not support a correct answer. This pattern accounts for 37.6% of retrieval-bound causes in the ARISE analysis. An engineer can often recognize the mismatch by reading the returned chunk against the question.

Stale results are harder to recognize because the retrieved content may once have been correct. The mistake is to treat what was true before as true now, so publication or index dates need to be checked along with relevance. The University of Pennsylvania citation study distinguishes hallucinated URLs that never existed from stale URLs that existed but are now unreachable. Both can leave the reader with an unusable citation, but stale retrieval is not fabrication. The study also finds that deep research agents give far more citations per query than search-augmented LLMs, but hallucinate URLs at higher rates. More citations cannot make up for poorly chosen evidence, and non-resolving rates vary so much across academic fields that the risk of staleness depends on the field.

Confidence-score mismatches are less visible still. A retrieval system assigns a high relevance score to a weak match, and downstream logic treats that score as permission to trust the chunk. The uncertainty quantification paper finds that confidence estimates from agents’ own responses do not consistently outperform a simple baseline. How confident the model says it is therefore cannot be trusted to correct a misleading retrieval score. Detecting this failure requires calibration analysis.

Why ranking errors are hard to catch at the point where they surface

The central difficulty is the distance between the cause and its visible effect. In multi-hop agentic RAG, retrieval, reasoning, and generation are interleaved rather than confined to separate, completed phases. A weak chunk selected during an early hop can influence intermediate reasoning before the final answer exposes the mistake. Those intermediate outputs may remain plausible because they are consistent with the evidence the agent received. Output monitoring sees the conclusion, though the evidence-selection error that caused it may have happened several steps earlier.

Because of that delay, local checks are not enough even when they look reassuring. The uncertainty quantification paper formally shows, through a counterexample, that step-level calibration does not imply trajectory-level calibration. The paper also finds that errors are coupled along a trajectory. An early retrieval mistake can therefore compound through later steps.

A later response can follow coherently from an earlier response while both depend on the same unsuitable chunk. Its coherence does not supply independent support for the original claim. As a result, inspecting only the point where the answer becomes visibly wrong can misplace the cause.

Aggregate confidence introduces another blind spot. The uncertainty quantification paper finds that averaging trajectories together hides overconfidence at later stages, which shows up when the results are split by horizon length. A single score compresses differences between stages into a summary that is too coarse to represent their unreliability. A high aggregate score can therefore coexist with overconfidence in the part of the trajectory that matters for the final answer.

Training that rewards confident, well-formed answers can push the model to fill gaps instead of flagging evidence that is thin or unclear. The output can consequently look complete even when its source material is not. Search bypass creates a separate attribution trap, where the agent skips retrieval and answers from parametric knowledge. A retrieved-looking answer does not prove that retrieval supported it, and recording whether any tool call occurred does not establish that the relevant claim was searched.

What structured failure attribution reveals that output monitoring misses

Structured attribution replaces the question of whether the answer is wrong with the question of which layer made it wrong. The distinction requires more than interpreting the final prose. The CIVI paper uses ARISE to separate four mutually exclusive failure modes, namely search bypass, retrieval failure, grounding failure, and comprehension failure. Its source-injection ablation isolates the role of retrieved evidence by controlling the source material shown to the model. This creates a basis for distinguishing a failure to obtain useful evidence from a failure to use evidence correctly.

If the supplied source supports the correct answer but the model mishandles it, the fault is not established as ranking. Conversely, an answer generated from unsuitable material cannot be diagnosed solely as a comprehension failure. The agent hallucination survey maps eighteen triggering causes across workflow stages, showing why the stage boundary matters. Addressing one cause can leave others untouched and the overall failure rate unchanged.

The University of Pennsylvania study separates fabrication from stale retrieval, although both can produce a non-resolving URL in the final answer. Reading the output alone does not show the history needed to tell them apart. Production traces provide the record for that attribution when every model invocation, tool call, retrieval result, and confidence score is logged at the step where it occurs. The purpose is not merely to retain more data, but to preserve the relationship between the evidence supplied and the decision that followed.

How retrieval ranking errors look inside a production trace

Consider a trace walkthrough in which the final answer contains a fluent but incorrect claim. Start with the model invocation that produced that claim and read the retrieval result supplied to it. The first signal is a chunk-answer mismatch. The chunk does not contain or entail the claim, even though the answer presents it as supported. This exposes the gap that the model filled rather than grounded in the retrieved evidence. It does not yet explain why that chunk was selected.

Next, read the retrieved document’s publication or index date against the known update to the queried fact. If the document predates that update and the answer treats its contents as current, the trace exposes a timestamp gap. Stale evidence can contain genuine content. Relevance to the question does not establish that the answer remains valid.

Then inspect the relevance score attached to the chunk. A high score paired with text about an adjacent topic, or text sharing vocabulary without addressing the same concept, exposes confidence-score inflation. The score and the evidence disagree about how useful the chunk is. Read backward through earlier retrieval calls if the trajectory has multiple hops. Because early and later errors are coupled, the important point is the earliest retrieval call that supplied weak evidence.

For a citation-producing agent, the trace can also show the limits of automated correction. The University of Pennsylvania study’s urlhealth tool checks liveness through HTTP and uses the Wayback Machine to classify non-resolving URLs as stale or likely hallucinated. Its agentic self-correction can sharply cut the number of non-resolving citations, but how well it works depends on how well the model uses tools. Trace inspection is still needed to understand why the correction workflow failed rather than treating the presence of that workflow as proof of success.

Fixing retrieval ranking at the harness layer without retraining the model

Once attribution identifies retrieval ranking as the fault, the intervention belongs in the harness. The harness controls retrieval configuration, chunk boundaries, index freshness policy, and confidence-score thresholds. Changing chunk size, re-ranking strategy, staleness filters, or score thresholds changes how evidence is chosen while leaving the model as it is. The agent hallucination survey identifies retrieval augmentation, including improvements to retrieval and filtering, as a primary mitigation category distinct from training-side interventions.

For confidence-score mismatches, the notes’ proposed intervention is a calibration layer between retrieval and the model. A cross-encoder or uncertainty-aware re-ranker can re-score chunks before they enter the context window. The relevant test is not simply whether individual scores look better after the change. The uncertainty quantification paper proposes Trajectory-Checkpoint Expected Calibration Error (TC-ECE) to evaluate calibration across the whole agent trajectory, providing an evaluation signal for whether a re-ranking change improves end-to-end calibration.

Agents that produce citations can use post-hoc URL verification in the harness. The agent calls urlhealth after generating citations, checks liveness, and retries retrieval for failures. The University of Pennsylvania citation study reports that, after mitigation, non-resolving citation rates were below 1% for all three models tested. However, the degree of reduction varies enormously by model. A successful correction workflow in one agent configuration cannot therefore be assumed to transfer to another.

Validation before deployment is essential because a ranking change that improves one query pattern can silently degrade another. Replaying the production traces that showed the original failures checks whether the change fixes the cause it was meant to address. The complete process connects a failed answer to its evidence, attributes the fault to a layer, changes that layer, and tests the result before shipping. It gives the team a repeatable way to determine what failed and what to fix, once retrieval ranking is recognized as a distinct, inspectable layer.