Controlled Replay for Isolating Agent Failure Causes

Controlled replay tests agent changes against recorded production traces by freezing upstream state, intervening on one step, and re-executing forward. “Controlled Replay for Isolating Agent Failure Causes” explains why this operation closes the gap between seeing a failure and identifying its cause. Applied to historical conversations and tool-use traces, it supports validation of prompt edits before deployment and turns diagnosed failures into regression tests.

Why observability and evaluation leave agent failures unattributed

The dashboard is full, the trace is complete, and the evaluation score is red. Where did the agent go wrong? Observability records reasoning steps, tool calls, arguments, responses, token spend, and latency, but that record does not distinguish a cause from its consequences. Evaluation gives a task-level verdict, but that verdict cannot show whether the failure began in retrieval, tool invocation, or planning.

REFLECT (2026) defines the sharpest version of this gap as silent failure, where execution finishes without runtime exceptions but produces the wrong answer. Nothing crashes, and the trace looks syntactically correct. LLM judges and trained classifiers can nominate suspicious steps, but REFLECT identifies their structural weakness as proposing attribution hypotheses without testing them. Retry and self-reflection can instead recover a correct answer, yet that success leaves the original cause unresolved. Correction is not attribution, because a different successful path does not explain which decision broke the failed one.

Why multi-step agent traces make attribution structurally hard

Compounding separates the point of failure from its visible symptom. A bad tool argument or wrong retrieval can propagate through otherwise valid steps until the final answer exposes the damage. As intermediate results accumulate, the original goal loses salience, so judging a later decision requires evidence scattered across earlier instructions and observations. TRAJDEBUG (2026) describes this scattered evidence as a main reason errors are hard to find in long-horizon trajectories.

Non-determinism then weakens the obvious diagnostic move of running the task again. SymTrace (2026) reports that unguided reruns reproduce failures most of the time and achieve only 6.90% pass@3. A rerun resamples upstream decisions as well as the suspected step, leaving any eventual success open to several explanations.

Multiple errors create another ambiguity because some are repaired, some remain harmless, and others merely reflect earlier damage. The decisive error need not be the first mistake or the one nearest the final answer. Lu et al. (2026) found that microservice root-cause analysis agents could select the correct service without inspecting the relevant metrics, showing that even success can conceal a broken process. A working attribution method must hold prior state fixed, intervene on one step, and observe what changes.

What controlled replay does that other methods cannot

Controlled replay freezes the prefix, intervenes on one step, and re-executes forward. The comparison is between the recorded failure and execution under a specific repair. This turns a diagnosis into an experiment. Unlike retry, which changes the whole trajectory, prefix-preserving replay removes upstream variation from the comparison so that the effect of the intervention can be tested.

REFLECT formalizes four requirements for this operation. Execution grounding requires evidence from an agent run. Prefix-preserving replay keeps everything before the candidate step exactly as recorded, while targeted intervention gives the agent a repair plan based on the diagnosis. Inference-time computation derives the attribution from the particular failed trace.

Repair or Resample? (2026) reports that symptom-driven intervention raises repair rates by 191.89% over rerun baselines in the SymTrace setting. That result favors directed repair over unguided retries. However, downstream generation can still succeed by chance if the agent ignores the proposed correction, so holding the prefix fixed is not sufficient by itself. The replay must also show that the intended intervention took effect.

How REFLECT turns a failed trace into a verified attribution record

An auditor LLM examines the failed trace and expected answer, then proposes an error step, failure reasoning, and suggested fix. This remains a candidate explanation until execution tests it. The distinction is operational, because the auditor’s output determines where the system rolls back and what correction it attempts.

The second stage translates that hypothesis into a structured repair plan. The plan contains the root-cause diagnosis, a specific correction instruction, and forbidden actions intended to prevent repetition of the mistake. REFLECT restores execution at the candidate step with the prior prefix intact and injects the plan into context. Its faithfulness gate then inspects the first replayed step. If the agent ignores the plan but reaches the right answer anyway, the system flags the attribution as spurious rather than crediting lucky resampling as a verified fix.

The third stage uses the execution result to resolve the hypothesis. When the outcome changes from wrong to right and the gate is satisfied, REFLECT confirms the candidate as the earliest decisive error. The contrastive evidence also refines the attribution. REFLECT achieves the highest localization accuracy among same-auditor methods on all four localization benchmarks evaluated, with its largest gains on structured tool-use traces.

How Chronicle regression-tests agent changes against production traces in CI

A verified diagnosis establishes what broke, but a committed regression test guards the correction against later changes. Chronicle (EMNLP REALM Workshop, 2026) connects those two engineering tasks by converting a production failure into a test that can run without live LLM calls. Its recorder keeps a record of decision boundaries, including LLM calls, tool calls, and routing choices.

Chronicle’s central operation is cut-point replay, which serves selected boundaries from the record while executing the complementary subset live with new code. This isolates the changed component without rerunning the entire upstream trajectory. The boundary choice is essential to the test. Serving every boundary from the record would prevent the test from exercising behavior that the code change needs to alter.

Recording adds 23 μs per crossing compared with a 300 ms model call, while full replay uses 0 model calls and stays bit-stable when run again. Those properties make repeated CI execution practical. In all six production incidents tested, cut-point tests fail on faulty code and pass on guarded and benign changes. In the mutation study, they catch every mutant that permits the recorded unsafe action, while a baseline that stubs every boundary catches none.

How TRAJDEBUG handles trajectories with many errors, most of them irrelevant

Controlled intervention still needs a well-chosen candidate when a trajectory contains several mistakes. TRAJDEBUG, developed by Tsinghua University and Tencent Hunyuan, addresses that candidate-selection problem through error-lifecycle tracing. Its stages are error trigger detection, error state classification, and candidate-set-guided causal attribution. The classification separates resolved, harmless, and propagating errors. This prevents a conspicuous but repaired mistake from receiving the same weight as an error whose effects persist to termination.

REFLECT tests a proposed attribution through a faithful outcome flip, while TRAJDEBUG distinguishes critical errors from the noise surrounding them. Its evaluation uses TRAJERRBENCH, which contains 486 manually annotated failed trajectories from τ²-Bench and SWE-Bench Pro across tool-use and coding scenarios. TRAJDEBUG achieves the best overall performance over advanced LLM prompting baselines and existing diagnostic systems across that benchmark and existing agent benchmarks. Its application study also shows that the diagnoses improve downstream agent success, connecting localization quality to changed behavior.

What AutoSaddler shows about turning replayed failures into durable harness improvements

Repairing one recorded incident does not establish that a change will help across other inputs. AutoSaddler (Microsoft Research et al., August 2026) uses offline learning to make changes that help across other inputs. It ingests mini-batches of failing traces, performs in-depth diagnosis, and generates structured patches across prompts, tools, and middleware. Patch selection depends on generalization. Updates must improve the targeted failure mode without regressing held-out behaviors.

Replay-grounded attribution provides a conceptual input to this process by turning a failed trace into a structured explanation of what needs repair. AutoSaddler performs its own diagnosis, and the sources do not describe it consuming REFLECT output. The connection is therefore a pipeline of engineering functions.

On GAIA2, SWE-Bench Pro, and Terminal-Bench 2.0, it outperforms the corresponding base harnesses by 9.0, 9.6, and 10.0 percentage points and beats the strongest automated baseline on each benchmark. It also reaches its best performance using far fewer traces than the next competing approach. That sample efficiency makes the improvement loop relevant to teams without massive failure datasets.

What the observability layer must capture to make replay possible

Task-level inputs and outputs cannot reconstruct the state around an internal decision, so agent observability must capture the sequence between them. That includes reasoning steps, tools considered and invoked, arguments, returned responses, token consumption, and latency at each hop. These records need a single hierarchical structure that supports step-by-step replay. Missing upstream evidence limits both the diagnosis and the intervention it can support.

The OpenTelemetry GenAI specification provides a vendor-neutral standard as of 2026 across client spans, agent and workflow spans, MCP conventions, events, metrics, and provider-specific attributes. Its two primary client histograms recommended for production deployments are gen_ai.client.operation.duration and gen_ai.client.token.usage. All GenAI metrics remain in a developmental status, and the unreleased main branch replaces gen_ai.client.token.usage with gen_ai.client.inference.usage.* counters.

Timing and usage metrics describe execution costs, but they do not replace the boundary records required to reconstruct execution. Chronicle records LLM calls, tool calls, and routing choices as they happen. Those boundaries cannot be recovered after the fact. Without them, an incident may remain inspectable while being impossible to replay surgically.

Why harness engineering, not model replacement, is the lever replay unlocks

When a faithful replay succeeds after correcting prompt ambiguity, a tool argument, a missing guardrail, or a broken feedback loop, it points improvement effort toward the harness. The demonstrated intervention changes execution without replacing the model. This establishes a testable alternative to assuming that every failed task requires a more capable model.

A loop-detection guardrail helps with a repeated read/edit cycle, while changing how constraints are kept helps with context rot as the context window grows. Tighter tool schemas and argument validation address nonexistent tools and invalid arguments. Feeding a typecheck or test-runner signal back into the loop helps with false completion, and a planner/executor split with explicit scope constraints helps with unrelated file changes. Each repair targets a distinct failure mechanism.

Co-Harness (2026) puts this improvement cycle into practice with a HarnessCritic that traces failed trajectories to structured harness-level causes and suggests local harness diffs. A diff is accepted only when validation rollouts improve the targeted failure mode without regressing held-out behaviors. The accepted harness then produces higher-quality trajectories for subsequent improvement. The lesson for release engineering is a reviewable change, a validation step, and a pass/fail signal before shipping. Controlled replay supplies the causal evidence that makes that discipline possible.