What Counts as a Meaningful Regression Signal When Replaying Production Traces

Replay-based evaluation against production traces tests agent changes before deployment by running the changed harness code against the execution boundaries in the record. A meaningful regression signal is a failure attributable to a prompt, tool schema, workflow, or memory change, reproducible under controlled replay, and validated against a representative trace slice. A different output alone is not enough.

Why replaying production traces does not automatically produce trustworthy regression signals

Replay seems to promise that the original failure will happen again. That intuition breaks down for agents because rerunning the same input does not necessarily recreate the execution that failed. Chronicle, by Chawla & Koul at REALM 2026, cites Atıl et al. (2025) for the finding that LLM inference is not bitwise reproducible even at temperature zero. An unchanged agent can therefore produce a different response before it reaches the suspected fault. The test has changed without any code changing.

Inference is only part of the problem. Tools read external state that may have changed since the incident, while retries and routing alter how many times a step runs. Those differences can redirect the multi-step trajectory away from the original failure or toward a different one. The same code then passes or fails because of the conditions and the order in which it runs, without a harness edit. Running it again cannot by itself show whether a change fixed the incident, caused a regression, or took a different path.

The three sources of replay noise that mimic regression

Inference non-determinism has an inconsistent signature across reruns. An output changes even though the harness does not, and the apparent failure may disappear on the next attempt. Such variation is evidence about repeatability. The harness must be the controlled variable before a difference can be attributed to it. Otherwise, ordinary variation can look like either improvement or deterioration.

Stale tool state produces a more localized pattern. A replayed tool call returns something different because the external state no longer matches the original run. Unlike inference noise, the divergence may recur consistently at tool-output spans. That consistency can make it look persuasive, but it does not establish that the agent broke. A changed tool result reflects a changed world unless the test separates that change from the harness under examination.

Distribution gaps are gaps in the corpus as a whole. Tail-based sampling keeps failed, expensive, or anomalous traces but keeps only a small share of the happy path, so the collection is biased by design. Failures clustered in those edge cases, without a matching failure in baseline traffic, may not support a claim of systematic deterioration. Broader replay can expose that limitation in the original sample. A meaningful signal must therefore show which effects came from the harness change and which came from inference variation, changed tool state, or sample bias.

What the five production failure categories actually look like in a trace

Schema compliance failures occur in intermediate tool calls even when the final answer is fine. At the context lengths common in production multi-step workflows, models can break schemas more often than benchmark scores suggest because final-output accuracy does not show how well intermediate tool calls follow the schema. Schema drift is a separate failure category. In that case, the model follows a tool contract learned from an earlier version while the system enforces a changed contract. The agent cannot distinguish a transient error from a structurally broken response, so identical retries produce identical invalid results.

Repeated tool calls, an abrupt increase in trace depth, or a task becoming unexpectedly expensive can expose a failure even without a dedicated step-efficiency metric. Latency and token cost then help show when the way the agent runs has changed along with its answer. In a loop, each call can look ordinary even though the workflow as a whole is clearly wrong. Its fingerprint lies in repetition and the relationships between steps.

Silent quality degradation is harder to detect because execution continues. The agent still gives outputs, but they become wrong or harmful. Document-store drift, prompt version changes, silent provider model updates, and input distribution shift can all cause this symptom. That range of causes makes the symptom a starting point for diagnosing a possible harness regression. A completed run is not necessarily a successful run.

These five categories become distinguishable through trace structure. A flat log can reduce schema failures, repeated calls, and deteriorating answers to broad labels such as “slow” or “error.” A span tree preserves parent-child causality, allowing the visible failure to be followed backward toward its originating step. That structure links classification to attribution. It exposes where a failure appeared without assuming that the first visible error caused it.

Why a divergence only becomes a regression signal when it can be attributed to a specific harness layer

Attribution connects an observed failure to a repairable cause. A wrong output, loop, or timeout can originate in a prompt, tool schema, workflow, or memory layer, and each origin implies a different change. A useful regression signal must identify the relevant layer and be specific enough to support a test of the suspected cause. Without that specificity, a passing rerun cannot explain why behavior improved.

Root-Cause Attribution Is a Search Problem describes automated attribution methods that typically ask LLM judges to inspect traces in a single rubric-guided pass. As execution logs become larger and more distributed, a judge can settle on a plausible failure before exploring the evidence fully. Plausibility is not a sufficient causal test. The same visible symptom may support several explanations, particularly when a later step inherits an earlier mistake. Naming the location of the symptom can therefore leave unresolved what needs to be repaired.

Chronicle makes attribution testable through cut-point replay. It places the test at semantic boundaries, including LLM calls, tool calls, and routing choices, and executes new code at selected boundaries while serving the others from the record. For a suspected tool-gate or router failure, cut-point replay tests whether changing that boundary changes the outcome. The claim now concerns a specific intervention rather than a general resemblance between two runs.

The Who&When Pro benchmark (2026) reflects this need for precise attribution through thousands of failure traces labeled for the responsible agent, decisive step, and failure mode. Those labels distinguish responsibility from the place where a failure finally becomes visible. Even that distinction does not prove a repair works. Attribution gives a specific hypothesis about what needs to change, and replay validation tests that hypothesis. The two serve different purposes and cannot substitute for each other.

How to test agent changes before deployment with cut-point replay

Chronicle distinguishes recording runs to trace or score them from recording runs to test a code change against them. Its mechanism records every non-deterministic boundary as an immutable envelope. Cut-point replay serves a chosen subset of those boundaries from the record and executes the complementary subset live with new code. The selected code is therefore exercised rather than merely replaced by a recorded answer. This is the bridge from an attributed incident to an executable regression test.

The benchmark provides a bounded demonstration of that mechanism. Across 6 recorded failures with simulated model boundaries, recording added 23 μs per crossing. Full replay made zero model calls and was bit-stable across 20 repetitions, so inference variation was absent from that replay mode. Cut-point tests failed on faulty code and passed on guarded and benign changes for all 6 incidents. These results support the technique within the benchmark.

The mutation study sharpens what the test proves. Cut-point tests caught every mutant that allowed the recorded unsafe action through, while a baseline that stubbed every boundary caught none. A test that replaces the relevant boundary cannot establish whether the new code there rejects the unsafe action. Recorded envelopes keep the incident’s inputs in place around the code being tested. The test also depends on whether the proposed repair runs against the recorded failure, beyond whether the data is real or synthetic.

Golden-set replay serves a different purpose. Replaying a golden trace set against current production on a schedule can warn of drift from model updates, prompt edits, and tool schema changes. It cannot by itself confirm that a particular fix resolves an attributed failure. Cut-point validation adds that causal test by exercising the candidate change at the relevant boundary. Validation also requires checking unrelated behavior, so that resolving the recorded incident does not become the sole condition for acceptance.

Building a trace sample that makes regression signals reproducible rather than lucky

A controlled replay is only as useful as the traces available to it. At high volumes, head-based sampling can discard the very executions needed to investigate an incident, while uniform or opportunistic sampling can capture important failures only by chance. Tail-based sampling preserves failed, expensive, and anomalous traces for later testing. That retention creates a durable source of difficult cases. It also creates the distribution bias that the interpretation of replay results must account for.

The practical requirement is to preserve failures without mistaking them for the whole workload. A tail-heavy corpus is well suited to finding and retaining production failure modes, but an apparent regression confined to that corpus needs testing against a broader slice. The two purposes are complementary rather than interchangeable. Keeping failures makes it possible to reproduce incidents, while broader coverage tests whether the same effect appears beyond those incidents. A uniform sample alone or a collection made up only of failures leaves one of those questions open.

Trace structure must survive the sampling process as well. Root, agent, LLM, tool, retriever, and embedding spans preserve the parent-child relationships needed to walk from a symptom to its originating layer. Retaining only the final answer or error loses the evidence needed for that attribution. More records cannot make up for missing causal structure.

Prompt versioning and A/B testing belong to the same testing infrastructure. Minor wording changes can silently degrade quality, so a recorded conversation without its prompt version cannot reliably tie the outcome to a particular edit. For teams validating prompt edits against historical conversations, the lesson is to combine versioned traces, retained failures, and a broader representative slice. That combination supports both incident-level reproduction and a defensible claim about behavior before deployment.

What a validated regression signal enables: harness iteration without guesswork

A validated signal gives harness iteration a concrete target. It identifies the layer responsible for a failure and supplies a replay test that distinguishes faulty behavior from the candidate repair. The resulting evidence is more useful than a broad score increase because it explains which behavior changed. It also limits the claim to the conditions exercised by the test. A representative trace slice then tests whether the change also works beyond the original incident.

The broader improvement case is not limited to repairing errors. Prompt mutation informed by trajectory reflection can outperform reinforcement learning approaches while using far fewer rollouts, demonstrating the leverage available in harness-layer changes. The Harness² framework extends that idea through two connected levels of recursion. Task-level recursion refines the execution harness for the current task, while harnessing-level recursion updates a persistent improvement harness to guide future tasks. A validated regression signal gives that improvement process a grounded input.

For deployment decisions, the consequence is a testing discipline. Teams should attribute failures to specific harness layers and validate candidate fixes against representative production trace slices before shipping. A plausible diagnosis or a successful rerun does not satisfy that requirement. The gating evidence is a controlled test that exercises the changed code, resolves the attributed failure, and checks unrelated behavior. Harness engineering then becomes subject to the same rigor as code review and testing, with release decisions grounded in demonstrated behavior.