Replay-Based Eval Workflows for Validating Agent Changes Before Shipping

Replay-based eval workflows validate agent changes before shipping by testing them against the production trajectories where real failures occurred. The evidence is the full run. A reliable workflow joins trace capture, per-dimension scoring, and CI gating so that each change repairs the failure it targets without degrading behavior that already worked.

Why validating agent changes is harder than validating model outputs

An agent produces a trajectory of tool calls, reasoning steps, and intermediate decisions. An error early in that sequence can cascade through the remaining steps. Standard unit tests and metric-based pipelines check input-output pairs, while an agent eval gives a score to a full trajectory. The unit of validation is therefore structurally different.

That difference makes final-answer checks insufficient for release decisions. An answer that looks right can hide wrong tool selection, brittle intermediate decisions, or reasoning errors that will be visible in another run. Checking only the answer is like grading a math exam without examining the working. The result says little about whether the process was sound.

The way a run unfolds also makes it harder to check because tools can call other tools, LLM calls can branch into parallel paths, and retries can build complex execution trees. Graph-based trajectory views show how those parts of a run connect in ways flat metric dashboards do not. An outcome-level signal can establish that a run failed. It cannot, by itself, identify the step or component responsible, leaving the engineer without the evidence needed to judge a change.

The three harness-layer failure categories that replay must catch

Most agent failures originate in the harness, including its prompts, tool schemas, and workflow logic. Tool schema drift makes that distinction concrete. The model follows one tool contract while the system enforces a changed contract, perhaps with a renamed field, a new required property, or a stricter validator. Schema validation on every tool call catches missing fields and malformed arguments. The repair belongs in the contract.

Prompt ambiguity and drift can start when a system prompt, expected output format, or tool definition changes. Unlike a rejected tool call, the resulting degradation may be gradual. The agent still seems to work as its output gets worse. Snapshot evals on every prompt PR show that drop by comparing scores with the prior version, so the prompt artifact itself gets regression testing.

Workflow loops form the third category. Here, the intervention is a step budget guard in the workflow logic. The replay must expose the repeated execution and test the guard against it. These categories require distinct checks because a valid tool argument does not establish that the prompt is clear or that the workflow will stop.

What a production trace actually contains and why it is the right evidence base

A production trace records the agent’s actual execution on a real task. It contains intermediate LLM calls with prompts and completions, tool calls with names, arguments, and return values, and token counts and latency per step. Planning or reasoning steps, retrieval results, and outcome metadata complete the record. Each field supports a different inspection. Arguments show where a call broke the contract, returned results show what evidence the agent had, and timing and token use show costs that the final answer hides.

AgentRR records interactions with the environment and the internal decision process, then summarizes them into a structured experience containing workflow and constraints. The trace is the raw material for that experience. Synthetic inputs lack this direct connection to actual behavior because imagined tasks can differ from the distribution of production traffic. So even when a synthetic suite passes, it cannot show that a change fixes a given production failure.

Traces need to be captured and stored in a structured form. OpenTelemetry spans record the tool call name natively, with arguments and return values available as opt-in attributes. Scores must attach to spans. Otherwise, a failing score has no direct connection to the execution that must be replayed.

Building the trajectory dataset that replay eval runs against

The offline dataset is a versioned set of full trajectories. Its coverage should span tool-call variants, argument edge-case buckets, and error codes, stratified by failure type. Happy-path coverage is not enough. Without those distinctions, an apparently healthy suite can omit the exact argument shapes or tool errors that break real runs.

Recorded tool outputs and sandboxed side effects make replay stable and cheap. Without sandboxing, replay can execute real side effects again and become non-deterministic. The dataset must also preserve enough context to retain the constraints behind the failure. AgentRR addresses this specificity-generality tradeoff through multi-level experience abstraction, since overly detailed experiences fail to generalize while overly abstract ones lose necessary constraints. Its check function provides a trust anchor for completeness and safety during replay.

Dataset maintenance requires a weekly promotion cadence for failing production traces. New intents, prompt revisions, tool schemas, and traffic shapes can make a frozen dataset stale within weeks, even if its scores remain stable. Weekly promotion keeps the test population connected to the product the team is actually operating, rather than the one that existed when the suite was assembled.

Rubric design and the scoring stack that turns trajectories into actionable signals

Trajectory scoring needs six distinct dimensions, which are tool selection, argument extraction, result utilization, error recovery, plan coherence, and task completion. An aggregate score can hide the failing axis. A score of 0.85, for example, can conceal argument extraction at 0.62 behind tool selection at 0.97. The argument defect remains a production risk even when the overall result looks healthy.

The scoring stack proceeds from deterministic checks to LLM-judge rubrics and then human review. That order reflects cost and speed. JSON schema validation, substring and regex checks, and non-empty return assertions catch a large share of harness failures without a model call. They are fast and repeatable, making them the foundation.

LLM judges cover qualities that deterministic checks cannot establish, including reasoning coherence, plan quality, and result utilization. Their scores require calibration against human labels before they can be trusted. A small human review queue holds traces that are hard to judge or have high stakes, and the corrections become few-shot examples for later judging. Custom dimensions such as memory consistency or domain-specific plan coherence need pinned judge prompts whose examples change as those corrections come in. The same rubric must then travel into production, or offline and live scores will describe different standards.

How the CI gate turns per-dimension scores into a regression block

The CI gate converts dimension scores into assertions that can block a merge. It must exit non-zero on the failing axis. Healthy tool selection and task completion cannot excuse degraded argument extraction.

The setup for fixtures and assertions must link the replay dataset to the rubric being evaluated. YAML is a way to express that setup. Beyond trajectory quality, the gate must cover success rate, step budget, tool error rate, latency, cost caps, and safety constraints. These checks protect operational behavior that an accuracy-only gate leaves outside its decision.

A no-regression constraint also rejects candidates that improve newly targeted tasks by sacrificing previously passing behavior. Harness changes can ship on a shorter release cycle because they need no model weight update, as long as each goes through the same repeatable validation.

The gate’s output must identify both the failing axis and the specific traces that failed it. A non-zero exit code on its own blocks the change but does not show what needs fixing. Those traces become the evidence for attribution.

Attributing a CI failure to the specific harness layer that caused it

A failed dimension narrows the investigation without identifying the cause. Poor argument extraction, for example, still requires inspection of the prompt and tool contract that shaped the call. Attribution connects the score to the responsible layer. The relevant evidence may appear well before the final failure or be distributed across distant steps, so systematic constraint evaluation is needed.

The intervention depends on that distinction. A model failure informs post-training objectives or model selection, while a harness failure directs changes to prompts, schemas, workflow, or memory. Environment and grader problems instead require dataset repair or rubric correction.

Multi-agent systems require attribution to both the responsible agent and the earliest error-inducing step. Inter-agent messages and tool outputs give the evidence, while the failure symptom helps narrow where to look. An auditable attribution log records constraint violations with supporting trajectory evidence for each failing trace. Engineers can then inspect why a layer was selected. A prompt ambiguity finding leads to a rewrite, schema drift leads to a corrected tool definition and CI assertion, and a workflow loop leads to a step budget guard.

How to test agent changes against real production traces before deploying

A candidate fix should first be replayed against the exact traces where attribution located the failure. The engineer applies the prompt rewrite, schema correction, or workflow modification, then scores the resulting runs with the same rubric used by CI. This tests a specific repair claim. A synthetic benchmark cannot give the same evidence if it lacks the execution conditions that brought the defect to light.

The replay must also preserve its completeness and safety constraints. AgentRR’s check function puts the same rule into a formal check and accepts replayed experience only when those constraints hold. A passing score alone is insufficient if the replayed trajectory is subtly broken.

The previously passing dataset remains part of this decision. Repairing the targeted traces is necessary, but the candidate must also pass the full per-dimension gate without degrading established behavior. Otherwise, the change has exchanged one failure for another. The reviewable result must show which traces passed, which still failed, and which new failures appeared before shipping is recommended. CI success on a dataset that omits the target failures does not validate the proposed repair, regardless of how healthy its other scores look.

Closing the loop with production monitoring after a validated fix ships

After release, the same rubric attaches to live production spans so that new failures become inputs to the next validation cycle. Scores joined to span IDs point straight to the traces that hold the evidence. That join keeps monitoring connected to replay. A score without its execution record can flag degradation, but it cannot supply the next test case.

Live failing traces are clustered by failure signature, with a 4-D score written on each and representative traces promoted into the offline dataset. This clustering record feeds failures back from production, while the CI gate keeps its per-dimension assertions. Weekly promotion turns those representatives into durable regression coverage. Continuous evaluation of live unlabeled traffic can then flag a degrading harness before failures accumulate into a visible production incident.

Without monitoring and promotion, performance can follow a sawtooth pattern, declining until a threshold triggers another search for a fix. Continual learning here means reading live trace signals, refreshing the dataset, and applying the same rubric-gated validation before each harness change ships. With every stage connected, harness engineering becomes as rigorous and repeatable as code review and testing.