MCP Tool Testing and Evaluation for Agent Pipelines

An agent that clears its full test suite in staging can still fail the moment a real user sends a query that drifts even slightly outside what the tests anticipated: it picks the wrong tool, passes malformed arguments, strings together an unnecessary call chain, and returns garbage output. That gap between staging and production is the reason a passing test suite tells you almost nothing about how an MCP-connected agent behaves once real traffic hits it MCPHunt: An Evaluation Framework for Cross-Boundary Data Propagation How to Evaluate MCP-Connected AI Agents in Production (2026).

The scale involved makes this more than a theoretical concern. MCP now sees over 97 million monthly SDK downloads and supports more than 10,000 published servers, so a failure mode that occurs in one percent of calls affects a meaningful share of production traffic and is not an edge case anyone can afford to shrug off MCPHunt: An Evaluation Framework for Cross-Boundary Data Propagation How to Evaluate MCP-Connected AI Agents in Production (2026). Static-tool agents, the kind that call a fixed, pre-registered set of functions, don't have this problem in the same way. MCP agents select tools against a catalog that's exposed dynamically at runtime, and the same prompt can produce different call chains on different days because the model's behavior is not fully deterministic.

That non-determinism produces silent failures that go undetected during normal operation. The agent still runs. The UI still renders. The logs look mostly normal. But a prompt edit somewhere upstream, or a model upgrade, or a schema change on a server the team doesn't even own, quietly alters what the agent does, and no traditional stack trace catches it because nothing actually crashed.

Part of what makes this hard is that an MCP server is really two products packaged inside one binary. One product is the functional surface: descriptions that match what the tools actually do, calls that succeed with correct arguments, results that integrate cleanly into the next turn of the conversation. The other is the security surface, and testing only one produces a server that either breaks under real workloads or fails an audit. Three shifts over the past year raised the stakes on getting both right MCPHunt: An Evaluation Framework for Cross-Boundary Data Propagation.

Each of the failure classes that follow (a wrong tool selected, a malformed argument, a broken call chain, a credential that leaks across a server boundary) originates at a distinct point in the pipeline. Catching all of them requires an eval framework that treats those points as separate, sequential layers rather than a single checklist run once before a release ships. What makes MCP agents structurally different from static-tool agents is the focus of this discussion.

The four failure classes MCP tools produce, and the eval layer that catches each

Failures don't sort themselves by severity in any obvious way. A schema mismatch is deterministic and cheap to catch in CI, while a tool-selection error under vague instructions can sail through every protocol check in the suite and only reveal itself in a full trajectory review months later.

The first failure class is tool-description drift. The description says one thing, the tool does another, and the agent ends up picking that tool for queries it was never built to satisfy. In the logs, this looks exactly like a model regression, but it's a documentation regression sitting inside the server itself.

The second is argument schema non-compliance: a required field goes missing, an integer arrives typed as a string, or an enum value doesn't match what the runtime actually allows. The call still dispatches. It just returns an error, or worse, a result that's wrong but doesn't look wrong.

The third is tool selection error under ambiguity, and it's the class most staging suites never exercise. Existing MCP benchmarks tend to specify execution steps explicitly, so the agent is never actually challenged to infer which tool fits an underspecified request. That's a real gap, not a minor one, because production users rarely phrase requests the way benchmark authors do.

The fourth is chain inefficiency and loop failure: retries that don't need to happen, redundant calls, an agent that gets a valid result and then fails to move forward with it, loops that never terminate MCPHunt: An Evaluation Framework for Cross-Boundary Data Propagation. The MAST taxonomy, from Cemri and colleagues in 2025, attributes a large share of multi-agent failures of this kind to specification flaws and weak verification steps rather than to any deficiency in the underlying model MCPHunt: An Evaluation Framework for Cross-Boundary Data Propagation.

A fifth class cuts across all of the above: compositional data propagation across server boundaries. This one isn't adversarial and it isn't a bug in any single tool call. It emerges from the topology of the workflow itself, from how data gets routed from a read operation on one server into a write operation on another, crossing a trust boundary neither server was individually designed to guard.

The layering matters because none of these failure classes implies the others are handled. A tool that passes schema validation cleanly can still steer the agent toward the wrong action through a drifted description. A tool selected correctly can still fail on argument compliance once it's actually called. Correct arguments don't guarantee the resulting chain completes efficiently, and a chain that behaves perfectly in isolation can still leak credentials the moment it's composed with a second server. When a failure does appear in a live trace, this layer map is what lets an engineering team attribute it to a specific root cause instead of defaulting to "the model did something weird," which is rarely the actual explanation.

Layer one, functional correctness: tool-description accuracy and call success rate

Evaluating tool descriptions works best as a two-pass structure. The first pass is deterministic: validate the tool's JSON schema against its actual signature. This catches missing required fields, mistyped parameters, and enum mismatches, and it runs in milliseconds with no model call involved.

The second pass is semantic, and it costs more but catches a different kind of drift. Run a golden corpus of prompts, some that should trigger the tool and some that deliberately shouldn't, and score how accurately the agent selects it. A description that has drifted from what the tool actually does tends to produce a recall problem, missing calls it should make. A description that overpromises tends to produce a precision problem, triggering the tool when it shouldn't. Both appear as the identical number in an aggregate score, so the only way to tell them apart is to split the result on a confusion matrix.

Call success rate ties the layer together. A call only counts as successful when three things all hold at once: the right tool got selected, the arguments validated against the schema, and the server responded inside its latency budget Toloka MCP evaluations article. The production target for schema compliance is 98 percent, and a number below that is a signal, not noise, usually meaning an upstream server changed its contract Toloka MCP evaluations article.

This layer has real limits. It doesn't evaluate whether the agent picks correctly among several tools with plausible-sounding descriptions, and it doesn't check whether a successful response actually gets used properly on the next turn. That second gap deserves its own sub-check: a clean 200 response can still fail the chain if the agent can't parse it, drops a field silently, or ignores it entirely and replans from scratch based on the original prompt. Score three things on that next turn: whether the agent actually references the returned data, whether the chain moves forward, and whether the final answer cites the correct value. Latency budgeting: if the user-facing budget for a four-call chain is 8 seconds, the per-tool p95 lives around 1.5 seconds; track p50, p95, p99 per tool and alarm on p95 growth above the 7-day baseline.

Layer two, argument schema compliance across model and client configurations

Argument compliance earns its own layer because tool selection and description accuracy can both be correct and the agent can still generate arguments that don't conform to the schema, wrong types, missing required fields, enum values out of range, simply because different model providers parse the identical schema in slightly different ways.

That compatibility problem is real and specific. The same tool, exposed identically across Claude, OpenAI, Gemini, and Cursor clients, can get called with structurally different argument patterns depending on which client is doing the calling. Compatibility evaluation needs to run pre-merge on every schema change, across at least four client configurations, before anyone assumes a fix in one environment holds in another.

None of them require a model call, so they run in CI without a model call.

The 98 percent schema compliance figure from the previous layer is a production target that applies across ongoing operation Toloka MCP evaluations article. It's a production baseline, and drift below it means an upstream server changed its contract without a coordinated update reaching the agent's prompt or its schema cache Toloka MCP evaluations article. That drift plays out in a specific, sneaky way in practice: a required field gets removed from a schema, the agent keeps sending it anyway out of habit, the server just ignores the field silently, and a downstream step that depended on it produces a wrong result without the chain ever throwing an error. Nothing crashes, so nothing gets flagged, unless someone is tracing arguments at that level.

This is exactly where the production gateway earns its keep. It is positioned at the chokepoint and captures every argument that passes through it. Compliance monitoring can therefore run continuously against live traffic instead of only against the golden test set.

Layer three, tool selection accuracy under realistic, underspecified instructions

Two of the more serious MCP benchmarks, MCP-AgentBench with its 33 servers, 188 tools, and 600 queries, and MCP-Bench, tested across 20 advanced LLMs, both push the field forward on realism and coverage. But both still lean heavily on task correctness and protocol compliance, and both specify execution steps explicitly enough that the agent, tested with 33 servers, 188 tools, and 600 queries in one case and across 20 advanced LLMs in the other, is never really forced to infer which tool fits an ambiguous request MCP-AgentBench. That's the gap most production failures fall into.

What that gap looks like in a live deployment is not subtle once you see it. A model that scores 95 percent on a coding benchmark can still pick the wrong API endpoint 8 percent of the time when it's orchestrating a multi-step customer service workflow, and an 8 percent error rate, invisible in a traditional eval, turns into a genuinely large volume of wrong actions once it's running at enterprise scale Toloka MCP evaluations article.

Output-only scoring can't catch this reliably, because trajectories matter more than final answers. The identical correct answer can be reached through a sound reasoning path or a broken one, and two separate errors that happen to cancel each other out will register as a clean success if nobody looks at the path taken. Trajectory-based evaluation catches both the outcome and the route the agent took to get there, which output scoring alone simply cannot do.

Building selection evals that actually cover this gap means writing prompts that name the goal without naming the tool, forcing the agent to infer selection from context and description rather than from an instruction that spells it out. It also means deliberately including near-neighbor tool pairs whose descriptions overlap, since that's where precision failures concentrate, and scoring precision and recall separately against targets of above 85 percent and above 90 percent respectively.

Automated scoring alone misses one class of error entirely: whether the agent followed a sound reasoning path even when it landed on the right answer, and whether it handled genuine ambiguity well rather than just guessing correctly. A hybrid approach that pairs automated rewards with expert annotators reviewing full trajectories catches that class where pure automation can't. When a selection failure does show up under ambiguous instructions, the root cause usually sits in the harness itself, a prompt that doesn't give the agent enough context to disambiguate, or tool descriptions so similar that no model could reasonably tell them apart.

Layer four, chain efficiency and workflow loop detection

Chain efficiency measures calls per task, retry counts, redundant calls (the same tool invoked twice with functionally equivalent arguments), and the ratio of productive steps to total steps taken, with a production target sitting above 0.7.

The loop failure mode is the sharpest version of this problem. A reliable agent loop needs visible state, a clear sense of what the next step should be, and explicit stopping conditions, and without token limits, retry rules, and stop criteria defined up front, an agent can re-enter the same call sequence indefinitely without ever tripping a protocol error. Nothing in the standard error-handling path catches that, because from the protocol's point of view, every individual call succeeded.

Seven deterministic trajectory metrics cover this layer well: task_completion, step_efficiency, tool_selection_accuracy, trajectory_score, goal_progress, action_safety, and reasoning_quality, all of which run as cheap heuristics inside CI and can be layered with judge-based rubrics once the agent is live. Groundedness against MCP resources is the closest measurable proxy for confirming the agent is actually using the previous tool's result rather than ignoring it, with a target above 85 percent, and it is tracked as a distinct metric from context utilization, not a synonym for it.

Loop and redundancy failures are notoriously hard to catch in staging, since they often only emerge under production load patterns, a query type the agent handles fine in isolation can spiral into a loop the moment it's combined with a slow tool response or a partially returned result. Most of these efficiency failures trace back to the harness rather than the model: missing stopping conditions, unclear logic for what the next step should be, or a memory layer that doesn't persist intermediate results between calls. This is squarely where the planning and action modules of the AgentErrorTaxonomy apply.

The cross-cutting security layer (compositional credential propagation across server boundaries)

Compositional data propagation deserves treatment as its own layer because it isn't caused by any single tool behaving badly. Faithful tool composition, each tool doing what it was designed to do, can still turn individually reasonable read and write permissions into a credential leaking across a boundary neither tool's designer anticipated. It is a structural side effect of how the workflow is shaped.

The scale of this, based on the MCPHunt evaluation, is large MCPHunt: An Evaluation Framework for Cross-Boundary Data Propagation How to Evaluate MCP-Connected AI Agents in Production (2026). That's a wide enough range to say the problem is systemic rather than tied to any one model's weaknesses.

Task-mandated propagation should be separated from policy-violating propagation, since they call for different responses. Task-mandated propagation happens when an instruction genuinely leaves the agent no choice but to carry credentials along, which is a deployment risk that calls for infrastructure controls rather than a model fix. Policy-violating propagation happens when credentials get included despite the task only asking for a derived artifact, and that one is a real safety failure attributable to the agent's behavior.

This isn't hypothetical. A cross-tenant isolation flaw in Asana's MCP server exposed project data from roughly 1,000 organizations, and separately, a prompt injection against the official GitHub MCP server allowed exfiltration of private repository contents.

There's some genuine encouraging news here too. A prompt-level mitigation study across three models, per the MCPHunt findings, reduced policy-violating propagation by as much as 97 percent while preserving 80.5 percent of task utility How to Evaluate MCP-Connected AI Agents in Production (2026). But effectiveness varied with how well each model followed instructions in the first place, which suggests prompt-level defenses alone probably won't be sufficient as the only line of protection How to Evaluate MCP-Connected AI Agents in Production (2026).

Practically, this means security evaluation runs as its own parallel gate, pre-merge on every server pull request and on a recurring red-team schedule, rather than as a substitute for any of the functional layers above it. The scan layer needs to run at server registration and again on every refresh of the tools list, while the runtime layer has to fire at both the chat-completion boundary and the per-tool-call hook, since a single checkpoint will always miss either the catalog itself or the calls happening inside a loop. Empirical scale from MCPHunt shows that across 3,615 main-benchmark traces from 5 models spanning 147 tasks and 9 mechanism families, policy-violating propagation rates reach 11.5–41.3% across all models; the range is 25× across mechanisms, concentrated in browser-mediated data flows. MCPSecBench catalogues 17 attack types across 4 attack surfaces (arXiv:2508.13220, August 2025), while MCP Security Bench (MSB, ICLR 2026) specifies 12 MCP-specific attack types and end-to-end tool-use pipeline evaluation (arXiv:2510.15994) MCPSecBench (arXiv:2508.13220) MCP Security Bench (arXiv:2510.15994). Excessive agency risk, as defined by OWASP LLM06:2025, provides the checklist for auditing harness permission scope against the principle of least privilege, where over-provisioned functions, unnecessary permissions, and missing approval mechanisms are the structural preconditions for this failure class.

Building the CI gate that wires all four layers into one PR check

None of these layers earns its keep as a standalone report that someone reads after the fact. The value comes from wiring all four, plus the security layer that cuts across them, into a gate that blocks a merge before a bad server definition or a drifted schema ever reaches production traffic MCPHunt: An Evaluation Framework for Cross-Boundary Data Propagation.

That gate works best as two stages rather than one. A second stage, slower and more expensive, runs the semantic and trajectory-based checks against the golden corpus, including the human-annotated review for ambiguous selection cases, and this stage is where the tool-description confusion matrix and the security scan both belong, since neither can be reduced to a fast heuristic without losing what makes them useful.

The whole point of separating these layers is traceability. When a production incident does surface, whether it's a wrong tool selected, a malformed argument, a runaway loop, or a credential that crossed a boundary it shouldn't have, the team needs to know within minutes which layer let it through, not spend a day re-running the entire suite hoping something turns red. Treating MCP evaluation as a structured pipeline, rather than a single checklist run once before shipping, is what makes that kind of fast attribution possible at all. The two-gate structure, per the FutureAGI step-by-step guide, is presented here.