Prompt Injection Detection in Production LLM Apps

Prompt injection is no longer a research curiosity or a red-team footnote. It's a live production problem with a documented failure rate, and treating it as something a single filter can catch at the front door is why so many deployments keep getting hit. OWASP's Top 10 for LLM Applications, in its August 2026 edition, ranks prompt injection (LLM01) at the top of the list for the third consecutive year. A May 2026 industry survey from Zylos AI found that prompt injection appeared in 73% of production AI deployments during 2025. This piece lays out where injection actually enters an agent's harness, why bolting on one guardrail doesn't hold, and how to build detection that tracks each layer separately, then gets sharper every time an attack is confirmed.

The stakes aren't abstract. Obsidian Security has pointed out that AI agents move roughly sixteen times more data than a human user doing the same job, so a single compromised agent isn't a leaked file, it's every system that agent touches, Salesforce, M365, Workday, all at once. Three incidents from 2026 illustrate this: a zero-click exploit against agentic browsers in March, a browser-hijacking campaign in September that compromised five separate AI assistants, and a URL-parameter flaw in Microsoft Copilot in August. None of these needed a user to make a mistake. And regulators have caught up to the urgency: Article 55 of the EU AI Act, covering adversarial testing obligations for general-purpose AI, took legal effect in August 2025, with enforcement power kicking in a year later. Compliance here isn't a suggestion anymore. The real question for a team running agents in production isn't whether someone will try to inject a malicious instruction. It's whether the team can tell which layer it came in through, and fix that layer faster than the attacker can find the next one.

The four distinct surfaces where injection enters the agent harness

Direct and indirect injection get lumped together in casual conversation, but they don't behave the same way, and they don't get caught by the same defenses. Direct injection is text a user types into the product, an override attempt, a jailbreak phrase, a role-play trick designed to get the model to drop its instructions. It's visible, because it arrives through a channel the team controls. It arrives through a channel the team controls, which means pattern matching and semantic classifiers have a fighting chance against it.

Indirect injection is a different animal entirely, and it shows up across three separate surfaces inside the harness.

The first is retrieval. Anything pulled into a model's context through RAG, a webpage, an email, a database record, a PDF, gets treated by the model as trusted the moment it lands in context, regardless of where it came from. Research published in 2026 found that five carefully written documents, planted where a retrieval system would find them, could steer AI responses in the desired direction 90% of the time. That's not a hypothetical edge case. That's a working attack recipe.

The second surface is tools, specifically the Model Context Protocol layer. MCP servers expose tool names and descriptions during discovery, and those metadata fields can carry hidden instructions the model reads as legitimate configuration. Worse, a compromised tool can return adversarial content in its own output, which then enters the model's context looking exactly like trusted data. GitHub's Copilot had a real instance of this: CVE-2025-53773 let externally fetched content push the agent into running commands the attacker chose, not the user.

The third surface is memory. Agents that keep state across sessions can have that state poisoned once and exploited repeatedly. The Cursor AI Agent incident is the clean example here: because the system didn't require user confirmation before accepting a new workspace settings file, the agent wrote a malicious .cursor/mcp.json configuration on its own, no approval needed, no user in the loop.

Indirect injection skips the front door completely. The attacker never sends a message through the product interface at all, which means any defense built to screen user input is simply blind to it. A large-scale study (arXiv:2604.27202), scanning a huge swath of URLs across a large number of hosts, found 15,300 validated indirect injection instances spread across 11,700 pages. Only about 5.1% of these were visible if a person actually looked at the rendered page. A majority were hidden using rendering tricks, and roughly 70% sat in parts of the HTML no browser ever displays, headers, comments, metadata. And the attacks weren't improvised: 54 prompt templates accounted for 95% of every case in the dataset. This is a structured ecosystem with reusable playbooks, not scattered noise. A detection system built only to check what a user types is missing three of the four surfaces, and those three are exactly the ones that scale as agents get more capable.

Why single-gate defenses fail in production agent architectures

OWASP's own LLM01 guidance is blunt about this: given the stochastic core of how these models actually work, there's no confirmed fool-proof way to prevent prompt injection outright. Defense-in-depth isn't a best practice teams are choosing to adopt; it's the only posture the field currently has confidence in.

The case against relying on a single application-layer check is straightforward, and the pattern holds up under scrutiny. Coverage gaps come first: every new microservice, every new internal tool, every new model integration has to remember to call the moderation layer on its own, and the one team that forgets ships an unprotected surface without knowing it. Credential sprawl follows close behind, since each service needs its own set of keys for whatever guardrail or moderation provider it's using, and rotating those credentials turns into a coordination exercise across a dozen teams instead of a five-minute task. Then there's the issue of auditing: when something does go wrong, investigating means pulling logs from every service that touched the request, one by one, because there's no single place that shows which requests tripped a rule, which users sent them, or which downstream systems were affected.

Tool poisoning deserves its own mention, because it resists even careful defenses. Attackers can manipulate tool descriptions directly, embedding explicit malicious instructions, or more subtly, planting misleading claims about what a tool does or doesn't do. A paper on this exact problem, TRUSTDESC (arXiv:2604.07536, April 2026), found that existing defenses catch the obvious version, anomalous instructions sitting in plain text, but do close to nothing against the subtler implicit version, where the description just lies about the tool's behavior in a way that steers the model's choices.

The Microsoft Copilot incident from August 2026 shows what this looks like when it actually happens. The assistant was steered through a URL-parameter flaw and used its own legitimately held permissions against the user's intent. That's a scope violation: the agent didn't lose its credentials, it used them exactly as designed, just on the attacker's behalf. Once an incident looks like that, the question stops being "do we have a guardrail" and becomes "does our detection actually map to the layer where the injection got in."

How to build layer-mapped detection logic across the harness

Each layer of the harness has its own data shape, its own trust assumptions, and its own way of failing. Detection logic has to match that, not get applied as one uniform check bolted onto the front of the pipeline.

Start with the user prompt layer, since it's the most mature. Azure AI Content Safety's Prompt Shields analyzes user prompts and documents on separate tracks and classifies for known attack patterns, including indirect ones. Lakera Guard screens both inputs and outputs for injection attempts, data leakage, and policy violations. A simpler pattern teams often build themselves is an LLM-as-judge classifier: send the incoming prompt to a model with a system prompt that asks for a binary label, injection or not. It's fast to stand up, but it needs calibration against human-labeled examples as the volume of traffic grows, or its accuracy drifts without anyone noticing. Publicly available datasets can serve as a starting point for evaluation, but production use requires layering in application-specific traffic and real indirect-injection cases on top. Whatever detector gets built, track false positives and false negatives separately. A high block rate on its own tells a team almost nothing about whether the detector is actually accurate.

The retrieval layer needs a different posture entirely: everything pulled in from outside gets treated as untrusted, full stop, and the check happens before that content ever enters the model's working context. The same study cited earlier ran 5,200 controlled experiments across 13 models and found that structured representations of retrieved content, rather than raw plain text, cut down how often models comply with embedded instructions, Plain-text retrieval let compliance run as high as 8% even on smaller models. The signal to capture in a trace is which retrieved chunk came right before the model's behavior changed or it made an unexpected tool call.

Tool descriptions and tool output need schema-level scrutiny. The TRUSTDESC approach doesn't trust the description a tool's author wrote at all. Instead, it generates a trusted description directly from the tool's implementation, using a three-stage pipeline: static analysis pulls out the minimal code relevant to what the tool actually does, a synthesis step writes a description from that code, and a dynamic verification stage runs synthesized test tasks to confirm the tool behaves the way its new description says it does. Pair that with MCP tool allow-lists at the gateway, so an agent can only invoke tools it's explicitly permitted to call, and injection-driven tool abuse gets blocked before a request ever reaches a model provider. The detection signal to watch here is any tool call that doesn't match the schema expected at that step of the workflow, or a tool call the workflow plan never anticipated in the first place.

Memory is the layer teams forget until something like the Cursor incident happens to them. Writes to memory that come from processing external content need to be treated differently than writes that came directly from a user action, full stop, no exceptions. The specific signal to flag is a memory write that changes behavioral rules, session policy, or configuration files, and this pattern is consistent with how the malicious.cursor/mcp.json write occurred. Requiring an explicit confirmation step before an agent acts on any workspace configuration change sourced from retrieved content is a direct mitigation for this class of incident.

Tie all four of these together at a gateway that every request has to pass through, and the coverage, credential, and audit fragmentation problems from the last section mostly disappear on their own: one place for credentials, one uniform audit trail, no forgotten microservice quietly running unprotected. Obsidian Security's assessment puts real weight behind the urgency here, finding that 40% of agents assessed carry critical risk ratings. Teams building this layer by layer are working against a live exposure surface, not a theoretical one.

Measuring detection accuracy in production without adding latency

A runtime guardrail and an evaluation pipeline answer two different questions, and conflating them is a common mistake. The guardrail decides, in the moment, whether a request proceeds. Evaluation against labeled data answers a slower question: are those in-the-moment decisions still accurate as the application, the model, and the traffic mix keep changing. Both matter, but they need different instrumentation, and neither substitutes for the other.

The pattern that avoids adding latency to the user-facing path is asynchronous scoring after the fact. Braintrust's September 2026 guide on quality, cost, and latency alerts for AI agents describes exactly this: log the trace first, run the injection detector against it afterward, and let blocking decisions stay entirely with the runtime guardrail. The scoring pass adds zero delay to the actual request because it isn't in the request's path at all.

What gets scored, concretely: whether a sampled input contains an injection attempt, whether each tool call in the trace matches the schema the workflow plan predicted at that step, whether a retrieved chunk contains instruction-like text inconsistent with what that document is supposed to be, and whether a memory write touches behavioral rules or configuration state it has no business touching. Any LLM-as-judge detector in this pipeline needs the same calibration discipline mentioned earlier: check it against human-labeled examples regularly, because the dataset it's scored against should keep growing, not stay frozen at whatever it looked like on day one.

The instrumentation has to capture hierarchical, causally linked spans, not just a flat log of prompts and responses. Without that structure, there's no way to trace an anomaly found in tool output back to the retrieved chunk that caused it, because standard APM tooling wasn't built to see tool-call failures or context-window anomalies in the first place; it needs agent-aware tracing to catch any of it. Platforms built for this in 2026 include Braintrust, LangSmith, Arize Phoenix, and AgentOps, the last of which tracks full agent session lifecycles, records every tool call and decision point, and supports replaying multi-agent workflows step by step.

Automated failure attribution still isn't reliable enough to trust on its own. The Who&When benchmark, an ICML 2025 Spotlight paper, found that the best automated method for identifying which agent caused a failure hit only 53.5% accuracy, and pinpointing the exact failure step dropped to 14.2%. That gap means manual trace review stays necessary for any confirmed injection incident, at least for now. Automation gets you the sample and the alert. A human still has to close the loop.

Converting confirmed injection attempts into regression signals

Every verified attack should turn into a labeled trace, and every labeled trace should become a regression case that gets checked against every future prompt or model change before it ships. That's the workflow, and it's the piece most teams skip.

Skipping it has a cost that compounds quietly. Without labeled production data pulled from real incidents, there's no way to quantify how many attacks got missed, no way to spot an emerging injection pattern before it's used a hundred times, and no way to know whether last week's prompt tweak or model swap quietly weakened protection that used to hold. Teams without this loop are measuring detection at a single point in time. They aren't tracking it as a property of the system that can drift.

Building a regression case out of a confirmed injection means capturing the full trace: the payload itself, which harness layer it came in through, what behavior it triggered in the model, and any tool calls or memory writes that followed. Label it by layer, user prompt, retrieved chunk, tool description, tool output, or memory write, because that label is what lets the regression catch the right failure mode the next time something similar happens. Then run the classifier against that case before any prompt, model, or retrieval change goes out. Pass, ship it. Fail, hold it.

Research on root-cause analysis of agent failures points to a trap: as execution logs get longer, automated judges tend to lock onto the first plausible-looking explanation instead of working through all the evidence. Iterative approaches that keep digging through unresolved parts of the log outperform single-pass review on long traces, and the same discipline applies directly to reviewing injection traces. Don't stop at the first explanation that fits.

Every harness change made in response to confirmed injection evidence, tightening a system prompt, adding a schema validator on a tool, cutting back memory write permissions, should be a reviewable, inspectable change that someone can explain later. The regression test is what makes it auditable: it's the proof that the fix actually closes the gap it claims to close.

The payoff compounds. Each new regression case narrows the distance between what the detection system was originally calibrated on and what production traffic is actually doing right now, and the calibration work described earlier stays anchored to real attacker behavior instead of drifting toward synthetic benchmarks that stopped reflecting reality months ago. Teams that treat every confirmed injection as a data point end up with detection that gets sharper with each incident and a growing, specific record of how their own agent harness is actually being targeted.