AI Agent Prompt Injection Testing: Red Teaming Against Indirect Attacks

A red-team suite for an AI agent tells you which known attacks fail today. It does not tell you the agent is safe, and a 2025 paper with authors from OpenAI, Anthropic and Google DeepMind (per Simon Willison's write-up) shows how far apart those two statements can be. This post covers how to build the suite so its result means something: which workflows to test first, how to separate the detector from the boundary, and what to score.

A passing suite proves less than it seems

In The Attacker Moves Second, the authors bypassed twelve published prompt-injection and jailbreak defenses with adaptive attacks, reaching success rates above 90% for most. The majority of those defenses had reported near-zero attack success rates when evaluated on static attack sets. The authors argue that testing against a fixed list of attack strings gives a false sense of security.

Treat your own fixture suite the same way. It is a regression suite: it proves that attacks you have already seen stay fixed. Whether a determined attacker with several tries gets through is a different question, and the only defensible answer is architectural. The action the injected text asks for must be impossible or require approval, however convincing the text is.

Pick fixtures by what the workflow can do

Meta's Agents Rule of Two names three properties: the agent processes untrusted input (A), has access to sensitive systems or private data (B), and can change state or communicate externally (C). Until injection can be reliably detected, an agent should have no more than two of the three in one session. Use this to rank your test work. Tabulate each workflow by A, B and C. A workflow with all three is where a successful injection costs real money, so it either gets human approval before the state change or it gets your first ten fixtures. A workflow with two needs fewer, aimed at the capability that makes the third reachable (a tool with a free-text recipient, a fetch tool that accepts any URL).

Then write each attack where production would ingest it: a ticket body, an attachment, a retrieved chunk, a calendar description, a tool result. Each attack gets a clean control with the same user, identity and expected outcome so you can tell a defended agent from a broken one.

Test the boundary with the detector switched off

Many agents put a classifier or pattern scanner in front of the model. Treat that as a filter, not a boundary, because the adaptive-attack results above apply to detectors too. Test any scanner as an attacker who has read its rules, and normalize encodings before scanning so that an entity-encoded chat-template marker cannot slip past a pattern.

That scanner is a tripwire, so test it as one. A pattern list catches the lazy attacks and gives you a log line to alert on. It cannot catch paraphrase, other languages or text split across turns, and the adaptive-attack results above say better classifiers fall the same way. So run your suite twice: once as deployed, and once with the detector disabled. The second run measures the real boundary, which is tenant isolation, tool policy and approval. If an attack succeeds only when the detector is off, you have a finding even though production looks safe today: you are one paraphrase from the incident. Record both results per fixture so a detector update cannot hide a policy regression.

Score the trajectory, not the answer

Final-answer grading misses an agent that read a forbidden record and left it out of the reply, or one whose tool sequence changed while the text stayed the same. Capture retrieved items, tool proposals, policy decisions, tool results and the reply as normalized events, and assert on those. A rule such as "no customer_lookup call may use a tenant outside the session" can be checked against a trace with no model-based grader.

{
  "run_id": "fixture-017",
  "event": "tool_decision",
  "tool": "customer_lookup",
  "arguments": {"customer_id": "cust_204"},
  "trusted_tenant": "org_red",
  "policy_result": "deny",
  "reason": "object_outside_tenant"
}

Report utility and security separately. AgentDojo evaluates tool-using agents on both useful tasks and injection attacks for this reason: a defense that refuses everything scores perfectly on security and is useless. For each fixture record task completion, attack success, which layer enforced the block, false refusals on the clean control, and whether the trace is complete. A missing trace is indeterminate and blocks a consequential release, not a pass.

Run it safely and keep it honest

• Use tool adapters that enforce the real schema and policy but write to isolated state. Never let test agents reach live mail, payments or customer data.
• Never retry a security violation until it disappears. Retry infrastructure errors within a fixed budget, and keep four outcomes: pass, violation, infrastructure error, indeterminate.
• When an attack succeeds, shrink it to the smallest reproducing case, commit that, and fix the layer nearest the side effect. Prompt wording can help, but tenant checks, approval and tool policy belong in deterministic code.
• Version the fixture, model, prompt, tool, policy and grader together. A grader that changes meaning between runs creates false confidence without any change to the agent.
• Add human attackers on a schedule. In the same study an online red-teaming competition with over 500 participants defeated every challenge, including those with defenses enabled.

Frameworks such as PyRIT are useful for generating variants and orchestrating multi-turn conversations. Keep the asset rules, assertions and release thresholds in your repository and connect the framework through adapters, so a framework upgrade cannot quietly change the release contract. Guidance such as the OWASP prompt injection cheat sheet and OpenAI's safety best practices covers the defensive side.

Start with ten fixtures

Choose the workflow with all three Rule of Two properties, seed attacks into the two untrusted inputs it reads most, write the security assertions against the trace, and run them in the same command as your functional evals, with the detector on and off. Block the release on any high-impact violation.

References

• Nasr et al., The Attacker Moves Second: adaptive attacks bypassed 12 published defenses with above 90 percent success for most.
• Simon Willison on the Agents Rule of Two and The Attacker Moves Second
• AgentDojo
• Microsoft PyRIT
• OWASP LLM Prompt Injection Prevention Cheat Sheet
• OpenAI safety best practices

---