This recipe turns indirect prompt injection into two numbers on a release gate. You run a tool-using agent through AgentDojo's banking suite twice, once with no attack and once with attacker text injected into the tool results the agent reads, and you report what a serious agentic-safety eval always reports as a pair: whether the agent still did the user's job (utility), and how often it did the attacker's job instead (attack success rate). Then you switch on a defense and watch both numbers move. AgentDojo is the right harness to build on: it is peer-reviewed, MIT-licensed, still maintained, and its scoring is state-based rather than transcript-based 1. The concepts behind it live in the agentic security chapter.
The hour: fifteen minutes to install and run the clean baseline, fifteen to run one attack and read the two numbers, thirty to add a defense and wire the metric into a script you can rerun.
What you build
Nothing, at first: AgentDojo ships the suites, the attacks, and the defenses. You run its benchmark CLI three times (no attack, attack, attack-plus-defense), then write one small aggregation script, asr_report.py, that reads the run logs and prints utility and attack success rate side by side, because the harness's own summary line is named in a way that will mislead you (more on that below).
Step 1: install and run a clean baseline
python -m venv .venv && source .venv/bin/activate
pip install agentdojo # PyPI; or: pip install git+https://github.com/ethz-spylab/agentdojo.git for HEAD
export OPENAI_API_KEY=... # for the OpenAI model entries; a local endpoint needs no key
# Baseline: the banking suite, no attack. This measures plain task utility.
python -m agentdojo.scripts.benchmark -s banking --model GPT_4O_MINI_2024_07_18
The banking suite has the fewest user tasks, 16 against 9 injection tasks, which is why it is the right place to start; the full cross of user tasks against injection tasks under one attack is 144 runs. The --model flag takes a fixed list (run --help to see it). In agentdojo 0.1.35, the latest PyPI release (October 2025), the newest entries are CLAUDE_3_7_SONNET_20250219 and GEMINI_2_5_PRO_PREVIEW_05_06, and the list has aged badly. Every Claude entry is a Claude 3.x model that Anthropic has retired on the Claude API, Claude 3.7 Sonnet on 2026-02-19 2. Of the OpenAI entries, GPT_4O_2024_05_13, GPT_4_TURBO_2024_04_09, and GPT_3_5_TURBO_0125 shut down on 2026-10-23 and GPT_4_0125_PREVIEW went on 2026-03-26 3. That leaves GPT_4O_MINI_2024_07_18, which the commands below use; it is not on OpenAI's deprecation list as of 2026-10-01.
Note the format: under click 8.2 and later (8.5.0 as of 2026-10-01), the CLI accepts the uppercase enum member names shown by --help, not the dashed model ids. A dashed id fails with "is not one of ...", and omitting --model fails the same way, because the built-in default is a dashed id. The Python API below still uses the dashed strings.
For a current frontier model or your own deployment, point the harness at a local OpenAI-compatible endpoint with --model LOCAL (it reads LOCAL_LLM_PORT, default 8000, and takes --model-id for the served model name), which is also how you evaluate the model you actually ship. The repository's main branch has since added an OPENAI_COMPATIBLE entry (merged June 2026) that reads OPENAI_COMPATIBLE_BASE_URL and OPENAI_COMPATIBLE_API_KEY, requires --model-id, and so reaches a hosted gateway rather than only localhost. It is not in the 0.1.35 wheel, so install from git to get it 4.
The clean run prints a line like Average utility: 81.25%. That is your ceiling: the fraction of user tasks the agent completes when nobody is attacking it. A defense that drops this number is charging you utility, and the whole point of the exercise is to see that bill.
Step 2: run an attack
python -m agentdojo.scripts.benchmark -s banking --model GPT_4O_MINI_2024_07_18 \
--attack important_instructions
important_instructions is AgentDojo's strong attack: it wraps the injected payload in official-looking delimiters and addresses the model by name, which works far better than a naive "ignore previous instructions." Attack strength changes the result completely, so the attack name is part of the number, the same rule that governs HarmBench reporting 5. The injected instruction in the banking suite pursues concrete goals like sending a small transaction to an attacker-controlled account, so a success is a real state change, not a suggestive sentence in the transcript.
CAUTION
The harness prints Average security: NN%, and that name points the wrong way. Under the hood AgentDojo records a per-case security flag that is True when the injection goal was achieved, so its "Average security" number rises as the agent gets less safe. It is an attack success rate wearing the opposite label. Do not paste that line into a report. Compute the rate yourself, with the direction stated, as in Step 3.
Step 3: report utility and attack success as a pair
AgentDojo writes one JSON log per run under ./runs, and the aggregation below replays those logs through the Python API, which returns a SuiteResults object whose utility_results and security_results are dictionaries keyed by (user_task_id, injection_task_id). Aggregate them explicitly:
"""Report utility and attack success rate for one AgentDojo run.
Usage:
python asr_report.py banking gpt-4o-mini-2024-07-18 important_instructions
"""
import sys
from pathlib import Path
from agentdojo.agent_pipeline import AgentPipeline, PipelineConfig
from agentdojo.attacks import load_attack
from agentdojo.benchmark import benchmark_suite_with_injections
from agentdojo.task_suite.load_suites import get_suite
suite_name, model, attack_name = sys.argv[1], sys.argv[2], sys.argv[3]
suite = get_suite("v1.2.2", suite_name)
pipeline = AgentPipeline.from_config(
PipelineConfig(llm=model, model_id=None, defense=None, system_message_name=None, system_message=None)
)
attack = load_attack(attack_name, suite, pipeline)
# Point logdir at the same directory the CLI wrote. Completed (user task,
# injection task) pairs are read back from those logs; only missing pairs
# execute, so running this after Step 2 costs nothing extra.
results = benchmark_suite_with_injections(
pipeline, suite, attack=attack, logdir=Path("./runs"), force_rerun=False
)
utility = list(results["utility_results"].values())
security = list(results["security_results"].values()) # True == the injection goal was achieved
util_rate = sum(utility) / len(utility)
asr = sum(security) / len(security) # attack SUCCESS rate: higher is worse
print(f"suite={suite_name} model={model} attack={attack_name}")
print(f"utility (user task still done): {util_rate:6.1%}")
print(f"attack success rate (attacker won): {asr:6.1%}")
The two lines are the deliverable, and they only mean something together. Utility alone hides the exploit; attack success rate alone rewards an agent that refuses to do anything. A useful defense pushes attack success rate down while holding utility up, and a change that drives attack success to zero by driving utility to zero is not a defense, it is a broken agent. That is the frontier the chapter describes, and this pair of numbers is one point on it.
Figure: AgentDojo scores indirect prompt injection on post-run environment state, not transcripts: one banking-suite run under the important_instructions attack yields two numbers, utility (the user task still completed) and attack success rate (the injected goal achieved), which only mean something when reported as a pair.
Step 4: add a defense and move the frontier
python -m agentdojo.scripts.benchmark -s banking --model GPT_4O_MINI_2024_07_18 \
--attack important_instructions --defense spotlighting_with_delimiting
spotlighting_with_delimiting wraps untrusted tool output in explicit delimiters and tells the model to treat everything inside as data, never as instructions. It is the harness version of the guidance every major vendor documents: deliver untrusted content to the model inside clearly labeled regions, never in the system prompt. Rerun asr_report.py (build the pipeline with defense="spotlighting_with_delimiting") and compare all four numbers to the undefended run. You will typically see attack success rate fall and utility dip a little; that delta, attack-success reduction bought at some utility cost, is exactly what a release review needs to see. AgentDojo also ships tool_filter, transformers_pi_detector, and repeat_user_prompt as comparison arms.
Read the defended number as a floor on your exposure, not a certificate. A fixed attack like important_instructions was written without seeing your defense. "The Attacker Moves Second" (Nasr, Carlini, and colleagues, October 2025) took 12 recent defenses against jailbreaks and prompt injections and attacked them adaptively, with gradient descent, reinforcement learning, random search, and human-guided exploration. It reports attack success above 90% for most, on defenses that had published near-zero 6. For anything that gates real tools, add an adaptive pass to the release review, a person or a search loop that can read the defense, and keep the static AgentDojo run for what it does well: a cheap regression gate on every change.
Adapting to your product
Swap in your agent by pointing --model LOCAL at your deployment, then move from the public suites to your own. Enumerate the untrusted channels your agent reads (retrieved documents, web pages, emails, tickets, and every third-party MCP server it connects to, where the payload can hide in a tool description rather than a tool result 7), and write injection tasks whose goals are the state changes that would actually hurt: exfiltrate a named secret, call a named dangerous tool, alter a named record. AgentDojo registers a custom injection task with a security() method that inspects the post-run environment, and a custom attack as a BaseAttack subclass loaded with -ml your_module. Score every case on the state change, keep a utility twin so a product-breaking defense is visible, and promote each confirmed injection into the standing corpus that runs on release next to your OWASP LLM01 and over-refusal numbers 8.