AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents

AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents. LLM stands for large language model. Edoardo Debenedetti and colleagues presented this work at NeurIPS in 2024. They built AgentDojo to test a practical question: can a tool-using agent finish the user’s legitimate job without carrying out attacker-chosen actions hidden in retrieved data? By the end, you’ll understand how the benchmark measures both outcomes, what its experiments found, and why no single score should be treated as permanent.

An indirect prompt injection is an attacker-controlled instruction embedded in data returned by a tool. Imagine a tool returning a document that quietly tells the agent to perform some other action. The danger is not limited to misleading text: in AgentDojo’s threat model, the injected instruction can steer the agent to act with the user’s authority. A useful evaluation therefore has to observe both whether the legitimate request was completed and whether the environment was changed in the attacker’s favor.

AgentDojo turns that tension into three measurements. Benign utility asks whether the agent succeeds when nobody attacks it. Utility under attack requires the legitimate task to be completed with no specified adversarial side effect. Targeted attack success rate tracks whether the targeted attack succeeds. These measurements must be read together, because a defense can appear secure simply by refusing useful work.

The framework represents the work as a mutable environment with tools, legitimate user tasks and attacker goals. It then evaluates agent pipelines, attacks and defenses with fixed pass-or-fail checks. The initial environments imitate office work in Workspace and Slack, as well as travel booking and electronic banking, using synthetic records. These form 629 cases in which a legitimate task and an attacker goal can interact. A known-correct sequence of tool calls identifies which retrieved content can carry an injection. After each run, the checks inspect both the agent’s output and changes to the environment, avoiding a model-based grader that an injection might influence.

Across the evaluated model configurations, greater task capability tended to come with greater success at executing attacker goals. Most models also lost 10 to 25 percentage points of task utility when attacked. This was not a simple winner-takes-all ranking. Claude 3.5 Sonnet completed more benign work than GPT-4o in the reported results, while GPT-4o had higher targeted attack success. Capability and resistance therefore have to be evaluated separately.

The defense experiments make the trade-off concrete. Some defenses changed how the prompt was presented by marking untrusted data or repeating the user’s request. Others tried to detect injections or expose only tools judged relevant to the task. For GPT-4o, tool filtering sharply reduced targeted attack success. The detector also suppressed attacks, but at the cost of a steep drop in benign utility. A low attack-success result can therefore reflect a system that aborts or refuses more legitimate work rather than one that safely completes it.

These scores do not establish how a defense will behave in every deployment. One set of limitations concerns realism and coverage: the benchmark uses synthetic, static records, manually authored tasks and checks, and limited task diversity. It also excludes multimodal agents and users assigning multiple tasks over time without a context reset. Another limitation concerns tool filtering. It fails when legitimate and malicious goals need the same tools, which occurs in 17% of the benchmark cases. Finally, an implementation bug was fixed across versions, and revised prose conflicted with revised results. Score comparisons therefore require exact version pinning.

For detection engineers and teams building tool-using agents, AgentDojo is best treated as a repeatable test harness rather than a certification. Extend it with tools, tasks and data changes resembling the intended deployment. Include attacks adapted to the defense, which the framework explicitly supports, and report benign utility, utility under attack and targeted success together. If relying on tool filtering, test workflows where benign and malicious actions require the same tool, plus sessions that accumulate multiple user tasks. That operational advice follows from the benchmark’s documented failure modes; it is not a demonstrated guarantee of safety.

AgentDojo provides an executable way to test two outcomes: whether agents finish legitimate work and whether they avoid attacker-selected side effects in mutable tool environments. Fixed checks examine both outputs and state changes. The measurements show that stronger task performance did not imply resistance to injection. They also show that some defenses lowered attack success by sacrificing useful work. Agent builders, detection engineers and security architects can use this approach to structure evaluations, but they should not infer production safety from one synthetic benchmark run or compare scores without pinning versions.