AgentDojo
AgentDojo is a dynamic benchmark environment for evaluating prompt injection attacks and defenses in LLM agents. It provides a suite of tasks and tools to measure the security of agent-based systems. It is intended for security researchers and developers building LLM agents.
✨ Key features
- Benchmark suite for prompt injection attacks and defenses
- Supports multiple user tasks and suites
- Configurable defenses and attacks
- Command-line interface for running benchmarks
- Results inspection via web page and registry
🎯 Use cases
- Evaluate the robustness of LLM agents to prompt injection
- Compare different defense mechanisms
- Research on prompt injection attack strategies
- Benchmark agent security in development pipelines
📦 Installation
🧰 Requirements: Python environment with pip; optional transformers extra for prompt injection detector; API keys for LLM models (e.g., OpenAI) may be required.
pip install agentdojo
If you want to use the prompt injection detector, you need to install the transformers extra:
pip install "agentdojo[transformers]"
🚀 Usage
python -m agentdojo.scripts.benchmark -s workspace -ut user_task_0 \
-ut user_task_1 --model gpt-4o-2024-05-13 \
--defense tool_filter --attack tool_knowledge
⚠️ Good to know
The API of the package is still under development and might change in the future.
❓ FAQ
How do I install AgentDojo?
You can install it via pip: pip install agentdojo. For the prompt injection detector, install with the transformers extra: pip install "agentdojo[transformers]".
How do I run a benchmark?
Use the command-line script: python -m agentdojo.scripts.benchmark with options like -s for suite, -ut for user tasks, --model, --defense, and --attack. See --help for details.
Where can I see the results?
Results are available on the dedicated results page at https://agentdojo.spylab.ai/results/ and also in the Invariant Benchmark Registry.
Is the API stable?
No, the API is still under development and might change in the future.
📊 Repository
🤖 Overview, features, install steps and FAQ were generated from the project's README on Sep 4, 2026. Always check the original source before running commands.