AgentBench
AgentBench is a benchmark for evaluating LLMs as autonomous agents across diverse environments, including operating systems, databases, knowledge graphs, and web tasks. It provides standardized tasks and a framework to test and compare agent performance. It is intended for researchers and developers working on LLM-based agents.
✨ Key features
- Evaluates LLMs as agents across 8 distinct environments
- Includes 5 new domains and 3 recompiled datasets
- Provides dev and test splits for each task
- Supports Docker-based task deployment
- Offers a leaderboard for comparing model results
- Integrates with AgentRL for function-calling version
🎯 Use cases
- Benchmarking LLM agents on operating system tasks
- Evaluating agents on database querying tasks
- Testing agents in knowledge graph reasoning
- Assessing agents in web shopping and browsing scenarios
- Comparing performance of different LLMs as agents
📦 Installation
🧰 Requirements: Python 3.9, Docker, and an OpenAI API key (for using OpenAI models).
cd AgentBench
conda create -n agent-bench python=3.9
conda activate agent-bench
pip install -r requirements.txt
Ensure Docker is installed and running: docker ps.
Build required images for dbbench-std and os-std:
docker pull mysql
docker pull ubuntu
docker build -f data/os_interaction/res/dockerfiles/default data/os_interaction/res/dockerfiles --tag local-os/default
docker build -f data/os_interaction/res/dockerfiles/packages data/os_interaction/res/dockerfiles --tag local-os/packages
docker build -f data/os_interaction/res/dockerfiles/ubuntu data/os_interaction/res/dockerfiles --tag local-os/ubuntu
🚀 Usage
python -m src.start_task -apython -m src.assigner⚠️ Good to know
The webshop environment requires ~16GB RAM, and the alfworld implementation leaks memory and disk space until the task worker is restarted.
❓ FAQ
What Python version is recommended?
Python 3.9 is recommended due to pinned older scientific dependencies like numpy~=1.23.x.
How do I run a specific task?
You can configure tasks in the config files and use the provided scripts. For example, to start tasks, run python -m src.start_task -a and then python -m src.assigner.
Can I use models other than OpenAI's?
Yes, you can configure other agents by modifying the agent config files and using the --agent parameter in the agent test script.
What is the difference between the original AgentBench and the FC version?
The FC version uses a function-calling style prompt and adds fully-containerized deployment support for five tasks, integrated with AgentRL.
📊 Repository
🤖 Overview, features, install steps and FAQ were generated from the project's README on Sep 4, 2026. Always check the original source before running commands.