ClawBench
ClawBench is an open-source benchmark that evaluates AI browser agents on everyday online tasks such as booking travel, ordering food, and applying for jobs across live websites. It measures end-to-end task success with a five-layer recording pipeline and an agentic evaluator that compares runs against human references. It is for developers and researchers building or testing AI agents that operate web browsers.
✨ Key features
- 281 tasks across 163 live websites in 15 life categories
- Five-layer recording pipeline captures actions, requests, screenshots, and more
- Agentic evaluator compares agent runs against human references
- Supports multiple harnesses: OpenClaw, Hermes, Claude Code, browser-use, and more
- Runs in isolated Docker or Podman containers with live noVNC viewing
- CLI tools for running, batch evaluation, analysis, and rescoring
🎯 Use cases
- Benchmark AI agents on real-world tasks like booking travel or ordering food
- Compare different agent harnesses and models on the same task suite
- Develop and test new browser agents against a standardized corpus
- Generate human reference runs for evaluation
- Analyze agent failure modes and performance across task categories
📦 Installation
🧰 Requirements: Python 3.11+, uv, and a container engine (Docker or Podman); API key for the judge model (deepseek-v4-pro) for scoring.
uv tool install clawbench-eval
Alternatively, pipx install clawbench-eval or python -m pip install clawbench-eval work too. For source checkout:
git clone https://github.com/TIGER-AI-Lab/ClawBench.git && cd ClawBench && ./run.sh
Prerequisites: Python 3.11+, uv, and Docker or Podman.
🚀 Usage
clawbench
$EDITOR models/models.yamluv run clawbench-run test-cases/v1/001-daily-life-food-uber-eats claude-sonnet-4-6clawbench-batch --models your-model --cases-suite v2 --all-cases⚠️ Good to know
The benchmark reports that even the best AI agent only completes about 1 in 3 tasks, indicating significant room for improvement.
❓ FAQ
What is the top score achieved on ClawBench?
The top score to date is 33.3%.
How do I watch an agent run in real time?
When you run a task, the script prints a noVNC URL (e.g., http://localhost:6080/vnc.html) that you can open in a browser to watch the agent operate.
Can I use my own model?
Yes, you configure models in models/models.yaml, and then use the model key with clawbench-run or clawbench-batch.
What is the difference between V1 and V2?
V1 contains 152 tasks across 143 live sites, and V2 contains 129 tasks across 63 live sites. They are separate corpora with different task sets.
📊 Repository
🤖 Overview, features, install steps and FAQ were generated from the project's README on Sep 4, 2026. Always check the original source before running commands.