open-operator-evals
Open-operator-evals is an open-source benchmark for evaluating web agents. It provides a reproducible methodology to compare different open-source operators, addressing the issue of high variance in agent performance by running tasks multiple times. It is intended for developers and researchers who want to assess and compare web agent systems.
✨ Key features
- Reproducible benchmark with full transparency
- Runs each task 8 times to reduce variance
- Uses WebVoyager30, a 30-task subset across 15 websites
- Provides metrics: self-report, LLM evaluation, time, reliability
- Includes replays and logs for every run
- Open-source and accessible for verification
🎯 Use cases
- Compare performance of different web agents
- Verify claims made by agent providers
- Evaluate agent reliability and self-assessment accuracy
- Research on web agent evaluation methodologies
🚀 Usage
To reproduce the benchmark results yourself, simply run for a given config:
cat configs/no
(Note: The README is truncated at this point, so the full command is not available.)
⚠️ Good to know
The benchmark uses a subset of WebVoyager (30 tasks) and runs on a specific machine (Macbook M1) with a residential IP in Switzerland, which may affect results due to location-specific web content and cookie consent popups.
❓ FAQ
How does the benchmark handle the high variance in web agent performance?
It runs each task multiple times (8 times) and averages the results to smooth out randomness and provide a more statistically sound estimate.
What is WebVoyager30?
It is a 30-task subset of the WebVoyager dataset, sampled across 15 diverse websites, designed to reduce variance and improve reproducibility while retaining complexity.
What metrics are reported?
The benchmark reports Agent Self-Report, LLM Evaluation, Time per Task, and Task Reliability. It also provides Alignment and Mismatch in breakdowns.
Can I reproduce the results?
Yes, the entire evaluation is open source and reproducible. You can run the benchmark yourself using the provided configs and data.
📊 Repository
🤖 Overview, features, install steps and FAQ were generated from the project's README on Sep 4, 2026. Always check the original source before running commands.