AgentLab
AgentLab is a framework for developing and evaluating web agents on benchmarks supported by BrowserGym. It solves the problem of running large-scale parallel agent experiments and provides building blocks for creating agents. It is intended for researchers and developers working on web agent research.
✨ Key features
- Easy large scale parallel agent experiments using ray
- Building blocks for making agents over BrowserGym
- Unified LLM API for OpenRouter, OpenAI, Azure, or self-hosted using TGI
- Preferred way for running benchmarks like WebArena
- Various reproducibility features
- Unified LeaderBoard across benchmarks
🎯 Use cases
- Run large-scale evaluations of web agents on benchmarks like WebArena and WorkArena
- Develop and test new web agent architectures
- Reproduce and compare agent results across different benchmarks
- Visualize agent traces and actions with AgentXray
📦 Installation
🧰 Requirements: Python 3.11 or 3.12, Playwright installed, and API keys for OpenAI, OpenRouter, or Azure depending on the LLM used.
pip install agentlab
If not done already, install Playwright:
playwright install
Make sure to prepare the required benchmark according to the instructions provided in the setup column.
export AGENTLAB_EXP_ROOT=<root directory of experiment results> # defaults to $HOME/agentlab_results
export OPENAI_API_KEY=<your openai api key> # if openai models are used
🚀 Usage
from agentlab.agents.generic_agent import AGENT_4o_MINI
from agentlab.experiments.study import make_study
study = make_study(
benchmark="miniwob", # or "webarena", "workarena_l1" ...
agent_args=[AGENT_4o_MINI],
comment="My first study",
)
study.run(n_jobs=5)
⚠️ Good to know
AgentLab is meant to provide an open, easy-to-use and extensible framework to accelerate the field of web agent research, not a consumer product; use with caution.
❓ FAQ
What Python versions are supported?
AgentLab requires Python 3.11 or 3.12.
How do I set up API keys?
Set environment variables like OPENAI_API_KEY, OPENROUTER_API_KEY, or AZURE_OPENAI_API_KEY and AZURE_OPENAI_ENDPOINT depending on the LLM provider you use.
Can I run experiments in parallel?
Yes, AgentLab uses Ray for parallel execution. You can specify the number of jobs with n_jobs in study.run().
How can I visualize experiment results?
Use the agentlab-xray command to launch a Gradio interface to visualize traces of your agents.
📊 Repository
🤖 Overview, features, install steps and FAQ were generated from the project's README on Sep 4, 2026. Always check the original source before running commands.