Harbor
Harbor is a framework for evaluating and optimizing agents and language models in container environments. It solves the problem of running benchmarks and experiments across many parallel environments, and is intended for developers working with agents like Claude Code, OpenHands, or Codex CLI.
✨ Key features
- Evaluate arbitrary agents like Claude Code, OpenHands, Codex CLI.
- Build and share custom benchmarks and environments.
- Run experiments in thousands of parallel environments via providers.
- Generate rollouts for reinforcement learning optimization.
- Official harness for Terminal-Bench-2.0.
- Supports third-party benchmarks like SWE-Bench and Aider Polyglot.
🎯 Use cases
- Run Terminal-Bench-2.0 to evaluate agent performance.
- Compare different agents and models on standard benchmarks.
- Create custom benchmarks for specific agent tasks.
- Scale agent evaluation to hundreds of concurrent environments.
- Generate training data for RL optimization of agents.
📦 Installation
🧰 Requirements: Requires Python or uv, Docker for local runs, and API keys for providers (e.g., ANTHROPIC_API_KEY, DAYTONA_API_KEY) depending on the environment.
uv tool install harbor
or
pip install harbor
🚀 Usage
export ANTHROPIC_API_KEY=<YOUR-KEY>
harbor run --dataset terminal-bench@2.0 \
--agent claude-code \
--model anthropic/claude-opus-4-1 \
--n-concurrent 4
⚠️ Good to know
No explicit limitations are stated in the README; it is a framework from the creators of Terminal-Bench.
❓ FAQ
What is Harbor?
Harbor is a framework for evaluating and optimizing agents and language models, created by the Terminal-Bench team.
How do I install Harbor?
You can install it with uv tool install harbor or pip install harbor.
How do I run a benchmark with Harbor?
Use the harbor run command with options like --dataset, --agent, --model, and --n-concurrent. For example, harbor run --dataset terminal-bench@2.0 --agent claude-code --model anthropic/claude-opus-4-1 --n-concurrent 4.
Can I run benchmarks on cloud providers?
Yes, pass the --env flag with a provider name like daytona and set the corresponding API key.
📊 Repository
🤖 Overview, features, install steps and FAQ were generated from the project's README on Sep 4, 2026. Always check the original source before running commands.