τ²-bench (tau2-bench) v1.0.1
τ²-bench is a simulation framework for evaluating customer service agents across real-world domains like airline, retail, telecom, and banking. It supports text-based half-duplex and voice full-duplex evaluation, providing a policy, tools, and tasks for each domain. It is intended for developers and researchers building or benchmarking conversational AI agents.
✨ Key features
- Text half-duplex and voice full-duplex evaluation modes
- Multiple domains: mock, airline, retail, telecom, banking_knowledge
- Configurable RAG pipelines for knowledge retrieval domain
- Gymnasium-compatible RL interface for training
- Leaderboard submission and comparison at taubench.com
- CLI tools for running and viewing evaluations
🎯 Use cases
- Benchmark customer service agents on realistic tasks
- Evaluate voice agents with realtime audio APIs
- Train agents using reinforcement learning with gym interface
- Test knowledge retrieval pipelines in banking domain
- Compare model performance on public leaderboard
📦 Installation
🧰 Requirements: Requires Python >=3.12, <3.14, uv package manager, and API keys for LLM providers (via LiteLLM). Voice features need system dependencies like portaudio and ffmpeg.
git clone https://github.com/sierra-research/tau2-bench
cd tau2-bench
uv sync # core only (text-mode: airline, retail, telecom, mock)
Optional extras:
uv sync --extra voice # + voice/audio-native features
uv sync --extra knowledge # + banking_knowledge domain (retrieval pipeline)
uv sync --extra gym # + gymnasium RL interface
uv sync --extra dev # + pytest, ruff, pre-commit (required for contributing)
uv sync --all-extras # everything
🚀 Usage
cp .env.example .envtau2 run --domain airline --agent-llm gpt-4.1 --user-llm gpt-4.1 \
--num-trials 1 --num-tasks 5Results are saved to data/simulations/. Use tau2 view to browse them.
⚠️ Good to know
v1.0.1 grading update fixes banking_knowledge task errors, so results from versions <1.0.1 are not comparable; other domains unaffected.
❓ FAQ
What Python version is required?
Python >=3.12 and <3.14 is required, as stated in the README.
How do I install the knowledge domain?
Install with uv sync --extra knowledge to include the banking_knowledge domain.
Can I use any LLM provider?
Yes, the tool uses LiteLLM, so any supported provider works with your API keys.
How do I submit results to the leaderboard?
Follow the instructions in docs/leaderboard-submission.md to submit results to taubench.com.
📊 Repository
🤖 Overview, features, install steps and FAQ were generated from the project's README on Sep 4, 2026. Always check the original source before running commands.