Agent Memory Benchmark (AMB)
AMB is an open benchmark for evaluating agent memory systems. It measures accuracy, speed, and token cost across datasets that include agentic tasks like tool calls and document research. It is for developers building or comparing memory architectures for AI agents.
✨ Key features
- Open-source benchmark with published datasets, prompts, and scoring logic.
- Measures accuracy, retrieval time, and token cost.
- Includes agentic datasets beyond simple chatbot history.
- Supports multiple memory providers and retrieval modes.
- Provides oracle mode to isolate generation quality.
- Includes a live leaderboard and local result viewer.
🎯 Use cases
- Compare memory providers (e.g., BM25 vs. Hindsight Cloud) on standard datasets.
- Evaluate the cost-effectiveness of a memory system for a specific agent workload.
- Test the impact of retrieval noise on agent performance using PrecisionMemBench.
- Reproduce published benchmark results to verify claims.
📦 Installation
🧰 Requirements: Python ≥ 3.11, a GEMINI_API_KEY, and optionally MEMBENCH_DATA_PATH for MemBench.
# Copy and fill in your API key
cp .env.example .env # or just create .env with:
# GEMINI_API_KEY=...
🚀 Usage
uv run amb run --dataset personamem --domain 32k --memory bm25uv run amb run --dataset personamem --domain 32k --memory bm25 --query-limit 20uv run amb run --dataset personamem --domain 32k --memory bm25 --oracle❓ FAQ
What does AMB measure?
AMB measures accuracy, retrieval time, and token cost for memory systems across various datasets.
How does the benchmark work?
It ingests documents into a memory provider, retrieves context for each query, generates an answer with a Gemini model, and judges the answer against gold answers with a second Gemini call.
What is PrecisionMemBench?
PrecisionMemBench is a dataset that asserts exactly which memories must and must not be retrieved; it runs in retrieval mode without answer generation or judging.
How can I see results?
Results are saved to output files and can be browsed with uv run amb view.
📊 Repository
🤖 Overview, features, install steps and FAQ were generated from the project's README on Sep 4, 2026. Always check the original source before running commands.