Strands Evals (AWS)
Strands Evals is a Python framework for evaluating AI agents and LLM applications. It supports output validation, trajectory analysis, tool usage, multi-turn conversation simulation, and automated test generation. It is for developers building and testing agents with Strands.
✨ Key features
- Multiple evaluation types: output, trajectory, tool usage, interaction
- Multimodal evaluation with MLLM-as-a-Judge for image-to-text tasks
- Dynamic simulators for multi-turn conversations and tool behavior
- LLM-as-a-Judge evaluators with structured scoring rubrics
- Trace-based evaluation via OpenTelemetry execution traces
- Automated experiment generation from context descriptions
🎯 Use cases
- Validate agent responses against custom rubrics
- Analyze agent tool usage and action sequences
- Simulate realistic multi-turn user interactions
- Test tool-using agents without live APIs or services
- Generate comprehensive test suites from context descriptions
📦 Installation
🧰 Requirements: Python 3.10+; requires Strands Agent and optionally an LLM API key for judge models.
python -m venv .venv
source .venv/bin/activate # On Windows use: .venv\Scripts\activatepip install -e .pip install -e ".[test]"pip install -e ".[test,dev]"🚀 Usage
from strands import Agent
from strands_evals import Case, Experiment
from strands_evals.evaluators import OutputEvaluator
# Create test cases
test_cases = [
Case[str, str](
name="knowledge-1",
input="What is the capital of France?",
expected_output="The capital of France is Paris.",
metadata={"category": "knowledge"}
)
]
# Create evaluators with custom rubric
evaluators = [
OutputEvaluator(
rubric="""
Evaluate based on:
1. Accuracy - Is the information correct?
2. Completeness - Does it fully answer the question?
3. Clarity - Is it easy to understand?
Score 1.0 if all criteria are met excellently.
Score 0.5 if some criteria are partially met.
Score 0.0 if the response is inadequate.
"""
)
]
# Create experiment and run evaluation
experiment = Experiment[str, str](cases=test_cases, evaluators=evaluators)
def get_response(case: Case) -> str:
agent = Agent(callback_handler=None)
return str(agent(case.input))
# Run evaluations
report = experiment.run_evaluations(get_response)
report.run_display()
❓ FAQ
What Python version is required?
Python 3.10 or higher is required.
How do I install the SDK?
You can install via pip with pip install strands-agents-evals or from source with pip install -e ..
What evaluation types are supported?
Output evaluation, trajectory analysis, tool usage assessment, interaction evaluation, and trace-based helpfulness evaluation.
Can I simulate tools without real infrastructure?
Yes, the ToolSimulator allows you to simulate tool behavior with LLM-generated, schema-validated responses.
📊 Repository
🤖 Overview, features, install steps and FAQ were generated from the project's README on Sep 4, 2026. Always check the original source before running commands.