simulate-sdk
Simulate is a Python SDK for testing AI voice and text agents against persona-driven scenarios before real users interact with them. It captures transcripts and audio, and integrates with Future AGI's evaluation tools to score agent performance. It is designed for developers building and deploying AI agents who need to validate behavior in realistic conversations.
✨ Key features
- Runs voice agents via LiveKit with WebRTC and multi-turn conversation.
- Captures per-speaker and combined WAV recordings plus full transcripts.
- Orchestrates thousands of text conversations via cloud backend.
- Provides built-in wrappers for OpenAI, Anthropic, Gemini, LangChain.
- Evaluation-ready with 50+ metrics via evaluate_report helper.
- Supports custom agent wrappers for any framework.
🎯 Use cases
- Test a customer support voice agent with simulated frustrated users.
- Run large-scale text simulations against an LLM-based agent.
- Evaluate agent responses for task completion and audio quality.
- Capture transcripts and audio for regression testing.
- Integrate simulation results into an evaluation pipeline.
📦 Installation
🧰 Requirements: Python 3.10–3.13. For voice mode, need LiveKit server credentials and OpenAI API key; for cloud mode, need Future AGI API keys. Optional Silero VAD download for voice.
pip install agent-simulatepip install "agent-simulate[livekit]"pip install "agent-simulate[evaluation]"pip install "agent-simulate[all]"Requires Python 3.10–3.13.
Voice mode: download Silero VAD weights (one time)
from livekit.plugins import silero
if __name__ == "__main__":
silero.VAD.load()
🚀 Usage
import asyncio
import os
from dotenv import load_dotenv
from fi.simulate import AgentDefinition, Scenario, Persona, TestRunner
from fi.simulate.evaluation import evaluate_report
load_dotenv()
async def main():
agent = AgentDefinition(
name="my-support-agent",
url=os.environ["LIVEKIT_URL"],
room_name="support-room",
system_prompt="Helpful support agent",
)
scenario = Scenario(
name="Password Reset",
dataset=[
Persona(
persona={"name": "Alice", "mood": "frustrated"},
situation="She cannot log into her account.",
outcome="The agent should guide her through a password reset.",
),
],
)
runner = TestRunner()
report = await runner.run_test(
agent_definition=agent,
scenario=scenario,
record_audio=True,
)
for r in report.results:
print(r.transcript)
print(r.audio_combined_path)
evaluated = evaluate_report(
report,
eval_specs=[
{"template": "task_completion",
"map": {"input": "persona.situation", "output": "transcript"}},
{"tem
⚠️ Good to know
Cloud mode returns an intentionally empty local TestReport; metrics live in the backend. Some features like conversation-graph scenarios and adversarial personas are on the roadmap but not yet shipped.
❓ FAQ
What Python versions are supported?
Python 3.10 through 3.13 are required.
How do I run voice simulations?
Install the 'livekit' extra, set LIVEKIT_URL, LIVEKIT_API_KEY, LIVEKIT_API_SECRET, and OPENAI_API_KEY, and use the TestRunner with an AgentDefinition pointing to your LiveKit room.
Can I use my own agent framework?
Yes, you can subclass AgentWrapper to wrap any custom agent, or use built-in wrappers for OpenAI, Anthropic, Gemini, and LangChain.
How do I evaluate the simulation results?
Use the evaluate_report helper from fi.simulate.evaluation, which sends results to Future AGI's evaluation SDK with 50+ metrics.
📊 Repository
🤖 Overview, features, install steps and FAQ were generated from the project's README on Sep 4, 2026. Always check the original source before running commands.