CIAgent
CIAgent is a pytest-native regression testing tool for AI agents that detects when eval scores are stable but individual verdicts are flaky. It runs eval suites multiple times, attributes verdict flips to the agent or the judge, audits LLM judges against ground truth, and replaces judge calls with deterministic fact-checks. It is for developers building and maintaining AI agents who want reliable, low-cost evaluation.
✨ Key features
- Runs eval suites N times to detect flaky verdicts
- Attributes flips to agent-variance, judge-flake, or infra-error
- Audits LLM judges against ground truth and deterministic checks
- Generates deterministic fact-checks from knowledge base
- Supports OpenAI, Anthropic, LangGraph, and imported traces
- Mock mode runs with zero API keys and zero cost
🎯 Use cases
- Detect flaky agent behavior in CI/CD pipelines
- Audit LLM judge reliability before trusting eval scores
- Replace expensive LLM judge calls with free deterministic checks
- Catch regressions in RAG agents when retrieval changes
- Set up automated agent testing with coding agents
📦 Installation
🧰 Requirements: Python 3.10 or later; API keys for OpenAI/Anthropic only when using live LLM judge; optional for mock mode.
pip install ciagent
🚀 Usage
ciagent test --mock --runs 3
⚠️ Good to know
The tool is Apache 2.0 licensed and built from a real study; it is not a replacement for human judgment on subjective answers.
❓ FAQ
How does CIAgent detect flaky verdicts?
It runs the eval suite multiple times (--runs N) and compares verdicts per query across runs. If a query's verdict flips, it attributes the flip to agent-variance (agent output changed), judge-flake (same output but judge changed mind), or infra-error (API call failed).
Can I use CIAgent without API keys?
Yes, use ciagent test --mock to run with synthetic traces and zero cost. You can also set CIAGENT_MOCK_FLAKY=1 to simulate flaky behavior.
What is the difference between agent-variance and judge-flake?
Agent-variance means the agent produced different output across runs, so you should fix the agent. Judge-flake means the output was identical but the LLM judge changed its verdict, so you should fix the eval or replace the judge with deterministic checks.
Does CIAgent support my agent framework?
It has native adapters for OpenAI, Anthropic, and LangGraph. It can also import exported traces from OpenTelemetry, Langfuse, and LangSmith, verified against real captures from OpenAI, Anthropic, CrewAI, Google ADK, and Claude Agent SDK.
📊 Repository
🤖 Overview, features, install steps and FAQ were generated from the project's README on Sep 4, 2026. Always check the original source before running commands.