Agent Learning Kit
AI-Evaluation SDK is a Python library for evaluating LLM outputs with 72 local metrics, guardrail scanners, streaming assessment, and cloud scoring. It solves the problem of LLMs passing evals but hallucinating in production by providing comprehensive evaluation tools. It is for developers building and deploying LLM applications who need robust testing and monitoring.
✨ Key features
- Unified evaluate() API with 72 local metrics
- LLM-as-Judge augmentation with Gemini/GPT/Claude
- Guardrail scanners for jailbreak, injection, PII in <10ms
- Streaming assessment with early-stop on safety violations
- AutoEval pipelines from description or template
- Feedback loop with ChromaDB for few-shot learning
🎯 Use cases
- Catch hallucinating medical chatbots
- Evaluate RAG pipelines for retrieval and generation quality
- Block prompt injection attacks in real-time
- Monitor streaming LLM output for toxicity
- Trace quality scores in OpenTelemetry backends
📦 Installation
🧰 Requirements: Python 3.10+; optional API keys for cloud features or LLM-as-Judge.
pip install ai-evaluation
Optional extras:
pip install ai-evaluation[nli] # DeBERTa NLI model for faithfulness/hallucination
pip install ai-evaluation[embeddings] # sentence-transformers for embedding similarity
pip install ai-evaluation[feedback] # ChromaDB for feedback loop
pip install ai-evaluation[celery] # Celery distributed backend
pip install ai-evaluation[ray] # Ray distributed backend
pip install ai-evaluation[temporal] # Temporal distributed backend
pip install ai-evaluation[all] # Everything
🚀 Usage
from fi.evals import evaluate
# Local metric — no API keys, sub-second
result = evaluate("faithfulness",
output="Take 200mg ibuprofen every 4 hours.",
context="Ibuprofen: 200mg q4h PRN. Max 1200mg/day.",
)
print(result.score) # 0.0 - 1.0
print(result.passed) # True/False
print(result.reason) # Explanation
❓ FAQ
What are the requirements to use the SDK?
Python 3.10+ is required. Optional extras are available for NLI, embeddings, feedback, and distributed backends.
Do I need API keys for local metrics?
No, local metrics run entirely on your machine with no API keys or network calls.
How can I use LLM-as-Judge?
Set augment=True and specify a model like gemini/gemini-2.5-flash to refine local heuristic scores with an LLM.
Can I use the SDK in TypeScript?
Yes, there is a TypeScript SDK available via npm install @future-agi/ai-evaluation.
📊 Repository
🤖 Overview, features, install steps and FAQ were generated from the project's README on Sep 4, 2026. Always check the original source before running commands.