AutoEvals
Autoevals is a tool for evaluating AI model outputs using a variety of automatic methods, including LLM-as-a-judge, heuristics, and statistical metrics. It solves the problem of quickly and easily assessing the quality of model completions, and is designed for developers working with AI applications who need to test and compare outputs.
✨ Key features
- Bundles LLM-as-a-judge, heuristic, and statistical evaluation methods.
- Supports custom model-graded evaluation prompts and scoring.
- Works with OpenAI-compatible APIs and Braintrust Gateway.
- Available for Python and TypeScript.
- Includes pre-built evaluators for factuality, moderation, RAG, and more.
- Returns a consistent Score object with name, score, and metadata.
🎯 Use cases
- Evaluate the factuality of LLM responses.
- Compare two model outputs to choose the better one.
- Test custom criteria using LLM-as-a-judge with custom prompts.
- Integrate with Braintrust for logging and comparison of evaluation results.
- Create custom heuristic scorers for specific output checks.
📦 Installation
🧰 Requirements: Python 3.9+ or Node.js, with an OpenAI API key (OPENAI_API_KEY) or Braintrust API key (BRAINTRUST_API_KEY) for gateway access.
TypeScript
npm install autoevals
Python
pip install autoevals
🚀 Usage
Python
from autoevals.llm import *
import asyncio
evaluator = Factuality()
input = "Which country has the highest population?"
output = "People's Republic of China"
expected = "China"
result = evaluator(output, expected, input=input)
print(f"Factuality score: {result.score}")
print(f"Factuality metadata: {result.metadata['rationale']}")
TypeScript
import { Factuality } from "autoevals";
(async () => {
const input = "Which country has the highest population?";
const output = "People's Republic of China";
const expected = "China";
const result = await Factuality({ output, expected, input });
console.log(`Factuality score: ${result.score}`);
console.log(`Factuality metadata: ${result.metadata?.rationale}`);
})();
❓ FAQ
What are the requirements to use Autoevals?
You need Python 3.9+ or Node.js, and an OpenAI API key (OPENAI_API_KEY) or Braintrust API key (BRAINTRUST_API_KEY) for gateway access.
Can I use Autoevals with other AI providers?
Yes, Autoevals can use any OpenAI-compatible API by setting OPENAI_BASE_URL, or it can use the Braintrust Gateway which provides access to multiple providers.
How do I create a custom evaluation prompt?
You can use LLMClassifier (Python) or LLMClassifierFromTemplate (TypeScript) to define a custom prompt template and scoring mechanism.
What does the Score object contain?
The Score object contains the scorer name, a score between 0 and 1 (or null if skipped), and optional metadata with details like rationale.
📊 Repository
🤖 Overview, features, install steps and FAQ were generated from the project's README on Sep 4, 2026. Always check the original source before running commands.