DeepEval
DeepEval is an open-source LLM evaluation framework for unit testing large-language model systems, similar to Pytest but specialized for LLM apps. It provides a variety of metrics powered by LLM-as-a-judge or local NLP models to evaluate end-to-end apps, agent trajectories, and individual steps. It is for developers building AI agents, RAG pipelines, or chatbots with frameworks like LangChain or OpenAI.
✨ Key features
- Large variety of ready-to-use LLM eval metrics with explanations
- Supports end-to-end and component-level LLM evaluation
- Build custom metrics integrated with DeepEval's ecosystem
- Generate single and multi-turn synthetic datasets for evaluation
- Integrates seamlessly with any CI/CD environment
- Benchmark any LLM on popular benchmarks in under 10 lines
🎯 Use cases
- Evaluate RAG-based customer support chatbots end-to-end
- Test agent trajectories for task completion and tool correctness
- Detect hallucination, bias, or toxicity in LLM outputs
- Optimize prompts based on evaluation results
- Benchmark LLMs on MMLU, HellaSwag, and other benchmarks
📦 Installation
🧰 Requirements: Python 3.9+ and an OpenAI API key (or custom LLM) for metrics; optional account for cloud reporting.
pip install -U deepeval
🚀 Usage
import pytest
from deepeval import assert_test
from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase, SingleTurnParams
def test_case():
correctness_metric = GEval(
name="Correctness",
criteria="Determine if the 'actual output' is correct based on the 'expected output'.",
evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT, SingleTurnParams.EXPECTED_OUTPUT],
threshold=0.5
)
test_case = LLMTestCase(
input="What if these shoes don't fit?",
actual_output="You have 30 days to get a full refund at no extra cost.",
expected_output="We offer a 30-day full refund at no extra costs.",
retrieval_context=["All customers are eligible for a 30 day full refund at no extra costs."]
)
assert_test(test_case, [correctness_metric])
Then run: deepeval test run test_chatbot.py
❓ FAQ
What Python version is required?
DeepEval works with Python 3.9 or higher.
How do I install DeepEval?
Install via pip with pip install -U deepeval.
Can I use a custom LLM for evaluation?
Yes, you can evaluate using your own custom model; see the docs for details.
How do I run tests?
Write test cases using pytest and run them with the CLI command deepeval test run test_chatbot.py.
📊 Repository
🤖 Overview, features, install steps and FAQ were generated from the project's README on Sep 4, 2026. Always check the original source before running commands.