Inspect AI
Inspect AI is a Python framework for evaluating large language models, created by the UK AI Security Institute. It provides built-in components for prompt engineering, tool usage, multi-turn dialog, and model-graded evaluations, and supports extensions via additional Python packages. It is intended for developers and researchers who need to run standardized or custom evaluations on LLMs.
✨ Key features
- Built-in components for prompt engineering and tool usage
- Supports multi-turn dialog and model-graded evaluations
- Extensible via Python packages for new techniques
- Includes over 200 pre-built evaluations ready to run
- Comprehensive documentation and structured index for coding agents
🎯 Use cases
- Run standardized safety or capability evaluations on any LLM
- Develop custom evaluation scenarios with multi-turn interactions
- Integrate new scoring or elicitation techniques via extensions
- Automate model evaluation pipelines in research or production
📦 Installation
To work on development of Inspect, clone the repository and install with the -e flag and [dev] optional dependencies:
git clone https://github.com/UKGovernmentBEIS/inspect_ai.git
cd inspect_ai
pip install -e ".[dev]"
Alternatively, if you use uv, sync the development environment from the checked-in lockfile:
uv sync --extra dev
The uv workflow is supported but not required. The uv.lock file records a reproducible development resolution; project dependencies are still declared in requirements*.txt and exposed through pyproject.toml. When changing dependencies, update the appropriate requirements file and refresh the lockfile rather than relying on uv add.
Optionally install pre-commit hooks via
make hooks
Run linting, formatting, and tests via
make check
make test
When working in a uv-managed environment, prefix those commands with uv run (for example, uv run make check).
🚀 Usage
The README does not provide a minimal usage example for running evaluations. It directs users to the documentation at https://inspect.aisi.org.uk/ for getting started.
❓ FAQ
How do I install Inspect for development?
Clone the repository and run pip install -e ".[dev]" or use uv sync --extra dev if you use uv.
Can I extend Inspect with custom evaluation techniques?
Yes, Inspect is extensible via other Python packages that can provide new elicitation and scoring techniques.
Are there pre-built evaluations available?
Yes, Inspect includes over 200 pre-built evaluations ready to run on any model, as described at https://inspect.aisi.org.uk/evals/.
Do I need to work on the TypeScript frontend to contribute?
No, the frontend development steps are only needed if you plan to work on the TypeScript/React frontend; Python-only contributors can skip them.
📊 Repository
🤖 Overview, features, install steps and FAQ were generated from the project's README on Sep 4, 2026. Always check the original source before running commands.