Sabot
Sabot is a crash-test harness for agent pipelines that plants controlled faults and scores how many are caught by the pipeline's own checks. It provides a frozen metric spec, a runnable tool, and a public scoreboard for frameworks like LangGraph, CrewAI, and AutoGen. It is for developers and researchers building or evaluating agent reliability.
✨ Key features
- Plants controlled faults in live agent pipelines
- Scores detection, reaction, and recovery with three verdicts
- Pre-registered spec and seeds for transparent results
- Offline deterministic scoring scripts (Python 3.9+, stdlib only)
- Publishes raw traces and full scoreboards
- Includes strict injection-evidence floor and uncertainty intervals
🎯 Use cases
- Benchmarking self-verification in agent frameworks
- Comparing detection rates across configurations
- Testing the impact of reviewer prompt changes
- Auditing agent reliability with reproducible fault injection
📦 Installation
🧰 Requirements: Python 3.9+ for offline scoring; Python 3.11+ and pinned framework virtualenvs to run the harness. No API keys needed for offline recompute.
git clone --depth 1 https://github.com/Jott2121/sabot && cd sabot/harness
No install step for offline scripts; harness setup via harness/scripts/setup_venvs.sh.
🚀 Usage
python3 scripts/score_strict.py --checkpython3 scripts/score_uncertainty.py --checkpython3 scripts/score_wave2.py --seeds-file ../seeds/wave2.json
git diff --stat runs/wave2/WAVE2-RESULTS.md⚠️ Good to know
Results are specific to the tested frameworks, model, and versions; replicate resampling cannot cover model-version drift or task-suite selection, and the scoreboard is not yet a ranking due to overlapping confidence intervals.
❓ FAQ
What is the Sabot Score?
It is the hard-tier detection rate: the fraction of planted faults that the pipeline's own components flag, adjudicated deterministically per the published mapping.
Can I recompute the results without API keys?
Yes, the offline scoring scripts are deterministic and use only Python 3.9+ standard library; raw traces are included in the repo.
What does the strict floor measure?
It counts only flags that quote injection-only tokens, deliberately under-counting paraphrase detections; the truth lies between the floor and the published number.
Why are the results not a ranking?
The five standard rows are not statistically distinguishable; every pairwise confidence interval contains zero, so the scoreboard does not rank frameworks.
📊 Repository
🤖 Overview, features, install steps and FAQ were generated from the project's README on Sep 4, 2026. Always check the original source before running commands.