whatbroke
whatbroke is a CLI tool that diffs AI agent behavior between two runs by comparing JSONL traces. It detects changes in tool calls, arguments, outputs, cost, and latency, helping developers identify regressions when swapping models, prompts, or frameworks. It is for developers building AI agents who need deterministic, offline regression testing.
✨ Key features
- Aligns runs and tool calls to report breaking, changed, and info findings
- Supports multi-sample runs to distinguish flaky behavior from real changes
- Provides proxy, Node, and Python recorders to capture traces without code changes
- Imports existing traces from OpenTelemetry, Langfuse, and LangSmith
- Behavior contracts to enforce guardrails on single traces
- Watch mode for live diffing while iterating on prompts
🎯 Use cases
- Compare agent behavior before and after swapping to a smaller or different model
- Detect silent tool call drops or argument drift in production agents
- Set up CI gates to fail on breaking changes in agent runs
- Monitor cost and latency regressions across agent versions
- Enforce behavior contracts on every run without a baseline
📦 Installation
🧰 Requirements: Requires Node.js for the CLI; Python recorder available via pip. No API keys or accounts needed.
npm install -g whatbroke-cli
Or run it directly:
npx whatbroke-cli diff before.jsonl after.jsonl
Try it right now with the bundled example traces:
git clone https://github.com/arthi-arumugam-git/whatbroke
cd whatbroke && npm install && npm run build
node dist/cli.js diff examples/support-agent-gpt4o.jsonl examples/support-agent-gpt5mini.jsonl
🚀 Usage
whatbroke diff old.jsonl new.jsonl
Example with recording:
whatbroke record --out traces/current.jsonl
OPENAI_BASE_URL=http://127.0.0.1:4141/v1 node my-agent.js
whatbroke diff traces/baseline.jsonl traces/current.jsonl
⚠️ Good to know
Semantic output comparison is on the roadmap but not yet implemented; tool arguments and model outputs may be missing from imported traces if not captured by the source instrumentation.
❓ FAQ
How does whatbroke handle flaky agents?
It supports multi-sample runs with suffixed run IDs (e.g., refund-flow#1) and compares every before sample against every after sample, reporting a rate for each finding. Findings that also occur between baseline samples are demoted to flaky info.
Can I use whatbroke without recording new traces?
Yes, you can import existing traces from OpenTelemetry, Langfuse, or LangSmith using whatbroke import, which converts them into diffable JSONL.
Does whatbroke require an API key or send data to the cloud?
No, it is deterministic, offline, and requires no API keys or accounts. Your traces never leave your machine.
How can I integrate whatbroke into CI?
The diff command exits with code 1 when breaking findings appear, so you can add it to CI. There is also a GitHub Action that outputs a markdown report to the job summary.
📊 Repository
🤖 Overview, features, install steps and FAQ were generated from the project's README on Sep 4, 2026. Always check the original source before running commands.