PerspectiveGap
PerspectiveGap is a benchmark for evaluating LLMs' ability to compose prompts for multi-agent systems. It provides data and scripts for rendering, scoring, and computing metrics. It is intended for researchers and developers working on multi-agent prompt engineering.
✨ Key features
- Benchmark data for multi-agent prompt composition tasks
- Scripts for rendering evaluations and calling model APIs
- Scoring and metric computation for predictions
- Supports multiple providers: OpenAI, Anthropic, DeepSeek, etc.
- Resume capability for interrupted model runs
- Integration with OpenCompass, Inspect Evals, and EvalScope
🎯 Use cases
- Evaluate LLM performance on role assignment tasks
- Evaluate LLM performance on prompt writing tasks
- Compare different models on multi-agent prompt composition
- Integrate benchmark into existing evaluation frameworks
📦 Installation
🧰 Requirements: Python 3.13+ and uv; API keys for model-calling commands (e.g., OPENAI_API_KEY).
git clone https://github.com/WhymustIhaveaname/PerspectiveGap.git
cd PerspectiveGap
uv sync
For library installation with model-calling support: pip install perspective-gap[runner].
🚀 Usage
uv run python scripts/score_predictions.py --predictions tests/fixtures/example_predictions.jsonl
To run a model smoke test:
uv run python scripts/run_model_predictions.py \
--provider openai --model <MODEL_YOU_HAVE_ACCESS_TO> \
--scenario-id pg_006 --shuffle-seed 1 --tasks role_assignment \
--out predictions/smoke.jsonl
uv run python scripts/score_predictions.py --predictions predictions/smoke.jsonl
⚠️ Good to know
Production-oriented features such as Batch API submission are intentionally not included.
❓ FAQ
What Python version is required?
Python 3.13+ is required.
How do I run model predictions?
Use scripts/run_model_predictions.py with --provider and --model, and set the corresponding API key environment variable.
Can I resume interrupted runs?
Yes, the runner skips completed (evaluation_id, model) requests when the output file already contains them.
What tasks are included?
Two tasks: role_assignment and prompt_writing, which can be run together or separately.
📊 Repository
🤖 Overview, features, install steps and FAQ were generated from the project's README on Sep 4, 2026. Always check the original source before running commands.