SWE-bench
SWE-bench is a benchmark for evaluating large language models on real-world software issues from GitHub. It provides a codebase and an issue, and the model must generate a patch that resolves the problem. It is intended for researchers and developers working on AI for code.
✨ Key features
- Evaluates LLMs on real GitHub issues
- Includes multiple datasets: full, verified, multimodal, multilingual
- Containerized evaluation with Docker for reproducibility
- CLI for evaluation, inference, and image management
- Supports cloud-based evaluation via Modal or sb-cli
- Provides pre-trained models and retrieval datasets
🎯 Use cases
- Benchmarking LLMs on software engineering tasks
- Comparing model performance on issue resolution
- Training models on SWE-bench datasets
- Evaluating multimodal models on visual software domains
- Creating new SWE-bench tasks from custom repositories
📦 Installation
🧰 Requirements: Requires Docker, Python, and a machine with at least 120GB free storage, 16GB RAM, and 8 CPU cores (x86_64 recommended).
git clone git@github.com:SWE-bench/SWE-bench.git
cd SWE-bench
pip install -e .git clone --depth 1 https://github.com/SWE-bench/swe-bench-tasks.git ./swe-bench-tasks
swebench dataset check ./swe-bench-tasks🚀 Usage
swebench eval verified --gold \
-i sympy__sympy-20590 \
--run-id validate-gold \
--task-repo ./swe-bench-tasks
⚠️ Good to know
Evaluation is resource-intensive and requires significant storage and CPU; support for arm64 is experimental.
❓ FAQ
How do I load the SWE-bench dataset?
You can load it using the Hugging Face datasets library: from datasets import load_dataset; swebench = load_dataset('princeton-nlp/SWE-bench', split='test').
What are the system requirements for running evaluation?
It is recommended to run on an x86_64 machine with at least 120GB of free storage, 16GB of RAM, and 8 CPU cores. Docker is required.
Can I run evaluation on the cloud?
Yes, you can use sb-cli for AWS or run on Modal. See the documentation for details.
How do I re-evaluate an instance with a different prediction?
You must use a different run_id because results are cached by run_id and instance_id.
📊 Repository
🤖 Overview, features, install steps and FAQ were generated from the project's README on Sep 4, 2026. Always check the original source before running commands.