Multi-SWE-bench
Multi-SWE-bench is a multilingual benchmark for evaluating large language models on real-world code issue resolution, spanning 7 languages (Java, TypeScript, JavaScript, Go, Rust, C, C++) with 1,632 curated instances. It addresses the lack of multilingual benchmarks beyond Python-centric ones like SWE-bench, providing a framework for automated issue resolution and reinforcement learning research. It is intended for researchers and developers working on LLM evaluation and code generation.
✨ Key features
- Covers 7 languages: Java, TypeScript, JavaScript, Go, Rust, C, C++
- 1,632 high-quality instances curated by 68 expert annotators
- Includes Multi-SWE-RL dataset with 4,723 instances for RL research
- Provides Docker-based reproducible evaluation environment
- Supports multiple agent frameworks: Agentless, SWE-agent, OpenHands
- Fully open-source data, code, and container images
🎯 Use cases
- Evaluate LLMs on multilingual code issue resolution
- Benchmark agent frameworks for automated bug fixing
- Train and evaluate reinforcement learning models for software engineering
- Research on multilingual code understanding and generation
📦 Installation
🧰 Requirements: Requires Docker for evaluation; Python environment for running the harness; no API keys mentioned.
git clone git@github.com:multi-swe-bench/multi-swe-bench.git
cd multi-swe-bench
make install
For development setup:
make install-dev
🚀 Usage
Prepare patch files and dataset files in JSONL format, then run:
python -m multi_swe_bench.harness.run_evaluation --config /path/to/your/config.json
Example config:
{
"mode": "evaluation",
"workdir": "./data/workdir",
"patch_files": ["./data/patches/<your_patch_file>.jsonl"],
"dataset_files": ["./data/patches/<to_evaluate_dataset_file>.jsonl"],
"force_build": false,
"output_dir": "./data/dataset",
"specifics": [],
"skips": [],
"repo_dir": "./data/repos",
"need_clone": false,
"global_env": [],
"clear_env": true,
"stop_on_error": true,
"max_workers": 8,
"max_workers_build_image": 8,
"max_workers_run_instance": 8,
"log_dir": "./data/logs",
"log_level": "DEBUG"
}
⚠️ Good to know
The benchmark is limited to 7 languages and may not cover all programming languages; it requires Docker for evaluation, which may be a constraint in some environments.
❓ FAQ
What languages are covered in Multi-SWE-bench?
It covers 7 languages: Java, TypeScript, JavaScript, Go, Rust, C, and C++.
How many instances are in the full benchmark?
The full benchmark contains 1,632 high-quality instances curated from 2,456 candidates.
Do I need to download Docker images before running evaluation?
No, it is optional. If images don't exist locally, they will be built during evaluation.
What is Multi-SWE-RL?
Multi-SWE-RL is an open-source dataset for reinforcement learning research, with an initial release of 4,723 instances.
📊 Repository
🤖 Overview, features, install steps and FAQ were generated from the project's README on Sep 4, 2026. Always check the original source before running commands.