MMLU-Pro
MMLU-Pro is an enhanced benchmark for evaluating language understanding models, featuring over 12,000 challenging questions across 14 domains with 10 answer choices each. It addresses the limitations of the original MMLU by increasing difficulty and reducing prompt sensitivity, making it suitable for researchers and developers seeking a more robust evaluation of model reasoning capabilities.
✨ Key features
- Over 12,000 curated questions from academic exams and textbooks
- 14 diverse domains including STEM, humanities, and more
- 10 answer choices per question to reduce random guessing
- More stable scores across 24 prompt styles (2% variance)
- Chain-of-Thought reasoning improves performance
- Includes evaluation scripts for local and API inference
🎯 Use cases
- Benchmarking large language models for research
- Comparing model performance across different domains
- Testing prompt robustness and reasoning abilities
- Evaluating models with Chain-of-Thought prompting
📦 Installation
🧰 Requirements: Python environment with necessary dependencies; API key required for API-based inference.
Clone the repository:
git clone https://github.com/TIGER-AI-Lab/MMLU-Pro.git
cd MMLU-Pro
Install dependencies (if any) as per the repository's requirements.
🚀 Usage
To run local inference, modify the model name in the script and execute:
cd scripts/examples/
sh eval_llama_2_7b.sh
For API inference, set your API key in evaluate_from_api.py and run:
sh eval_gpt_4.sh
⚠️ Good to know
The benchmark is designed for research purposes; no specific limitations are mentioned in the README.
❓ FAQ
How is MMLU-Pro different from MMLU?
MMLU-Pro increases answer choices from four to ten, includes more reasoning-focused questions, and shows lower sensitivity to prompt variations.
What models are included in the leaderboard?
The leaderboard includes models like Claude-3.5-Sonnet, GPT-4o, Gemini-1.5-Pro, and many others, with accuracy scores.
Can I use my own model for evaluation?
Yes, you can modify the model name in the evaluation scripts to run inference with your own model.
Does MMLU-Pro support API-based evaluation?
Yes, it provides scripts for API inference, including support for OpenAI-compatible APIs.
📊 Repository
🤖 Overview, features, install steps and FAQ were generated from the project's README on Sep 4, 2026. Always check the original source before running commands.