GPQA Diamond
GPQA is a benchmark dataset of graduate-level, Google-proof multiple-choice questions for evaluating AI systems. It is designed to be difficult for both humans and AI, with questions that are not easily answerable via web search. It is intended for researchers and developers working on question answering and reasoning systems.
✨ Key features
- Graduate-level, Google-proof multiple-choice questions
- Includes canary string for dataset tracking
- Supports closed-book and open-book baselines
- Implements GPT-3.5 and GPT-4 models
- Provides prompt types: zero-shot, few-shot, chain-of-thought, retrieval
- Caches API responses to reduce costs
🎯 Use cases
- Evaluate AI models on expert-level reasoning tasks
- Benchmark question answering systems against a challenging dataset
- Test retrieval-augmented generation with Bing search
- Train answer-only baselines using T5 or CBOW models
📦 Installation
🧰 Requirements: Python 3.9, OpenAI API key, and optionally a Bing Web Search API key for open-book baselines.
conda create -n gpqa python=3.9
conda activate gpqapip install -r requirements.txt🚀 Usage
# Run a closed-book baseline with GPT-4
python baselines/run_baseline.py main --model_name gpt-4 --data_filename dataset/gpqa_main.csv --prompt_type zero_shot --verbose
⚠️ Good to know
The codebase currently only supports GPT-3.5-turbo-16k-0613 and GPT-4 models; other models are not implemented.
❓ FAQ
How do I download the dataset?
The dataset is available as a password-protected zip file (dataset.zip) with password deserted-untie-orchid, or on Hugging Face at https://huggingface.co/datasets/idavidrein/gpqa.
What prompt types are available?
The prompt types are: zero_shot, few_shot, zero_shot_chain_of_thought, chain_of_thought, retrieval, and retrieval_content.
Do I need a Bing API key?
A Bing API key is required only for open-book baselines (retrieval and retrieval_content). Closed-book baselines only need an OpenAI API key.
Can I use models other than GPT-3.5 and GPT-4?
No, the current codebase only implements GPT-3.5-turbo-16k-0613 and GPT-4.
📊 Repository
🤖 Overview, features, install steps and FAQ were generated from the project's README on Sep 4, 2026. Always check the original source before running commands.