OSWorld
OSWorld is a benchmark and environment for evaluating multimodal agents on real computer tasks across operating systems. It provides virtual machines with realistic tasks in Ubuntu, Windows, and macOS, solving the problem of standardized agent evaluation in interactive digital environments. It is for researchers and developers building or testing AI agents that operate GUIs.
✨ Key features
- Supports multiple providers: VMware, VirtualBox, Docker, AWS, Modal, Daytona
- Includes benchmark tasks across Office, Daily, and Professional categories
- Provides parallel execution for faster evaluation
- Offers manual examination tool for task verification
- Supports various observation types (screenshot, etc.) and action spaces
- Includes result visualization and detailed scoring
🎯 Use cases
- Evaluate multimodal agents on real computer tasks
- Benchmark agent performance across different OS environments
- Develop and test GUI automation agents
- Run large-scale parallel evaluations using cloud providers
- Manually verify task correctness and evaluation metrics
📦 Installation
🧰 Requirements: Python >= 3.10, VMware Workstation Pro/Fusion or VirtualBox (for local), or Docker with KVM support, or cloud accounts (AWS, Modal, Daytona). Some tasks require Google account and proxy configuration.
git clone https://github.com/xlang-ai/OSWorld
cd OSWorldpip install -r requirements.txtAlternatively, install without benchmark tasks: pip install desktop-env
For VMware, install VMware Workstation Pro (or Fusion for Apple chips) and configure vmrun. Verify with vmrun -T ws list.
For Docker, check KVM support with egrep -c '(vmx|svm)' /proc/cpuinfo and install Docker. For Modal: pip install 'modal>=1.5.0' and modal setup. For Daytona: pip install daytona and set API key.
🚀 Usage
python quickstart.pypython quickstart.py --provider_name vmware --path_to_vm "path/to/your/vm.vmx"For running an agent baseline (example with GPT-4o):
export OPENAI_API_KEY='changeme'
python run.py \
--provider_name vmware \
--path_to_vm Ubuntu/Ubuntu.vmx \
--headless \
--observation_type screenshot \
--model gpt-4o \
--sleep_after_execution 3 \
--max_steps 15 \
--result_dir ./results \
--client_password password
⚠️ Good to know
Some tasks require Google account setup and proxy configuration; otherwise they fail. macOS hosts do not support KVM, so Docker is not recommended on macOS. VirtualBox may not support parallelism and macOS on Apple chips.
❓ FAQ
What are the default credentials for the virtual machines?
For providers vmware, virtualbox, and docker, the username is 'user' and password is 'password'. For cloud providers like AWS, the default is osworld-public-evaluation.
How can I run OSWorld on a server without KVM?
You can use Docker without KVM, but performance may be worse. Alternatively, use cloud providers like AWS, Modal, or Daytona which do not require local KVM.
How do I get my results on the verified leaderboard?
You need to schedule a meeting with the maintainers to run your agent code on their side, and they will report the results. You must allow disclosure of your agent implementation.
Can I use a custom OpenAI-compatible API endpoint?
Yes, you can set the OPENAI_BASE_URL environment variable to use a custom endpoint.
📊 Repository
🤖 Overview, features, install steps and FAQ were generated from the project's README on Sep 4, 2026. Always check the original source before running commands.