Desktop Control
DesktopCtl is a local CLI for AI agents to observe and control a computer via screen, mouse, and keyboard. It runs fully local, with no cloud dependency, and works with any AI model, even without vision. It is designed for developers building AI agents that need desktop automation.
✨ Key features
- Local-first runtime, no cloud dependency
- Bring your own AI, any model
- GPU-accelerated text recognition and computer vision
- Selector-first automation with text/token and coordinate fallback
- Explicit waits and post-action verification
- Stable JSON contracts for agent integrations
🎯 Use cases
- Automate GUI workflows via AI agents
- Control applications like Notes using natural language commands
- Extract screen content as tokens for agent grounding
- Invoke native macOS menu items programmatically
📦 Installation
🧰 Requirements: macOS (current support target), Rust toolchain (cargo), just command runner, Accessibility and Screen Recording permissions for DesktopCtl.app.
make install
🚀 Usage
raw="$(desktopctl app open Notes --json)"
win_id="$(printf '%s' "$raw" | jq -r '.result.window_id // empty')"
desktopctl keyboard press cmd+f --active-window "$win_id" --no-observe
desktopctl keyboard type "Shopping list" --active-window "$win_id" --no-observe
desktopctl screen tokenize --active-window "$win_id"
⚠️ Good to know
macOS-first, active development; multi-platform support is upcoming.
❓ FAQ
Does DesktopCtl send screenshots to the cloud?
No, it runs fully local and no screenshots are sent to the cloud.
What are the two binaries and their roles?
DesktopCtl.app (desktopctld) is the daemon that owns perception, state, execution, and verification; desktopctl is a stateless CLI for actions and queries over local IPC.
How can I invoke a native menu item?
Use desktopctl menu list --active-window "$win_id" to list menus, then desktopctl menu click --id menu_file_new_window --active-window "$win_id" to click by the returned #menu_* ID.
What permissions are required?
Accessibility and Screen Recording permissions for DesktopCtl.app.
📊 Repository
🤖 Overview, features, install steps and FAQ were generated from the project's README on Sep 4, 2026. Always check the original source before running commands.