AppAgent
AppAgent is an LLM-based multimodal agent framework that operates smartphone apps through human-like actions like tapping and swiping, without needing backend access. It learns to use new apps via autonomous exploration or human demonstrations, generating a knowledge base for task execution. It is for developers and researchers building AI agents for GUI automation.
✨ Key features
- Operates apps via simplified action space (tap, swipe, etc.)
- Learns new apps through autonomous exploration or human demos
- Generates documentation for UI elements for later use
- Supports GPT-4V and qwen-vl-max as multimodal models
- Works on real Android devices or emulators via ADB
- Includes optional grid overlay for precise UI interaction
🎯 Use cases
- Automate tasks on Android apps like following users on social media
- Pass CAPTCHA tests by mimicking human interactions
- Navigate and operate apps without backend access
- Generate documentation for UI elements to assist future automation
📦 Installation
🧰 Requirements: Requires Python 3, Android Debug Bridge (adb), an Android device with USB debugging enabled (or Android Studio emulator), and an OpenAI API key for GPT-4V or a Dashscope API key for qwen-vl-max.
cd AppAgent
pip install -r requirements.txt
🚀 Usage
python learn.pypython run.py⚠️ Good to know
GPT-4V is not free (each request/response pair costs around $0.03) and qwen-vl-max has poorer performance compared to GPT-4V.
❓ FAQ
What models can power AppAgent?
AppAgent uses GPT-4V by default, but you can also use qwen-vl-max (通义千问-VL) as an alternative. For custom models, you need to write a new model class in scripts/model.py.
How does AppAgent learn to use an app?
It can either explore the app autonomously by attempting a given task, or learn from human demonstrations. Both methods generate documentation for UI elements that is used in the deployment phase.
Can I use AppAgent without an Android device?
Yes, you can use the Android Studio emulator. AppAgent can detect the emulated device and operate apps on it just like a real device.
What is the cost of using AppAgent?
GPT-4V is not free; each request/response pair costs around $0.03. qwen-vl-max is currently free but has poorer performance.
📊 Repository
🤖 Overview, features, install steps and FAQ were generated from the project's README on Sep 4, 2026. Always check the original source before running commands.