GenAgent
ComfyBench is a benchmark and agent framework for evaluating and building LLM-based agents that autonomously design collaborative AI systems in ComfyUI. It provides a set of tasks, documentation, and workflows to test an agent's ability to generate executable workflows from natural language instructions. It is intended for researchers and developers working on AI agents and workflow generation.
✨ Key features
- Benchmark with 200 task instructions and 20 curriculum workflows
- Documentation for 3205 ComfyUI nodes
- Metrics: pass rate and resolve rate
- ComfyAgent pipeline that generates workflows from instructions
- Includes evaluation scripts for automated assessment
🎯 Use cases
- Benchmarking LLM-based agents for workflow generation in ComfyUI
- Developing agents that can autonomously design collaborative AI systems
- Evaluating agent performance on image generation and editing tasks
- Learning from existing workflows to create new ones
📦 Installation
🧰 Requirements: Python 3.12, ComfyUI server, API key for LLM, and necessary models/extensions for ComfyUI.
git clone https://github.com/xxyQwQ/ComfyBench
cd ComfyBench
conda create -n comfybench python=3.12
conda activate comfybench
pip install -r requirements.txt
Then modify config.yaml to set your ComfyUI server and API key.
🚀 Usage
conda activate comfybench
python main.py \
--instruction "task-instruction" \
--agent_name "comfy" \
--save_path "path/to/save/result"
⚠️ Good to know
Requires manual setup of ComfyUI with necessary models and extensions; some models must be downloaded manually.
❓ FAQ
What is ComfyBench?
ComfyBench is a benchmark for evaluating LLM-based agents in ComfyUI, focusing on their ability to design collaborative AI systems by generating workflows.
How is performance measured?
Performance is measured by pass rate (workflow executes correctly) and resolve rate (task requirements are realized).
What does ComfyAgent do?
ComfyAgent is an agent that builds collaborative AI systems in ComfyUI by generating workflows from task instructions, learning from existing workflows.
How do I run the evaluation?
Use the provided scripts: script/inference.py to generate workflows and script/evaluation.py to evaluate them, after setting up ComfyUI and environment variables.
📊 Repository
🤖 Overview, features, install steps and FAQ were generated from the project's README on Sep 4, 2026. Always check the original source before running commands.