Crawl4AI
Crawl4AI is an open-source web crawler and scraper that converts web pages into clean, LLM-ready Markdown for use in RAG, agents, and data pipelines. It solves the problem of extracting structured data from websites without requiring API keys or paid services. It is designed for developers building AI applications, data pipelines, or anyone needing reliable web extraction.
🎬 Crawl4AI Official Tutorial, Full 1hr with Quickstart Examples · Unclecode
✨ Key features
- Generates clean, structured Markdown with headings, tables, and code
- LLM-driven structured data extraction with chunking and cosine similarity
- Supports Chromium, Firefox, and WebKit with full browser control
- Handles dynamic content, lazy loading, and infinite scroll pages
- Provides Docker deployment with monitoring dashboard and API
- Includes stealth mode to avoid bot detection
🎯 Use cases
- Crawl websites to generate Markdown for RAG or LLM training
- Extract structured data like product prices or article metadata
- Deep crawl documentation sites for content analysis
- Automate data collection for research or monitoring
- Build custom scraping pipelines with hooks and caching
📦 Installation
🧰 Requirements: Requires Python 3.7+ and Playwright for browser automation; no API keys needed for basic use.
pip install -U crawl4aipip install crawl4ai --precrawl4ai-setupcrawl4ai-doctorIf you encounter any browser-related issues, you can install them manually:
python -m playwright install --with-deps chromium
🚀 Usage
import asyncio
from crawl4ai import *
async def main():
async with AsyncWebCrawler() as crawler:
result = await crawler.arun(
url="https://www.nbcnews.com/business",
)
print(result.markdown)
if __name__ == "__main__":
asyncio.run(main())
⚠️ Good to know
The synchronous version is deprecated and will be removed in future versions; the project is in active development with frequent security patches.
❓ FAQ
Do I need an API key to use Crawl4AI?
No, Crawl4AI is open-source and does not require any API keys for basic usage. You can install it via pip and start crawling immediately.
What browsers are supported?
Crawl4AI supports Chromium, Firefox, and WebKit via Playwright. The default setup installs Chromium.
Can I use Crawl4AI with Docker?
Yes, Crawl4AI provides a Docker image with a FastAPI server, monitoring dashboard, and playground. You can pull and run it with docker run.
How do I extract structured data like JSON?
You can use LLM-driven extraction with a schema, or CSS-based extraction using XPath and CSS selectors. The README provides examples for both.
📊 Repository
🤖 Overview, features, install steps and FAQ were generated from the project's README on Sep 4, 2026. Always check the original source before running commands.