ScrapeGraphAI
ScrapeGraphAI is a Python web scraping library that uses LLMs and graph logic to create scraping pipelines for websites and local documents (XML, HTML, JSON, Markdown). It lets you specify what information to extract and the library handles the scraping. It is for developers who want to extract structured data from web pages or documents using natural language prompts.
✨ Key features
- Uses LLMs to understand prompts and extract data
- Supports multiple scraping pipelines: SmartScraper, Search, Speech, ScriptCreator
- Works with various LLMs: OpenAI, Groq, Azure, Gemini, local via Ollama
- Handles single and multi-page scraping
- Can generate Python scripts and audio files from scraped data
- Integrates with Langchain, Llama Index, and other frameworks
🎯 Use cases
- Extract company info, founders, and social links from a website
- Scrape data from multiple search results
- Generate Python scripts for custom scraping tasks
- Convert webpage content into audio summaries
- Extract structured data from local HTML or JSON files
📦 Installation
🧰 Requirements: Python environment with pip; optional Ollama for local models; API keys for cloud LLMs like OpenAI.
pip install scrapegraphaiplaywright installNote: it is recommended to install the library in a virtual environment to avoid conflicts with other libraries.
🚀 Usage
from scrapegraphai.graphs import SmartScraperGraph
graph_config = {
"llm": {
"model": "ollama/llama3.2",
"model_tokens": 8192,
"format": "json",
},
"verbose": True,
"headless": False,
}
smart_scraper_graph = SmartScraperGraph(
prompt="Extract useful information from the webpage, including a description of what the company does, founders and social media links",
source="https://scrapegraphai.com/",
config=graph_config
)
result = smart_scraper_graph.run()
import json
print(json.dumps(result, indent=4))
⚠️ Good to know
The README states that ScrapeGraphAI is meant for data exploration and research purposes only, and the project is not responsible for misuse.
❓ FAQ
What LLMs can I use with ScrapeGraphAI?
You can use various LLMs through APIs such as OpenAI, Groq, Azure, Gemini, and MiniMax, or local models using Ollama.
How do I use a different LLM like OpenAI?
Change the llm config in graph_config to include your API key and model, e.g., "api_key": "YOUR_OPENAI_API_KEY", "model": "openai/gpt-4o-mini".
What are the main scraping pipelines available?
The main pipelines include SmartScraperGraph for single pages, SearchGraph for search results, SpeechGraph for audio generation, ScriptCreatorGraph for Python scripts, and multi-page versions.
Can I opt out of telemetry?
Yes, set the environment variable SCRAPEGRAPHAI_TELEMETRY_ENABLED=false to opt out of anonymous usage metrics.
📊 Repository
🤖 Overview, features, install steps and FAQ were generated from the project's README on Sep 4, 2026. Always check the original source before running commands.