Universal Scraping Architect
Universal Scraping Architect designs complete, robust data-extraction pipelines with intelligent routing, validation, and token-budget tracking. It supports three modes—API-driven (Firecrawl), local Python, and hybrid—and produces validated, clean output in CSV, JSON, or Markdown.
✨ What it does
- Routes extraction to Firecrawl, local Python, or a hybrid pipeline based on source type and data sensitivity.
- Tracks API quotas and LLM token budgets before large jobs.
- Provides editable runner templates with offline sample mode.
- Validates every extraction result with a structural validation script.
- Flags hardcoded API keys, private data leakage, and missing pagination proactively.
- Outputs data in CSV, JSON, or Markdown with summary logs.
🎯 When to use it
- When you need to scrape a public website or crawl a domain, especially if it's dynamic or requires search-first discovery.
- When extracting data from local files (PDF, Excel, CSV) or handling private/sensitive data that shouldn't be sent to external APIs.
- When building a data pipeline that requires validation, checkpointing, and token-budget awareness.
- When you need to combine web extraction with local cleaning and structuring (hybrid pipeline).
🚀 How to use
Trigger by asking to scrape, crawl, or extract data from a website, document, or API. The skill expects a target source (URL, file path, or document) and optionally the desired output format. It will route to the appropriate mode and generate a validated script. Example prompts:
Scrape this site: https://example.com/products
Extract all tables from this PDF and save as CSV
📄 Output: A validated Python extraction script or a clean dataset (CSV/JSON/Markdown) with a summary log.
📦 Add this skill to Claude Code
# 1. Get the skills repo
git clone --depth 1 https://github.com/alirezarezvani/claude-skills /tmp/claude-skills
# 2. Copy this skill into your project (or ~/.claude/skills for all projects)
mkdir -p .claude/skills
cp -r /tmp/claude-skills/engineering/universal-scraping-architect/skills/universal-scraping-architect .claude/skills/universal-scraping-architectSkill source: engineering/universal-scraping-architect/skills/universal-scraping-architect/SKILL.md
⚠️ Good to know
Requires a Firecrawl API key for Mode 1 (loaded via environment variable); local modes require Python dependencies (beautifulsoup4, pandas) installed.
❓ FAQ
What happens if I run the example scripts without network access?
You can run them with the --sample flag to see the expected summary shape offline.
How do I validate my extraction output?
Run python3 scripts/validate_extraction.py output.json --json; it exits 0 only on status 'ok'.
What should I do if my target site has hundreds of records?
The skill will flag missing pagination and add checkpointing to handle multi-page jobs safely.
🤖 Overview, features, install steps and FAQ were generated from the project's SKILL.md on Sep 4, 2026. Always check the original source before running commands.