--- name: web-scraper description: Smart web scraping and browser automation with automatic provider selection. Chooses between Firecrawl (hardest sites, Cloudflare, JS-heavy), Playwright + Stealth (logged-in sessions, forms, stealth anti-bot), or browser tool (quick visual tasks). Use for ANY web-scraping, data-extraction, or site-interaction task. The skill handles provider selection automatically. ---

Web Scraper — Auto Provider Selection

🎯 Decision Tree (I pick automatically)

Task: visit/extract URL
 ├─ Protected site (Cloudflare/LinkedIn/Instagram/Amazon)?
 │   → Firecrawl (paid, always works)
 ├─ Needs logged-in session / complex form / multi-tab?
 │   → Playwright MCP (+ stealth plugin if anti-bot detected)
 ├─ Just need readable content from a normal URL?
 │   → Firecrawl scrape (fastest, markdown-ready)
 ├─ Needs visual confirmation / screenshot for user?
 │   → browser tool (user can see)
 └─ Scraping dozens of pages?
     → Firecrawl crawl (batch, efficient)

🛠️ Providers Available

1. Firecrawl (paid API, best for hard sites)

bash
source /root/.openclaw/env.sh
curl -s https://api.firecrawl.dev/v1/scrape \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","formats":["markdown"]}'

2. Playwright MCP (23 tools, stealth-capable)

Use via mcporter:

bash
mcporter --config /root/.openclaw/workspace/config/mcporter.json call playwright.browser_navigate url=https://example.com
mcporter --config /root/.openclaw/workspace/config/mcporter.json call playwright.browser_snapshot

3. Playwright + Stealth (Node.js script for max stealth)

bash
node /root/.openclaw/workspace/skills/web-scraper/stealth_scrape.js <url>

4. Browser tool (browser action, built-in)

For visual tasks where user wants to see or act.

🧠 Selection Rules (Automatic)

| Signal | Pick | |--------|------| | "Cloudflare protected" / 403 / bot wall | Firecrawl | | "LinkedIn" / "Instagram" / "X (Twitter)" | Firecrawl | | "login" / "my account" / "dashboard" | Playwright (persistent profile) | | "screenshot" / "show me" / "click" | browser tool | | "scrape 50 pages" / "crawl" | Firecrawl crawl endpoint | | "simple GET" / "blog post" / "news article" | Firecrawl scrape (faster than Playwright) | | Playwright fails with anti-bot → retry | stealth_scrape.js | | All fail | browser tool fallback |

Do NOT ask the user which provider — pick silently and switch if the first attempt fails.