scraping

Original：🇺🇸 English
Translated

Web scraping via progressive escalation and social media platform scrapers. USE WHEN scraping, crawl, scrape URL, bot detection, CAPTCHA, spider.

7installs
Sourcedanielmiessler/personal_ai_infrastructure
Added on

NPX Install

npx skill4agent add danielmiessler/personal_ai_infrastructure scraping

Tags

Translated version includes tags in frontmatter
web-scrapingcrawlerdata-collectionproxy-rotationcaptcha

SKILL.md Content

View Translation Comparison →

Scraping Skill

Progressive Escalation

  1. Direct fetch — curl/fetch with standard headers
  2. Browser rendering — Headless browser for JS-heavy sites
  3. Proxy rotation — For rate-limited or geo-restricted content
  4. Specialized APIs — Platform-specific scrapers

Rules

  • Respect robots.txt
  • Rate limit requests
  • Handle pagination
  • Extract structured data
  • Store raw + processed versions