All Skills Search Tools

|

Loading...

AI Agent Skills Directory with categorization, English/Chinese translation, and script security checks.

Sitemap

Home
All Skills
Search
Tools

About

About Us
Disclaimer
Copyright

Help

FAQ
Privacy
Terms

Contact Us:osulivan147@qq.com

© 2026 Skill4Agent. All rights reserved.

Home/All Skills/Data Processing/scraping

scraping

Original：🇺🇸 English

Translated

Web scraping via progressive escalation and social media platform scrapers. USE WHEN scraping, crawl, scrape URL, bot detection, CAPTCHA, spider.

5installs

Sourcedanielmiessler/personal_ai_infrastructure

Added on2026-04-11

NPX Install

npx skill4agent add danielmiessler/personal_ai_infrastructure scraping

Tags

Translated version includes tags in frontmatter

web-scraping crawler data-collection proxy-rotation captcha

SKILL.md Content

View Translation Comparison →

Scraping Skill

Progressive Escalation

Direct fetch — curl/fetch with standard headers
Browser rendering — Headless browser for JS-heavy sites
Proxy rotation — For rate-limited or geo-restricted content
Specialized APIs — Platform-specific scrapers

Rules

Respect robots.txt
Rate limit requests
Handle pagination
Extract structured data
Store raw + processed versions

Download

Download TranslatedIncludes original Skill.md

(1.2 KB)Download Original(537 B)

Related Skills

brightdata-cli

Guide for using the Bright Data CLI (`brightdata` / `bdata`) to scrape websites, search the web, extract structured data from 40+ platforms, manage proxy zones, and check account budget. Use this skill whenever the user wants to scrape a URL, search Google/Bing/Yandex, extract data from Amazon/LinkedIn/Instagram/TikTok/YouTube/Reddit or any other platform, check their Bright Data balance or zones, or do anything involving web data collection from the terminal. Also trigger when the user mentions brightdata, bdata, web scraping CLI, SERP API, or wants to install Bright Data skills into their coding agent.

xlsx

Use this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.

firecrawl-build-scrape

Integrate Firecrawl `/scrape` into product code for single-page extraction. Use when an app already has a URL and needs markdown, HTML, links, screenshots, metadata, or structured page output. Prefer this skill over broader crawl patterns when the feature is page-level.

azure-kusto

Query and analyze data in Azure Data Explorer (Kusto/ADX) using KQL for log analytics, telemetry, and time series analysis. USE FOR: KQL queries, Kusto database queries, Azure Data Explorer, ADX clusters, log analytics, time series data, IoT telemetry, anomaly detection DO NOT USE FOR: SQL databases (use azure-postgres), NoSQL queries (use azure-storage), Elasticsearch, AWS analytics tools