Search Results: web-crawling
Found 17 Skills
tavily-best-practices
Build production-ready Tavily integrations with best practices baked in. Reference documentation for developers using coding assistants (Claude Code, Cursor, etc.) to implement web search, content extraction, crawling, and research in agentic workflows, RAG systems, or autonomous agents.
tavily-crawl
Crawl websites and extract content from multiple pages via the Tavily CLI. Use this skill when the user wants to crawl a site, download documentation, extract an entire docs section, bulk-extract pages, save a site as local markdown files, or says "crawl", "get all the pages", "download the docs", "extract everything under /docs", "bulk extract", or needs content from many pages on the same domain. Supports depth/breadth control, path filtering, semantic instructions, and saving each page as a local markdown file.
multi-search-engine
Multi search engine integration with 17 engines (8 CN + 9 Global). Supports advanced search operators, time filters, site search, privacy engines, and WolframAlpha knowledge queries. No API keys required.
crawl
Crawl any website and save pages as local markdown files. Use when you need to download documentation, knowledge bases, or web content for offline access or analysis. No code required - just provide a URL.
crawl4ai
This skill should be used when users need to scrape websites, extract structured data, handle JavaScript-heavy pages, crawl multiple URLs, or build automated web data pipelines. Includes optimized extraction patterns with schema generation for efficient, LLM-free extraction.
a-stock-daily-report
Automatic generation system for A-share daily briefings. It crawls real-time data from East Money and generates daily reports covering complete information such as market indices, popular sectors, and capital trends.
site-crawler
Crawl and extract content from websites
crawl4ai
Use when crawling web pages, extracting markdown content, or scraping website data with intelligent chunking and skeleton planning. Use when the user provides a URL or link to fetch or crawl.
jb-docs-scraper
Scrape documentation websites into local markdown files for AI context. Takes a base URL and crawls the documentation, storing results in ./docs (or custom path). Uses crawl4ai with BFS deep crawling.
spider
Web crawling and scraping with analysis. Use for crawling websites, security scanning, and extracting information from web pages.
content-source-aggregator
统一信息源热点采集。从 X/Twitter、YouTube、B站、GitHub、Reddit、LinuxDo 六大平台免费获取热门内容，输出标准化热点池供内容创作流水线使用。全部使用免费公开 API，无需付费。
toutiao-news-trends
Fetch news hot list/trending search data from Toutiao (www.toutiao.com), covering popular Chinese news across multiple fields such as current affairs, finance, social events, international news, technological development, and entertainment gossip, and output hot topic titles, popularity values, and jump links.