Loading...
Loading...
High-performance web crawler for discovering and mapping website structure. Use when users ask to crawl a website, map site structure, discover pages, find all URLs on a site, analyze link relationships, or generate site reports. Supports sitemap discovery, checkpoint/resume, rate limiting, and HTML report generation.
npx skill4agent add leobrival/serum-plugins-official website-crawlercd ~/.claude/scripts/crawler
bun src/index.ts <URL> [options]| Option | Short | Default | Description |
|---|---|---|---|
| | 2 | Maximum crawl depth |
| | 20 | Concurrent workers |
| | 2 | Rate limit (requests/second) |
| | - | Use preset profile (fast/deep/gentle) |
| | auto | Output directory |
| | true | Use sitemap.xml for discovery |
| | auto | Allowed domain (extracted from URL) |
| - | false | Enable debug logging |
| Profile | Workers | Depth | Rate | Use Case |
|---|---|---|---|---|
| 50 | 3 | 10 | Quick site mapping |
| 20 | 10 | 3 | Thorough crawling |
| 5 | 5 | 1 | Respect server limits |
bun src/index.ts https://example.combun src/index.ts https://example.com --depth 5 --workers 30 --rate 5bun src/index.ts https://example.com --profile fastbun src/index.ts https://example.com --profile gentle{
"stats": {
"pages_found": 150,
"pages_crawled": 147,
"external_links": 23,
"errors": 3,
"duration": 45.2
},
"results": [
{
"url": "https://example.com/page",
"title": "Page Title",
"status_code": 200,
"depth": 1,
"links": ["..."],
"content_type": "text/html"
}
]
}bun src/index.ts <url> --rate 1
# or
bun src/index.ts <url> --profile gentlecd ~/.claude/scripts/crawler/engine
go build -o crawler main.gobun src/index.ts <url> --depth 1 --workers 50plugins/crawler/commands/crawler.mdplugins/crawler/skills/website-crawler/reference.mdplugins/crawler/skills/website-crawler/scripts/plugins/crawler/skills/website-crawler/scripts/config/profiles/