Loading...
Loading...
Use when crawling web pages, extracting markdown content, or scraping website data with intelligent chunking and skeleton planning. Use when the user provides a URL or link to fetch or crawl.
npx skill4agent add tao3k/omni-dev-fusion crawl4aicrawl_urlwebCrawl| Parameter | Type | Default | Description |
|---|---|---|---|
| str | - | Target URL to crawl (required) |
| str | "smart" | Action mode: "smart", "skeleton", "crawl" |
| bool | true | Clean and simplify markdown output |
| int | 0 | Maximum crawling depth (0=single page) |
| bool | false | Also return document skeleton (TOC) |
| list[int] | - | List of section indices to extract |
| Mode | Description | Use Case |
|---|---|---|
| LLM generates chunk plan, then extracts relevant sections | Large docs where you need specific info |
| Extract lightweight TOC without full content | Quick overview, decide what to read |
| Return full markdown content | Small pages, complete content needed |
# Smart crawl with LLM chunking (default)
@omni("crawl4ai.CrawlUrl", {"url": "https://example.com"})
# Skeleton only - get TOC quickly
@omni("crawl4ai.CrawlUrl", {"url": "https://example.com", "action": "skeleton"})
# Full content crawl
@omni("crawl4ai.CrawlUrl", {"url": "https://example.com", "action": "crawl"})
# Extract specific sections
@omni("crawl4ai.CrawlUrl", {"url": "https://example.com", "chunk_indices": [0, 1, 2]})
# Deep crawl (follow links up to depth N)
@omni("crawl4ai.CrawlUrl", {"url": "https://example.com", "max_depth": 2})
# Get skeleton with full content
@omni("crawl4ai.CrawlUrl", {"url": "https://example.com", "return_skeleton": true})| Topic | Description | Reference |
|---|---|---|
| Skeleton Planning | LLM sees TOC (~500 tokens) not full content (~10k+) | smart-chunking.md |
| Chunk Extraction | Token-aware section extraction | chunking.md |
| Deep Crawling | Multi-page crawling with BFS strategy | deep-crawl.md |
skeletonchunk_indicesmax_depthfit_markdown=true