crw-scrape

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

crw-scrape — single-page extraction

crw-scrape — 单页内容提取

When to use

使用场景

You have one (or a handful of) known URLs and want their content.
Step 2 in the crw ladder: if you don't have a URL yet, go to crw-search (step 1) first. For many pages under a site, use crw-crawl (step 4). For a local PDF, use crw-parse (step 5).
JS-heavy page? You usually don't need anything special — crw auto-detects and renders. This is a crw advantage: no separate "interact"/browser step.

你拥有一个（或少量）已知URL，并需要获取其内容。
属于crw工作流阶梯的第2步：如果你还没有URL，请先使用crw-search（第1步）。若要抓取站点下的多个页面，请使用crw-crawl（第4步）。处理本地PDF请使用crw-parse（第5步）。
页面包含大量JS？通常无需额外操作——crw会自动检测并渲染页面。这是crw的优势：无需单独的“交互”/浏览器步骤。

Quick start

快速开始

CLI (binary on PATH):

bash

crw scrape "https://example.com"                       # → markdown to stdout
crw scrape "https://example.com" --format json -o page.json
crw scrape "https://example.com" --js --css "article.main"
crw scrape "https://example.com" --format links -o .crw/links.txt

MCP (inside an agent harness):

crw_scrape(url="https://example.com", formats=["markdown"], onlyMainContent=true)

REST (drop-in for Firecrawl SDKs — just swap the base URL):

bash

curl -X POST "$CRW_API_URL/v1/scrape" -H "Authorization: Bearer $CRW_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://example.com","formats":["markdown"],"onlyMainContent":true}'

CLI（已添加到PATH的二进制文件）：

bash

crw scrape "https://example.com"                       # → 将markdown输出到标准输出
crw scrape "https://example.com" --format json -o page.json
crw scrape "https://example.com" --js --css "article.main"
crw scrape "https://example.com" --format links -o .crw/links.txt

MCP（在Agent harness中使用）：

crw_scrape(url="https://example.com", formats=["markdown"], onlyMainContent=true)

REST（可直接替代Firecrawl SDK——只需替换基础URL）：

bash

curl -X POST "$CRW_API_URL/v1/scrape" -H "Authorization: Bearer $CRW_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://example.com","formats":["markdown"],"onlyMainContent":true}'

Options

选项

Need	CLI flag	MCP / REST field
Output format	`--format markdown\|html\|rawhtml\|text\|links\|json`	`formats: [...]`
Strip nav/footer/sidebar	(on by default; `--raw` to disable)	`onlyMainContent: true`
Force JS rendering	`--js`	`renderJs: true` (null = auto)
Wait after load	—	`waitFor: 2000` (ms)
Keep only selectors	`--css "article"` / `--xpath …`	`includeTags: ["article"]`
Drop selectors	—	`excludeTags: ["nav","footer"]`
Pick renderer	—	`renderer: "auto\|lightpanda\|chrome\|chrome_proxy\|playwright"` ( `auto` is default)
Save to file	`-o FILE`	(write the response yourself)
Structured JSON	`--extract '<schema>'`	`extract: {schema: {...}}` — see crw-extract
Use a proxy	`--proxy URL` `--stealth`	`proxy` , `proxyRotation` , `stealth`

需求	CLI 参数	MCP / REST 字段
输出格式	`--format markdown\|html\|rawhtml\|text\|links\|json`	`formats: [...]`
移除导航栏/页脚/侧边栏	默认开启；使用 `--raw` 关闭	`onlyMainContent: true`
强制JS渲染	`--js`	`renderJs: true` （null = 自动检测）
加载后等待	—	`waitFor: 2000` （毫秒）
仅保留指定选择器	`--css "article"` / `--xpath …`	`includeTags: ["article"]`
移除指定选择器	—	`excludeTags: ["nav","footer"]`
选择渲染器	—	`renderer: "auto\|lightpanda\|chrome\|chrome_proxy\|playwright"` （默认值为 `auto` ）
保存到文件	`-o FILE`	（自行处理响应结果）
结构化JSON	`--extract '<schema>'`	`extract: {schema: {...}}` — 详见crw-extract
使用代理	`--proxy URL` `--stealth`	`proxy` , `proxyRotation` , `stealth`

Tips

小贴士

Quote URLs —
```
?
```
and
```
&
```
are shell-special. Always wrap in quotes.
Multiple URLs = run them concurrently. Fire several
```
crw scrape … &
```
and
```
wait
```
, or issue parallel MCP calls.
Blank page / loading skeleton? Add
```
--js
```
/
```
renderJs: true
```
, optionally a
```
waitFor
```
. crw's auto-detect covers most SPAs without it.
Don't dump huge pages into context. Write to
```
.crw/
```
, then
```
grep
```
/
```
head
```
. MCP truncates to ~15 000 chars (
```
maxLength: 0
```
to opt out).
Want a typed object, not prose?
```
--format json
```
returns the raw full-page object (metadata + content), not schema-extracted data. For structured extraction against a schema use
```
--extract '<schema>'
```
— this calls an LLM and requires a configured LLM provider. See the dedicated crw-extract skill.
Source is a file, not a URL? Use crw-parse instead.

给URL加引号 —
```
?
```
和
```
&
```
是shell的特殊字符，请始终用引号包裹URL。
多URL可并发处理。执行多个
```
crw scrape … &
```
并使用
```
wait
```
等待，或发起并行MCP调用。
页面空白/显示加载骨架？ 添加
```
--js
```
/
```
renderJs: true
```
，可选择搭配
```
waitFor
```
参数。crw的自动检测功能可覆盖大多数SPA场景，无需手动设置。
不要将大页面内容直接传入上下文。写入
```
.crw/
```
目录后，再用
```
grep
```
/
```
head
```
处理。MCP会自动截断至约15000字符（设置
```
maxLength: 0
```
可取消截断）。
需要类型化对象而非纯文本？
```
--format json
```
返回的是原始全页对象（元数据+内容），而非基于 schema 提取的数据。若要基于 schema 进行结构化提取，请使用
```
--extract '<schema>'
```
——此功能会调用LLM，需要配置LLM提供商。详见专门的crw-extract工具。
源是文件而非URL？ 请使用crw-parse替代。

crw-scrape

Original

Translation

crw-scrape — single-page extraction

crw-scrape — 单页内容提取

When to use

使用场景

Quick start

快速开始

Options

选项

Tips

小贴士

See also

相关工具