web-crawl

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Skill: web-crawl

Fetch a web page and turn it into clean markdown for the wiki / an LLM. Backend is a quarantined adapter (

harness/web-crawl.config.yaml

verified:false

) — the builtin works now, no key.

将网页抓取并转换为适合 wiki / LLM 的整洁 markdown。后端是一个隔离式适配器（

harness/web-crawl.config.yaml

，

verified:false

）——内置后端当前即可使用，无需 key。

WHAT

WHAT（内容）

Purpose và context

目的与上下文

Purpose: turn a URL (or a site) into clean markdown and land it in
```
raw/
```
or a wiki draft, never raw HTML.
Trigger (when to use):
- User gives a URL and wants its content ("crawl this", "scrape", "fetch the page", "đọc trang này").
- Pulling external docs/articles into
```
raw/
```
  or a wiki draft.
- Bulk reading many pages for research (prefer a premium backend then).
Non-goals: not cloning a page's UI/look (that is
```
/web-clone
```
); not writing straight into
```
wiki/
```
(goes via
```
/propose
```
/
```
ingest
```
); not crawling auth-walled or disallowed content.

**目的：**将 URL（或站点）转换为整洁的 markdown，并保存到
```
raw/
```
或 wiki 草稿中，绝不保留原始 HTML。
触发条件（何时使用）：
- 用户提供 URL 并希望获取其内容（“抓取这个链接”、“抓取”、“获取页面”、“阅读此页面”）。
- 将外部文档/文章拉取到
```
raw/
```
  或 wiki 草稿。
- 为研究而批量阅读许多页面（此时优先使用高级后端）。
**非目标：**不克隆页面的 UI/外观（应使用
```
/web-clone
```
）；不直接写入
```
wiki/
```
（应通过
```
/propose
```
/
```
ingest
```
）；不抓取需要身份验证或被禁止的内容。

Mental model

心智模型

URL → backend (WebFetch in-session · builtin urllib+regex · premium adapter firecrawl/crawl4ai/jina) → markdown → raw/ or wiki draft → /propose or ingest → wiki

. The backend is a build-now-adapt-later quarantine: builtin = offline fallback, premium = the real upgrade behind one config file.

URL → 后端（会话内 WebFetch · 内置 urllib+regex · 高级适配器 firecrawl/crawl4ai/jina）→ markdown → raw/ 或 wiki 草稿 → /propose 或 ingest → wiki

。后端采用“立即构建、后续适配”的隔离模式：内置后端 = 离线回退方案，高级后端 = 通过一个配置文件实现的真正升级。

Input và output contract

输入与输出约定

	Field	Required?	Ý nghĩa
In	URL	có	page or site to fetch; or a local `.html` file for `md` mode
In	`--out` path	không	where to save; convention `raw/<slug>.md`
In	premium backend config	không	`backend` + `api_key_env` / `endpoint` + `verified:true` in `harness/web-crawl.config.yaml`
Out	markdown file	có	clean text in `raw/` or a wiki draft
Out	backend disclosure	có	which backend produced it; builtin quality limits stated, never passed off as premium

	字段	必填？	含义
In	URL	是	要获取的页面或站点；或用于 `md` 模式的本地 `.html` 文件
In	`--out` 路径	否	保存位置；约定为 `raw/<slug>.md`
In	高级后端配置	否	`backend` + `api_key_env` / `endpoint` + `harness/web-crawl.config.yaml` 中的 `verified:true`
Out	markdown 文件	是	`raw/` 或 wiki 草稿中的整洁文本
Out	后端说明	是	哪个后端生成了内容；必须说明内置后端的质量限制，不得冒充为高级后端

Rules và capabilities

规则与能力

RULE-01 (MUST): Markdown, not raw HTML — that is the point (token savings + clean text).
RULE-02 (MUST): Respect robots.txt / site Terms; do not crawl auth-walled or disallowed content.
RULE-03 (MUST): The builtin backend is the offline fallback; the premium backend is the BNAL adapter — never present an un-wired premium backend as working.

RULE-04 (MUST): Self-test:

python3 harness/scripts/web-crawl.py --self-test

Capabilities: outbound HTTP fetch (or in-session web fetch with JS render); HTML→markdown conversion; write to
```
raw/
```
or wiki draft only.

RULE-01 (MUST): 必须是 Markdown，而不是原始 HTML——这正是该 skill 的目的（节省 token + 简洁文本）。
RULE-02 (MUST): 遵守 robots.txt / 站点服务条款；不要抓取需要身份验证或被禁止的内容。
RULE-03 (MUST): 内置后端是离线回退方案；高级后端是 BNAL adapter——绝不能把尚未接通的高级后端说成可用。

RULE-04 (MUST): 自行测试：

python3 harness/scripts/web-crawl.py --self-test

。

能力：出站 HTTP 获取（或在会话中使用支持 JS 渲染的 WebFetch）；HTML→markdown 转换；仅写入
```
raw/
```
或 wiki 草稿。

Failure boundaries

失败边界

Site is JS-rendered and only the builtin is available → partial (builtin has no JS render); prefer WebFetch in-session or wire premium.
Whole-site crawl requested without a wired premium backend → blocked on config (builtin is single page only); say so, do not fake it.
robots.txt / Terms disallow or page is auth-walled → blocked, do not crawl.
Fetch error → failed, report URL + error.

站点由 JS 渲染，但只有内置后端可用 → partial（内置后端不支持 JS 渲染）；优先使用会话内 WebFetch，或接通高级后端。
请求整站爬取，但未接通高级后端 → blocked（配置受限）；如实说明，不要伪造已完成的抓取。
robots.txt / 服务条款禁止访问，或页面需要身份验证 → blocked，不要抓取。
抓取出错 → failed，报告 URL + 错误。

HOW

HOW（方法）

Main workflow

主工作流

Step	Type	Inputs	Action	Outputs/exit	Failure/next
W01	judgment	URL, session type	JS-heavy / rendered site in an agent session → WebFetch tool (B01)	markdown	not in session → W02
W02	effect	URL	Scripted single static page: `web-crawl.py fetch` builtin (or `md` for local HTML)	`raw/<slug>.md`	JS/whole-site need → W03
W03	effect	config	Real crawling → wire premium backend in `harness/web-crawl.config.yaml` , flip `verified:true` (B02)	premium output	not wired → report limit, do not claim
W04	effect	markdown	Land in `raw/` or wiki draft, then `/propose` / `ingest`	file saved	never into `wiki/` directly

Chi tiết từng bước (nguồn chân lý cho W01–W04):

JS-heavy / rendered site? In an agent session, prefer the in-session WebFetch tool (it renders JS + returns markdown). That is the richest path and needs no setup.
Scripted, single static page (offline, no key):
```
python3 harness/scripts/web-crawl.py fetch "<URL>" --out raw/<slug>.md
```
⚠️ The
```
builtin
```
backend is BASIC on purpose:
```
urllib
```
fetch + regex HTML→markdown — no JS rendering, single page only, no smart extraction. It is the offline fallback, NOT Firecrawl-quality. Local HTML already downloaded?
```
web-crawl.py md <file.html>
```
.
Real crawling (JS render, whole-site, clean extraction) → wire the real engine: the premium backend is where the quality is —
```
backend: firecrawl|crawl4ai|jina
```
+
```
api_key_env
```
/
```
endpoint
```
in
```
harness/web-crawl.config.yaml
```
, then flip
```
verified:true
```
. Firecrawl (managed, crawl-to- markdown, ~5-10x token reduction, ships an MCP for agent loops) or Crawl4AI (open-source, self-host, adaptive selectors). The builtin exists so the skill runs today; these are the real upgrade.
Land it: save markdown to
```
raw/
```
(human inbox) or a wiki draft, then
```
/propose
```
/
```
ingest
```
to bring the distilled bits into the wiki — never write straight into
```
wiki/
```
(R1/R2).

步骤	类型	输入	操作	输出/退出	失败/下一步
W01	判断	URL、会话类型	在 agent 会话中遇到 JS 密集型/渲染型站点 → 使用 WebFetch 工具 (B01)	markdown	不在会话中 → W02
W02	操作	URL	脚本化抓取单个静态页面：使用内置 `web-crawl.py fetch` （本地 HTML 则使用 `md` ）	`raw/<slug>.md`	需要 JS/整站 → W03
W03	操作	配置	真实爬取 → 在 `harness/web-crawl.config.yaml` 中接通高级后端，并改为 `verified:true` (B02)	高级后端输出	未接通 → 报告限制，不得声称已执行
W04	操作	markdown	保存到 `raw/` 或 wiki 草稿，然后执行 `/propose` / `ingest`	文件已保存	绝不直接写入 `wiki/`

各步骤详情（W01–W04 的真实依据）：

JS 密集型/渲染型站点？ 在 agent 会话中，优先使用会话内 WebFetch 工具（它可渲染 JS 并返回 markdown）。这是功能最完整的路径，且无需设置。
脚本化抓取单个静态页面（离线、无需 key）：
```
python3 harness/scripts/web-crawl.py fetch "<URL>" --out raw/<slug>.md
```
⚠️
```
builtin
```
后端是有意保持基础的：
```
urllib
```
抓取 + regex HTML→markdown——不支持 JS 渲染，仅限单页，也不具备智能提取。它是离线回退方案，达不到 Firecrawl 的质量。已下载本地 HTML？使用
```
web-crawl.py md <file.html>
```
。
**真实爬取（JS 渲染、整站、整洁提取）→ 接通真正的引擎：**高级后端才是质量核心——在
```
harness/web-crawl.config.yaml
```
中设置
```
backend: firecrawl|crawl4ai|jina
```
+
```
api_key_env
```
/
```
endpoint
```
，然后改为
```
verified:true
```
。Firecrawl（托管式爬取转 markdown，约减少 5–10 倍 token，并提供用于 agent 循环的 MCP）或 Crawl4AI（开源、自托管、自适应选择器）。内置后端的存在是为了让该 skill 立即可用；这些才是真正的升级方案。
**落地：**将 markdown 保存到
```
raw/
```
（人工收件箱）或 wiki 草稿，然后通过
```
/propose
```
/
```
ingest
```
将提炼后的内容纳入 wiki——绝不直接写入
```
wiki/
```
（R1/R2）。

Branches

分支

ID	Kind	Guard	Hành vi	Skip / failure	Rejoin
B01	capability_optional	running in an agent session with WebFetch, page is JS-heavy	use WebFetch instead of the builtin script	no WebFetch → W02 builtin (no JS render, partial)	W04
B02	capability_optional	premium backend configured and `verified:true`	crawl via firecrawl/crawl4ai/jina adapter (JS, whole-site)	not wired → stay on builtin, state its limits (RULE-03)	W04
B03	user_optional	local HTML already downloaded	`web-crawl.py md <file.html>` instead of fetch	—	W04

ID	类型	条件	行为	跳过/失败	汇合
B01	capability_optional	在 agent 会话中运行且可使用 WebFetch，页面为 JS 密集型	使用 WebFetch 代替内置脚本	无 WebFetch → W02 内置后端（无 JS 渲染，仅部分完成）	W04
B02	capability_optional	已配置高级后端且 `verified:true`	通过 firecrawl/crawl4ai/jina adapter 抓取（支持 JS、整站）	未接通 → 保持使用内置后端，并说明其限制 (RULE-03)	W04
B03	user_optional	已下载本地 HTML	使用 `web-crawl.py md <file.html>` ，而不是 fetch	—	W04

Validation và stopping

验证与停止条件

Backend is healthy when

python3 harness/scripts/web-crawl.py --self-test

passes. Done when the markdown file exists in

raw/

or a draft and the reply names the backend used. Quality of builtin output (extraction, JS content) is review-only, not claimed.

当

python3 harness/scripts/web-crawl.py --self-test

通过时，说明后端健康。完成标准是 markdown 文件已存在于

raw/

或草稿中，且回复中注明了所用的后端。内置后端的输出质量（提取效果、JS 内容）仅供复核，不得作出超出实际能力的声明。

Examples

示例

Positive: "crawl https://example.com/blog/post into raw" (static page, no key) →
```
python3 harness/scripts/web-crawl.py fetch "https://example.com/blog/post" --out raw/example-post.md
```
→ markdown saved, then suggest
```
ingest
```
.
Boundary/failure: "scrape the whole docs site" with
```
verified:false
```
→ builtin is single page only → report that a premium backend must be wired in
```
harness/web-crawl.config.yaml
```
; do not claim the site was crawled.

正向示例：“将 https://example.com/blog/post 抓取到 raw”（静态页面，无需 key）→
```
python3 harness/scripts/web-crawl.py fetch "https://example.com/blog/post" --out raw/example-post.md
```
→ markdown 已保存，然后建议执行
```
ingest
```
。
边界/失败示例：“抓取整个文档站点”且
```
verified:false
```
→ 内置后端仅支持单页 → 报告必须在
```
harness/web-crawl.config.yaml
```
中接通高级后端；不得声称已抓取该站点。

Reference — Related

参考——相关内容

harness/scripts/web-crawl.py

harness/web-crawl.config.yaml

(the backend adapter).

Real engines (the upgrade): Firecrawl (
```
github.com/mendableai/firecrawl
```
, managed + MCP), Crawl4AI (
```
github.com/unclecode/crawl4ai
```
, OSS self-host), Jina Reader. MinerU (in your stars) for PDFs/docs.
```
/web-clone
```
— when you want the page's UI/look, not its text.
```
build-now-adapt-later
```
— the quarantine pattern this backend follows.

harness/scripts/web-crawl.py

harness/web-crawl.config.yaml

（后端适配器）。

**真正的引擎（升级选项）：**Firecrawl（
```
github.com/mendableai/firecrawl
```
，托管式 + MCP）、Crawl4AI（
```
github.com/unclecode/crawl4ai
```
，开源、自托管）、Jina Reader。MinerU（位于你的收藏中）适用于 PDF/文档。
```
/web-clone
```
——当需要页面的 UI/外观而非文本时使用。
```
build-now-adapt-later
```
——此后端遵循的隔离模式。