web-fetcher

Original：🇺🇸 English

Translated

1 scriptsChecked / no sensitive code detected

Fetch any URL as clean markdown. ALWAYS use this skill instead of the WebFetch tool when you need to read a URL's content — it has a 5-layer fallback (Jina Reader, defuddle.md, markdown.new, OpenCLI, raw HTML) that produces better results and handles JS-rendered pages (Twitter/X, SPAs), login-required platforms (zhihu, reddit, weibo, xiaohongshu), and complex web pages that WebFetch cannot parse. Invoke whenever the user provides a URL and wants to read, extract, summarize, analyze, or convert its content to markdown. Keywords: 'fetch page', 'read URL', 'grab content from', 'summarize article', 'extract text from webpage', '抓取网页', '读链接', '网页转 markdown'. NOT for: web search without URL, file downloads, screenshots, form filling, or accessibility checks.

13installs

Sourcejiahao-shao1/sjh-skills

Added on2026-04-03

NPX Install

npx skill4agent add jiahao-shao1/sjh-skills web-fetcher

SKILL.md Content

View Translation Comparison →

Web Fetcher

Extract web page content as clean text/markdown from a given URL using a fallback chain of free services.

Usage

bash

python3 <skill-path>/scripts/fetch.py <url>

Save to file:

bash

python3 <skill-path>/scripts/fetch.py <url> -o output.md

Fallback Chain

The script tries these sources in order, falling back on failure:

Jina Reader (
```
r.jina.ai/{url}
```
) — best markdown quality, supports JS-rendered pages
defuddle.md (
```
defuddle.md/{url}
```
) — by Obsidian creator @kepano
markdown.new (
```
markdown.new/{url}
```
) — 3-layer strategy with browser rendering fallback
OpenCLI — platform-specific commands with browser login state (zhihu, reddit, twitter, weibo)
Raw HTML — direct fetch as last resort

When to Use

JS-rendered pages that WebFetch can't handle (Twitter/X, SPAs)
Login-required pages on supported platforms (zhihu, reddit, twitter, weibo, xiaohongshu)
Bulk content extraction
When you need clean markdown instead of summarized content

OpenCLI Supported Platforms

When free services fail, OpenCLI auto-detects the platform from URL and routes to the right command:

URL Pattern	OpenCLI Command
`zhihu.com/question/xxx`	`opencli zhihu question`
`zhuanlan.zhihu.com/p/xxx`	`opencli zhihu download`
`reddit.com/r/.../comments/...`	`opencli reddit read`
`twitter.com/x.com/.../status/xxx`	`opencli twitter thread`
`weibo.com/...`	`opencli weibo search`

Requires:

npm i -g @jackwener/opencli

+ Browser Bridge extension in Chrome/Arc.

Limitations

WeChat articles (微信公众号) not supported by any strategy
OpenCLI requires browser extension setup (one-time)

Rate Limits

Service	Limit
Jina Reader	20 req/min (free), 10M token key available at jina.ai/reader
markdown.new	500 req/day/IP
defuddle.md	Not documented
OpenCLI	No documented limits (uses browser session)