blocked-page-recovery

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Blocked-Page Recovery

被拦截页面恢复

When a page won't fetch — 403/429, Cloudflare "Just a moment...", a paywall, or a bot-detection interstitial — don't give up and don't loop on the same URL. Third-party services often hold a copy of the page. Work down this ladder, cheapest first.

当页面无法获取时——比如返回403/429状态码、Cloudflare "Just a moment..."验证页面、付费墙或机器人检测弹窗——不要放弃，也不要反复请求同一个URL。第三方服务通常会保存页面的副本。按照以下优先级尝试，优先使用成本最低的方案。

The ladder

方案优先级

1. Wayback Machine  — archive.org "available" API  (snapshot + timestamp)
2. archive.today    — domain rotation: archive.ph → .md → .li → .is
3. Jina Reader      — only if JINA_API_KEY is set  (live server-side render)
4. API-first pivot  — look for /api/, /graphql, .json, or RSS on the same host
5. Real browser     — browser tool as the last, most expensive resort

Run it in one shot with the bundled script:

bash

python3 scripts/recover_page.py "https://example.com/blocked-article" --json

The script tries each route in order, validates every body (see "Fake successes" below), and prints the first genuine hit with its provenance.

1. Wayback Machine  — archive.org "available" API  (快照 + 时间戳)
2. archive.today    — 域名轮换：archive.ph → .md → .li → .is
3. Jina Reader      — 仅在设置JINA_API_KEY时可用  (服务器端实时渲染)
4. API优先转向  — 在同一主机下查找/api/、/graphql、.json或RSS接口
5. 真实浏览器  — 将浏览器工具作为最后、成本最高的备选方案

使用捆绑脚本一键运行：

bash

python3 scripts/recover_page.py "https://example.com/blocked-article" --json

该脚本会按顺序尝试每个方案，验证返回内容（见下文“虚假成功”部分），并输出第一个真实有效的结果及其来源信息。

Provenance discipline (non-negotiable)

来源规范（必须遵守）

Every recovered copy carries a provenance you MUST preserve when citing:

Route	Provenance	How to cite
Wayback / archive.today	`snapshot`	Cite WITH the snapshot date: "as archived 2026-08-06". Never present a snapshot as the live page — it may be stale.
Jina Reader	`live`	Server-side re-render of the live page; cite normally.
Live fetch / browser	`live`	Cite normally.

If the user needs current data (prices, availability, breaking news), a snapshot is context, not an answer — say so explicitly and note its age.

每个恢复的副本都带有来源信息，引用时必须保留：

方案	来源类型	引用方式
Wayback / archive.today	`snapshot`	引用时必须包含快照日期：“存档于2026-08-06”。绝不能将快照当作实时页面展示——它可能已过期。
Jina Reader	`live`	服务器端实时渲染的页面；按常规方式引用。
实时获取 / 浏览器	`live`	按常规方式引用。

如果用户需要当前数据（如价格、库存、突发新闻），快照仅作为上下文参考，而非答案——需明确说明这一点并标注其时效性。

Manual routes

手动操作步骤

1. Wayback Machine (best provenance, try first)

1. Wayback Machine（来源可信度最高，优先尝试）

bash

undefined

bash

undefined

Discovery: returns closest snapshot URL + timestamp as JSON

查找：返回最近的快照URL + 时间戳（JSON格式）

curl -sL "https://archive.org/wayback/available?url={URL}"

Then fetch archived_snapshots.closest.url

然后获取archived_snapshots.closest.url对应的内容


For enumerating many snapshots (or recovering deleted pages), the CDX index:

```bash
curl -sL "https://web.archive.org/cdx/search/cdx?url={URL}&output=json&limit=10"

CDX intermittently returns 503 under load — if it does, fall back to the

available

API; don't retry-hammer it.

Works for: any publicly crawled URL. Fails for: robots-blocked sites, never-crawled URLs, JS-only SPAs (snapshots don't render).


如需枚举多个快照（或恢复已删除页面），可使用CDX索引：

```bash
curl -sL "https://web.archive.org/cdx/search/cdx?url={URL}&output=json&limit=10"

CDX在负载过高时会间歇性返回503状态码——遇到这种情况，改用

available

API；不要反复重试请求。

适用场景：任何公开爬取过的URL。失效场景：被robots协议拦截的网站、从未被爬取的URL、纯JS单页应用（快照无法渲染）。

2. archive.today (paywalls, deleted content)

2. archive.today（突破付费墙、恢复已删除内容）

User-submitted archives — often has paywalled news articles Wayback lacks. Rate-limits aggressively (429) and rotates domains, so iterate:

bash

for d in archive.ph archive.md archive.li archive.is; do
  curl -sL --max-time 20 "https://$d/newest/{URL}" -o /tmp/page.html \
    -w "%{http_code}" && break
done

Validate the body, not the status code — a 429 still ships several KB of rate-limit HTML that looks like a success to a size check alone.

用户提交的存档——通常包含Wayback没有的付费新闻文章。限制请求频率严格（返回429）且会轮换域名，因此可按以下方式遍历：

bash

for d in archive.ph archive.md archive.li archive.is; do
  curl -sL --max-time 20 "https://$d/newest/{URL}" -o /tmp/page.html \
    -w "%{http_code}" && break
done

验证返回内容，而非状态码——返回429时仍会返回数KB的限流提示HTML，仅通过大小检查会误判为成功。

3. Jina Reader (requires JINA_API_KEY)

3. Jina Reader（需要JINA_API_KEY）

r.jina.ai

re-renders the live page in a real browser server-side and returns markdown. Anonymous access is dead (401 → Turnstile); a key is required:

bash

curl -s -H "Authorization: Bearer $JINA_API_KEY" "https://r.jina.ai/{URL}"

Handles JS SPAs that archives can't. Skip this route entirely when the env var is unset.

r.jina.ai

会在服务器端通过真实浏览器重新渲染实时页面，并返回markdown格式内容。匿名访问已失效（返回401 → 触发Turnstile验证）；必须使用密钥：

bash

curl -s -H "Authorization: Bearer $JINA_API_KEY" "https://r.jina.ai/{URL}"

可处理归档工具无法渲染的纯JS单页应用。如果未设置环境变量，请跳过此方案。

4. API-first pivot

4. API优先转向

WAFs protect the HTML surface far more aggressively than the data endpoints behind it. After 2-3 blocked attempts on a site, stop fighting the HTML and look for:

```
/api/...
```
,
```
/graphql
```
, or
```
.json
```
variants of the page URL
An RSS/Atom feed (
```
/feed
```
,
```
/rss
```
,
```
<link rel="alternate">
```
in any copy you did recover)
A sitemap (
```
/sitemap.xml
```
) revealing canonical URLs that may not be gated

WAF对HTML页面的防护远比对后端数据接口的防护严格。在某个网站尝试2-3次被拦截后，不要再执着于HTML页面，转而查找：

页面URL对应的/api/...、/graphql或.json变体
RSS/Atom订阅源（/feed、/rss，或在已恢复的任何页面副本中查找
```
<link rel="alternate">
```
标签）
站点地图（/sitemap.xml），其中可能包含未被限制的规范URL

Fake successes — routes that LIE

虚假成功——返回误导性结果的方案

These return HTTP 200 with a plausible body that is NOT the page. The script rejects them automatically; reject them manually too:

Google Cache is dead (since mid-2024).
```
webcache.googleusercontent.com
```
returns 200 + tens of KB, but it's a Google Search interstitial with a JS redirect, not a cache. Never use it.
AMP caches (
```
*.cdn.ampproject.org
```
) mostly return a ~300-byte
```
<title>Redirecting</title>
```
meta-refresh stub pointing back at the original (blocked) URL. Treating that as success creates a fetch loop.
Rate-limit bodies: archive.today 429 pages are multi-KB HTML. Check for the target's actual content (title words, expected strings), not just size.

Detection heuristics the script applies: body under a per-route byte floor; meta-refresh/JS-redirect stubs whose target is the original host; interstitial titles ("Just a moment", "Redirecting", "Google Search", "Attention Required").

这些方案会返回HTTP 200状态码和看似合理的内容，但并非目标页面。脚本会自动识别并拒绝这些结果；手动操作时也需注意：

Google Cache已停用（自2024年年中起）。
```
webcache.googleusercontent.com
```
会返回200状态码和数十KB内容，但这是带有JS重定向的Google搜索弹窗，而非缓存页面。请勿使用。
AMP缓存（
```
*.cdn.ampproject.org
```
）大多会返回约300字节的
```
<title>Redirecting</title>
```
元刷新代码，指向原（被拦截的）URL。将其视为成功会导致请求循环。
限流提示内容：archive.today的429页面是数KB的HTML。需检查是否包含目标页面的实际内容（标题关键词、预期字符串），而非仅检查内容大小。

脚本采用的检测规则：内容大小低于对应方案的字节下限；元刷新/JS重定向的目标为原主机；弹窗标题（如“Just a moment”、“Redirecting”、“Google Search”、“Attention Required”）。

Proxy relays: don't

代理中继：请勿使用

Generic "web proxy" relays are man-in-the-middle by construction. Never send cookies or Authorization headers through one, and don't use them for anything the user will rely on — provenance is unverifiable. Prefer archives, which at least timestamp their copies.

通用“网页代理”中继本质上是中间人攻击。切勿通过此类代理发送Cookie或Authorization头，也不要将其用于用户依赖的任何内容——来源无法验证。优先使用归档工具，至少它们会为副本添加时间戳。