read-deleted-pages

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Read deleted pages

读取已删除的页面

Archives work because removal is a decision made after publication. The beginner's mistake is treating the Wayback Machine as a website you browse: the calendar UI shows one page at a time, while the CDX API enumerates every URL the archive ever saw under a host, including paths nobody remembers.

归档之所以有效，是因为删除通常是在内容发布之后才做出的决定。新手常犯的错误，是把 Wayback Machine 当作一个可以浏览的网站：日历 UI 每次只显示一个页面，而 CDX API 能枚举归档曾在一个 host 下发现的每一个 URL，包括无人记得的路径。

Which archive first

首先选择哪个归档源

Situation	Go to	Why
A URL that 404s now	Wayback, then archive.today	Different corpora, different removal pressure
"What did this site say before X"	Wayback CDX with `collapse=digest`	Gives the dates content changed, not every capture
A JS-heavy page or a social post	archive.today	Captures the rendered DOM; Wayback often replays an empty shell
Wayback says the URL is excluded	archive.today, then Common Crawl	Neither answers to Wayback's exclusion process
Paths and subdomains you don't know about	CDX with `matchType=domain`	It is an enumeration tool, not a lookup tool
A URL Wayback never captured	Common Crawl	Independent crawler, different seeds
Regional or national-TLD content	A Memento aggregator, then the national archive	Some national archives offer full-text search
Evidence you cannot lose	Capture it yourself, then submit to two archives	Nobody else is preserving it for you

Coverage and removal policy per source: reference/archive-comparison.md.

情况	首选工具	原因
现在返回 404 的 URL	Wayback，然后 archive.today	归档语料库不同，删除压力也不同
“这个网站在 X 之前说了什么”	使用 Wayback CDX，并设置 `collapse=digest`	显示内容发生变化的日期，而不是每次捕获
JavaScript 密集型页面或社交帖子	archive.today	可捕获渲染后的 DOM；Wayback 通常只回放一个空壳
Wayback 提示该 URL 已被排除	archive.today，然后 Common Crawl	二者都不受 Wayback 排除流程的约束
不知道存在的路径和子域名	使用设置了 `matchType=domain` 的 CDX	它是枚举工具，而不是查询工具
Wayback 从未捕获过的 URL	Common Crawl	独立的 crawler，种子来源也不同
区域性网站或国家/地区级 TLD 内容	先使用 Memento 聚合器，再查询国家归档	部分国家归档提供全文搜索
不能丢失的证据	自行捕获，然后提交给两个归档源	不能指望其他人替你保存

各来源的覆盖范围与删除政策： reference/archive-comparison.md。

The CDX API is the actual tool

CDX API 才是真正的工具

The CDX server indexes capture records — one row per capture — and lets you filter, collapse and enumerate them. Every URL a crawler ever saw under a host, deduplicated:

bash

curl -s 'http://web.archive.org/cdx/search/cdx?url=example.com/*&output=json&fl=original&collapse=urlkey' \
  | jq -r '.[1:][] | .[0]' | sort -u

Use

output=json

rather than the default text: it survives URLs containing spaces and quotes, which the text format mangles. The first JSON row is a header naming the columns, not a capture — hence the

.[1:]

Four parameters do nearly all the work:

matchType
—
```
exact
```
,
```
prefix
```
,
```
host
```
, or
```
domain
```
.
```
matchType=domain
```
returns the host and all subdomains, routinely surfacing staging, legacy and internal hosts long gone from DNS. Take them to
```
find-hidden-subdomains
```
.
filter=<field>:<regex>
, invertible with
```
!
```
and combinable. Filter to
```
statuscode:200
```
for captures that returned content, or
```
!mimetype:text/html
```
to isolate documents, archives and config files.
from
/
to
take a timestamp prefix at any precision —
```
from=2019
```
,
```
from=201903
```
— so you can bracket the window around a known event.
collapse
suppresses adjacent rows sharing a field.
```
collapse=urlkey
```
is the enumeration idiom;
```
collapse=digest
```
matters most.

digest

hashes the captured payload, so

collapse=digest

reduces a page's history to the captures where content actually changed. Four hundred captures become six rows, and those six dates are the story.

bash

curl -s 'http://web.archive.org/cdx/search/cdx?url=example.com/about/team&output=json&collapse=digest&fl=timestamp,digest,statuscode'

Full parameters, paging, and a ready enumeration pass: reference/cdx-cookbook.md.

CDX server 会索引捕获记录——每次捕获对应一行——并允许你筛选、折叠和枚举这些记录。它列出了 crawler 曾在某个 host 下见过的所有 URL，并进行了去重：

bash

curl -s 'http://web.archive.org/cdx/search/cdx?url=example.com/*&output=json&fl=original&collapse=urlkey' \
  | jq -r '.[1:][] | .[0]' | sort -u

请使用

output=json

，而不是默认的文本格式：这样可以正确处理包含空格和引号的 URL，而文本格式会破坏这些内容。JSON 的第一行是用于说明各列名称的表头，并非捕获记录，因此需要使用

.[1:]

跳过它。

以下四个参数几乎可以完成所有工作：

matchType
——可以是
```
exact
```
、
```
prefix
```
、
```
host
```
或
```
domain
```
。
```
matchType=domain
```
会返回该 host 及其所有子域名，通常还能发现早已从 DNS 中消失的 staging、legacy 和 internal host。可再使用
```
find-hidden-subdomains
```
深入调查。
filter=<field>:<regex>
——可使用
```
!
```
进行反向筛选，也可以组合使用。筛选
```
statuscode:200
```
可获得返回了内容的捕获；使用
```
!mimetype:text/html
```
则可单独找出文档、归档文件和配置文件。
from
/
to
——接受任意精度的时间戳前缀，例如
```
from=2019
```
、
```
from=201903
```
，可用于划定已知事件前后的时间窗口。
collapse
——抑制共享同一字段的相邻行。
```
collapse=urlkey
```
是 URL 枚举的惯用方式，而
```
collapse=digest
```
最为关键。

digest

是对捕获 payload 计算的哈希值，因此

collapse=digest

会将页面的历史记录缩减为内容实际发生变化的那些捕获。400 次捕获会变成 6 行，而这 6 个日期就勾勒出了内容的发展历程。

bash

curl -s 'http://web.archive.org/cdx/search/cdx?url=example.com/about/team&output=json&collapse=digest&fl=timestamp,digest,statuscode'

完整参数、分页方式及可直接使用的枚举流程： reference/cdx-cookbook.md。

Replay is not the capture

Replay 不等于 Capture

The page at

web.archive.org/web/<timestamp>/<url>

is reconstructed. The archive injects a toolbar, rewrites links and resource references back into the archive, and — the part that misleads people — pulls each resource from its own nearest capture, which may be months away from the HTML you asked for. The rendered page you screenshot may never have existed.

Append

id_

to the timestamp to get the raw archived response with no injection and no rewriting:

bash

curl -s 'https://web.archive.org/web/20190412093000id_/http://example.com/pricing' > pricing-2019.html

Use

id_

for anything you will parse, hash, diff, or cite. A page that replays as blank may have been captured fine, so fetch the raw response before concluding the archive has nothing — but archived JavaScript that called an API at view time has no data behind it, because the API response was never captured. Those pages are shells by nature, not by failure.

web.archive.org/web/<timestamp>/<url>

处的页面是重新构建的。归档会注入工具栏，将链接和资源引用重写到归档中；更具有误导性的一点是，它会从每个资源各自最近的捕获中提取该资源，而这个时间点可能比所请求的 HTML 早数月。你所看到的渲染后页面可能从未真实存在过。

在 timestamp 后附加

id_

，即可获取未经注入和重写的原始归档响应：

bash

curl -s 'https://web.archive.org/web/20190412093000id_/http://example.com/pricing' > pricing-2019.html

对于需要解析、计算哈希、比较差异或引用的内容，应始终使用

id_

。一个回放后显示为空白页面的 URL 可能已被完整捕获，因此在断定归档中没有任何内容之前，应先获取原始响应。不过，如果归档的 JavaScript 在回放时调用了 API，并且 API 响应从未被捕获，那么其背后自然不会有数据。这些页面本来就是空壳，而不是捕获失败。

What this actually recovers

实际可以恢复什么

Pull the raw captures at two dates from the

collapse=digest

list and compare them as text. The removals are the evidence: a person deleted from a team page between two dates is a departure with a date range attached, and a

find-anyone

lead with no remaining reason to be loyal. What surfaces, in practice:

Staff and team pages. The current site lists who is there now; the archive lists everyone who ever was, with titles, bios, and often direct email addresses from before the org moved to a contact form.
Prior pricing and terms — what was charged, what was promised, and what the refund policy said on the date a dispute began. Also removed posts and press releases: partnerships, funding or clients both sides stopped mentioning.
Old contact details. Phone numbers, street addresses and named individual contacts later replaced by a form. Straight into
```
whose-number-is-this
```
.
Pre-redaction copy. Diffing a page published in full against its edited successor shows exactly which words somebody paid a lawyer to remove — a very strong signal about what matters.
Forgotten files — document paths, backup filenames and admin surfaces indexed once and never cleaned up. See
```
secrets-in-git-history
```
.
The prior owner of a domain. A domain that changed hands shows a different business in its archive history — the fastest way to spot a reputation-laundering purchase.

从

collapse=digest

列表中选择两个日期，获取对应的原始捕获并以文本形式进行比较。被删除的内容本身就是证据：某人如果在两个日期之间从团队页面中被删除，就意味着可以确定其离职时间范围；而一个通过

find-anyone

找到、已没有理由继续效忠的线索对象，也可能由此暴露。实际应用中通常会发现：

员工和团队页面。 当前网站列出的是现在的人员；归档记录的则是所有曾经存在过的人员，包括职位、简介，以及组织改用联系表单前的直接邮箱地址。
以往价格和条款——当时收取多少费用、作出过什么承诺，以及争议发生当日退款政策如何规定。还包括已删除的帖子和新闻稿：双方后来都不再提及的合作、融资或客户。
旧联系方式。 后来被表单取代的电话号码、街道地址和具体联系人。可直接转至
```
whose-number-is-this
```
。
删改前文案。 将完整发布的页面与其编辑后的版本进行 diff，可以准确看出有人花钱请律师删除了哪些文字——这是判断哪些内容真正重要的有力信号。
被遗忘的文件——曾经建立索引、之后从未清理的文档路径、备份文件名和管理界面。参见
```
secrets-in-git-history
```
。
域名的前任所有者。 域名易主后，其归档历史会显示另一家企业——这是识别信誉洗白型收购的最快方法。

Preserve it before you look further

在继续调查前先保全证据

Anything you find can be removed, and investigating sometimes causes the removal. Capture first, analyse second.

Submit the live URL to the Internet Archive's Save Page Now at

https://web.archive.org/save/

and to archive.today, so two independent organisations hold a timestamped copy you did not create. Then take your own record: a WARC preserves the full HTTP exchange including headers, which a screenshot does not.

bash

wget --warc-file=evidence-001 --page-requisites --no-parent https://example.com/pricing
sha256sum evidence-001.warc.gz | tee evidence-001.sha256

Record the capture time in UTC, the URL exactly as requested, and the hash. A screenshot alone is weak — trivially forged, no headers. A third-party archive URL plus your own hashed WARC is strong, because they corroborate each other and one is outside your control. That is what

write-the-intel-brief

expects.

找到的任何内容都可能被删除，而调查本身有时就会促使内容被删除。先捕获，再分析。

将 live URL 提交给 Internet Archive 的 Save Page Now：

https://web.archive.org/save/

，同时提交给 archive.today，这样两个彼此独立的组织就会各自持有一份带 timestamp 的副本，而且都不是由你创建的。随后再制作自己的记录：WARC 会保存完整的 HTTP exchange，包括 headers，而截图做不到这一点。

bash

wget --warc-file=evidence-001 --page-requisites --no-parent https://example.com/pricing
sha256sum evidence-001.warc.gz | tee evidence-001.sha256

记录 UTC 捕获时间、请求时使用的准确 URL 以及哈希值。单独的截图证明力很弱——它很容易被伪造，而且不包含 headers。第三方归档 URL 加上你自己生成并经过哈希的 WARC 则具有较强证明力，因为二者可以相互印证，而且其中一个不受你控制。这正是

write-the-intel-brief

所要求的证据形式。

Verifying a capture is genuine

验证捕获的真实性

Anyone can put content into an archive: both Save Page Now and archive.today accept arbitrary URLs from anonymous submitters. A capture proves that URL served that content to that crawler at that time, and nothing about who controlled the URL. The realistic spoofing routes:

A forged screenshot of an archive page. Trivial. Never accept a screenshot of an archive; work from the archive URL itself and re-fetch it.
Content injected via the URL. A reflected query parameter, a user-generated content path or an open redirect lets someone archive attacker-chosen text on a legitimate-looking domain. Read the
```
original
```
field character by character: anything after
```
?
```
or
```
#
```
, any unfamiliar path segment, any lookalike host.
Capture after a domain changed hands. The domain was the buyer's, not the original business's. Check transfer dates against the capture timestamp with
```
who-owns-this-domain
```
.
Selective capture — a real page up for eleven minutes, archived by the person who put it up.

Corroboration, in order of strength: confirm the record exists independently in the CDX index with

statuscode

200 and a

digest

; look for adjacent captures made by the archive's own crawler rather than by on-demand submission, since crawler captures are much harder to stage; check a second independent archive near the same date; check consistency with the site's other history. A lone on-demand capture with no neighbours is an assertion by its submitter.

任何人都可以将内容放入归档：Save Page Now 和 archive.today 都接受匿名提交者提供的任意 URL。捕获记录只能证明该 URL 在该时间向该 crawler 提供了该内容，无法证明当时由谁控制该 URL。现实中常见的伪造方式包括：

伪造归档页面截图。 非常容易。不要接受任何归档截图；应直接使用归档 URL 重新获取内容。
通过 URL 注入内容。 反射型 query parameter、用户生成内容路径或 open redirect，都可能使攻击者将自行选择的内容归档到一个看似合法的域名下。逐字符检查
```
original
```
字段：
```
?
```
或
```
#
```
之后的任何内容、任何陌生路径段，以及任何相似的 host 都应被视为风险信号。
域名易主后进行捕获。 此时域名属于买方，而非原企业。使用
```
who-owns-this-domain
```
将所有权转让日期与捕获 timestamp 进行核对。
选择性捕获——让一个真实页面上线 11 分钟，再由发布该页面的人完成归档。

按证明力从强到弱进行交叉验证：确认该记录独立存在于 CDX index 中，并具有

statuscode

200 和

digest

；查找由归档自身 crawler 而非按需提交生成的相邻捕获，因为 crawler 捕获更难人为布置；检查同一日期附近是否存在第二个独立归档；检查内容与网站其他历史记录是否一致。孤立的按需捕获且没有相邻记录，只代表提交者单方面作出的主张。

Where this goes wrong

容易出错的地方

Absence is not evidence. No capture means unlinked, robots-blocked, behind a login, too obscure to crawl, or removed — not that the page never existed. Check two archives before writing "no record".
Exclusion looks like absence. An explicit exclusion notice means somebody acted; a plain 404 means nothing. Different facts — note which you got.
Retroactive removal. Wayback has historically applied a site's current robots.txt to its whole archive history, so a new
```
Disallow
```
could pull years of captures at once — making an expired-domain purchase plus a restrictive robots.txt a cheap way to launder a site's past. The practice has narrowed, but go to archive.today and Common Crawl when a history looks too clean.
Composite renders. Mixed-date resources mean the rendered page is a reconstruction. Cite the HTML capture timestamp, not "how it looked".
Capture time is not publication time. A page first captured in March may have been published in January. The capture bounds the date from above only.
Dynamic and personalised content. The crawler saw one variant — one geography, one A/B bucket, logged out. Prices and availability especially.
Sparse sampling hides everything. A page captured twice a year can change and revert with no trace;
```
collapse=digest
```
shows observed changes, not all changes. Timestamps are UTC, and forgetting that breaks a timeline by a day.

不存在记录并不能证明证据不存在。 没有捕获可能意味着页面未被链接、受到 robots 阻止、位于登录后、过于冷门而未被 crawler 发现，或者已被删除——不能据此断定该页面从未存在。在写下“没有记录”之前，应检查两个归档源。
排除看起来像是没有记录。 明确的排除通知意味着有人采取了行动；普通 404 本身则不能说明任何问题。这是两种不同的事实——应记录实际遇到的是哪一种。
追溯性移除。 Wayback 过去曾将网站当前的 robots.txt 应用于其全部归档历史，因此新增的
```
Disallow
```
可能一次移除多年的捕获记录——这使得购买一个已过期域名，再配置限制性很强的 robots.txt，成为清洗网站历史记录的廉价手段。这种做法后来有所收敛，但当历史记录干净得不自然时，仍应使用 archive.today 和 Common Crawl。
复合渲染。 来自不同日期的混合资源意味着渲染页面只是重建结果。应引用 HTML 捕获的 timestamp，而不是“页面当时看起来如何”。
捕获时间不等于发布时间。 一个 3 月才首次被捕获的页面，可能在 1 月就已发布。捕获时间只能给出发布日期的上界。
动态和个性化内容。 crawler 只看到了其中一个变体——一个地区、一个 A/B 分组，或未登录状态下的页面。价格和可用性尤其容易出现这种情况。
稀疏采样会隐藏所有变化。 一个每年仅捕获两次的页面可能多次更改后又恢复原状，且不留任何痕迹；
```
collapse=digest
```
只能显示已观察到的变化，而不是全部变化。timestamp 使用 UTC，忽略这一点会让时间线偏差一天。

Confidence grading

置信度分级

Confirmed — a crawler-initiated capture with
```
statuscode:200
```
, retrieved raw with
```
id_
```
, on a URL verified as the subject's, corroborated by a second archive or by adjacent captures in the same crawl series.
Probable — a single well-formed Wayback capture with no second source, or an archive.today capture consistent with the site's other history.
Unconfirmed — an on-demand capture with no neighbours; a URL carrying query parameters or user-generated content paths; a screenshot; content read from a composite replay rather than the raw response.
Rejected — a capture whose URL was not under the subject's control at that date, or one taken after a domain transfer and presented as the prior owner's.

Cite the full archive URL with timestamp, the original URL, the capture time in UTC, and how you retrieved it. An archive citation without a timestamp is not a citation.

已确认——由 crawler 主动发起的捕获，
```
statuscode:200
```
，使用
```
id_
```
获取原始响应，URL 已验证处于目标主体控制之下，并得到第二个归档源或同一 crawl 系列中相邻捕获的印证。
** probable**——单条格式正确的 Wayback 捕获且没有第二来源，或 archive.today 捕获与网站其他历史记录一致。
未确认——没有相邻记录的按需捕获；带有 query parameters 或用户生成内容路径的 URL；截图；或读取自复合 replay 而非原始响应的内容。
已否定——捕获时该 URL 并不在目标主体控制之下，或是在域名转让后捕获，却被当作原所有者时期的页面。

引用时应包含带 timestamp 的完整归档 URL、原始 URL、UTC 捕获时间以及获取方式。没有 timestamp 的归档引用不能算作引用。

Worked example

实例

A supplier's site claims an industry certification. The client wants it verified before signing.

collapse=digest

on the certifications page returns nine content changes over six years. The raw captures either side of the most recent one show the certification body's name changed — from a recognised accreditor to a similarly named entity — with no announcement.

The dead end: the accreditor's own register has no live entry either way, and its archived member lists replay as empty shells. Fetching them with

id_

shows why — the list was rendered client-side from an API call, so the crawler captured a page with no data in it. Nothing recoverable there.

CDX enumeration on

matchType=domain

then turns up a staging host holding an archived PDF of the original certificate. Its metadata carries an expiry date two years before the current site claims coverage, and

secrets-in-file-metadata

gets an author name from the same file.

Grade: the wording change confirmed — two crawler captures, raw, both read. The lapse probable — one document, no confirmation from the accreditor.

某供应商的网站声称拥有一项行业认证。客户希望在进行签约前验证该认证。

在认证页面上使用

collapse=digest

，会返回六年内九次内容变化。最近一次变化前后的原始捕获显示，认证机构的名称发生了变化——从一个受认可的认证机构，变成了一个名称相似的实体，而且没有发布任何公告。

此路不通：认证机构自己的注册名录中，无论按哪个名称都找不到有效条目，其归档成员列表回放后也只是空壳。使用

id_

获取原始响应后，原因便很清楚——该列表由 client-side API call 渲染，因此 crawler 捕获到的只是一个不含数据的页面。那里没有任何可恢复的内容。

随后使用

matchType=domain

进行 CDX 枚举，发现了一个 staging host，其中保存着原始证书的归档 PDF。其 metadata 显示，到期日期比当前网站所称的覆盖期早两年；

secrets-in-file-metadata

还能从同一个文件中获取作者姓名。

评级：措辞变化为已确认——存在两次 crawler 捕获，均为原始响应，且都已读取。认证失效为** probable**——只有一份文件支持，且未得到认证机构确认。

Pivots

后续方向

New selector	Goes to
Former staff names, titles, bios	`find-anyone` , `pattern-of-life-from-socials`
Email addresses from pre-form contact pages	`what-an-email-reveals`
Phone numbers and street addresses	`whose-number-is-this`
Subdomains and hosts from CDX enumeration	`find-hidden-subdomains` , `who-owns-this-domain`
Archived documents and PDFs	`secrets-in-file-metadata`
Archived images	`find-the-original-image` , `geolocate-from-pixels`
Forgotten paths and exposed file names	`google-like-a-spy` , `secrets-in-git-history`
Prior corporate identity of a domain	`x-ray-a-company` , `who-really-owns-it`
A timeline of edits and removals	`write-the-intel-brief` , `graph-the-network`

新选择器	转向
旧员工姓名、职位和简介	`find-anyone` 、 `pattern-of-life-from-socials`
改用联系表单前的邮箱地址	`what-an-email-reveals`
电话号码和街道地址	`whose-number-is-this`
通过 CDX 枚举发现的子域名和 host	`find-hidden-subdomains` 、 `who-owns-this-domain`
归档文档和 PDF	`secrets-in-file-metadata`
归档图片	`find-the-original-image` 、 `geolocate-from-pixels`
被遗忘的路径和暴露的文件名	`google-like-a-spy` 、 `secrets-in-git-history`
域名以往的企业身份	`x-ray-a-company` 、 `who-really-owns-it`
编辑和删除的时间线	`write-the-intel-brief` 、 `graph-the-network`

Legal and handling notes

法律与处理注意事项

Reading an archive is passive; it touches the archive's servers, not the target's. Fetching a recovered path against the live site is not passive and lands in the target's logs — decide deliberately which you are doing.

Archived personal data is still personal data. Content removed under a right-to-erasure request may persist in archives, and republishing it can create the liability the removal was meant to extinguish. Weigh the investigative need against the reason for removal, and minimise per ../../ETHICS.md. Reproducing substantial archived content in a published report is a separate copyright question from using it as evidence.

Bulk retrieval strains a nonprofit's infrastructure: rate-limit enumeration and raw fetches, and pull the index once rather than repeatedly.

读取归档是被动行为；它只会访问归档服务器的流量，而不会触及目标站点。对 live site 请求已恢复的路径则不是被动行为，并且会出现在目标站点的日志中——应明确判断自己正在进行的是哪一种操作。

归档中的个人数据仍然属于个人数据。根据删除权请求移除的内容可能继续存在于归档中，而重新发布这些内容可能造成原本希望通过删除来消除的责任。应权衡调查需求与删除原因，并按照 ../../ETHICS.md 尽量减少数据处理。在公开报告中大量转载归档内容，还涉及一个独立于“将内容作为证据使用”的版权问题。

批量获取会给非营利组织的基础设施带来压力：应限制枚举和原始获取的请求速率，并只拉取一次 index，而不是反复获取。