Back to Details

verify-docs-findable

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

verify-docs-findable — cold-read the docs with fresh eyes

verify-docs-findable — 用全新视角冷读文档

The insight this operationalizes: complete ≠ findable. An index that lists every doc (and even passes a "every doc is indexed" check) can still make a reader take 5 hops and fall back to source code to answer a basic question. The only honest test is to give the question to someone who has never seen the repo and watch how far they get from the docs alone. A fresh cheap agent is that someone.
本方法的核心洞见是：完整 ≠ 可查找。一个列出所有文档的索引（甚至通过了“所有文档已索引”的检查）仍可能让读者跳转5次，最后不得不查看源代码才能解答一个基础问题。唯一可靠的测试方法是，将问题交给从未接触过该仓库的人，观察他们仅通过文档能走多远。全新的低成本Agent就是这样的“测试者”。

Why HAIKU, and why "docs only"

为什么选择HAIKU，为什么要求“仅用文档”

  • Haiku (a small, cheap model), fresh (no session context). The point is a naive cold reader, NOT a clever one. A strong model with your context will grep the source, reason around gaps, and hide the doc problem. Haiku, told to use only the docs, exposes it.
  • "Use ONLY the docs, start at the index." The moment a verifier has to open source code to answer, the docs have failed — that's the signal. Steps-from-index is the metric.
  • Haiku（一款小型低成本模型），全新状态（无会话上下文）。我们需要的是天真的冷读者，而非聪明的读者。具备上下文的强大模型会搜索源代码、填补缺口进行推理，从而掩盖文档存在的问题。而Haiku在被要求仅使用文档时，会暴露这些问题。
  • “仅使用文档，从索引开始”。一旦验证者不得不打开源代码来解答问题，就说明文档失败了——这就是信号。从索引开始的跳转步数是衡量指标。

Procedure

流程

1. Get the questions (3–8)

1. 获取问题（3–8个）

Test the recurring, load-bearing, and mis-stated questions — the ones that actually cost time. Sources, in order:
  • What the user (or you) just got confused about this session — the highest-signal source.
  • The "canonical answers" the docs claim to provide (a docs corpus usually has ~5–8 facts it exists to convey; test those).
  • If none given: ask the user for the questions, or infer them from the corpus's top-level topics. Don't invent trivia — test what a real contributor needs.
Phrase each as a real question a reader would ask, not a doc title.
测试反复出现、关键且表述不清的问题——那些真正耗时的问题。问题来源优先级如下：
  • 你或用户在本次会话中刚遇到的困惑——信号最强的来源。
  • 文档声称要提供的“标准答案”（文档库通常存在5–8个核心要传达的事实；测试这些内容）。
  • 如果没有现成问题：向用户索要问题，或从文档库的顶级主题推断。不要编造无关问题——测试真实贡献者需要了解的内容。
每个问题都要表述为读者会实际提出的真实问题，而非文档标题。

2. Spawn one fresh Haiku verifier per question (in parallel)

2. 为每个问题生成一个全新的Haiku验证者（并行执行）

Use the Agent tool with 
model: "haiku"
, 
run_in_background: true
, one per question. Give each the SAME strict template (fill in 
<QUESTION>
):
You are a fresh engineer opening 
<repo path>
for the first time. Answer using ONLY the repo's docs — START at the index (
<the index file(s)>
) and follow pointers; prefer docs over reading source code. QUESTION: 
<QUESTION>
REPORT: (a) your answer; (b) the exact file(s) where you found it; (c) how many steps from the index (did a pointer take you straight there, or did you dig?); (d) clear/unambiguous or scattered/confusing? (e) did you have to read source code because the docs didn't say? Be honest — this tests whether the docs make this findable.
Run all in one message so they go concurrently. (Haiku + parallel = the audit is cheap.)
使用Agent工具，设置
model: "haiku"
、
run_in_background: true
，每个问题对应一个Agent。为所有Agent提供相同的严格模板（填充
<QUESTION>
）：
你是首次打开
<repo path>
的新工程师。仅使用该仓库的文档来回答问题——从索引（
<the index file(s)>
）开始，跟随链接跳转；优先查看文档，而非阅读源代码。 问题：
<QUESTION>
报告内容：(a) 你的答案；(b) 找到答案的具体文件；(c) 从索引开始的跳转步数（是直接通过链接到达，还是需要逐层查找？）；(d) 内容清晰明确还是杂乱无章？(e) 是否因文档未提及而不得不查看源代码？ 请如实作答——这是为了测试文档是否具备可查找性。
将所有请求放在一条消息中，以便并发执行。（Haiku+并行执行意味着审核成本很低。）

3. Score each answer

3. 为每个答案评分

SignalGoodGap
Correct?matches ground truthwrong/partial → the doc is wrong or missing
Steps from index1–23+ → no signposted pointer
Dead-endsnonelanded on a wrong-but-plausible doc first → title/scoping is misleading
Needed source code?noyes → the docs don't actually state it
Contradictions—cited two docs that disagree → a cohesion bug (fix immediately)
Verifiers surface contradictions for free — a cold reader citing two docs that say different things is the cheapest contradiction-finder you have.
信号良好情况存在缺口的情况
答案是否正确？与基准答案一致错误/不完整 → 文档内容错误或缺失
从索引开始的跳转步数1–2步3步及以上 → 缺少指向明确的链接
是否遇到死胡同？无首先进入了看似合理但错误的文档 → 标题/范围描述存在误导
是否需要查看源代码？否是 → 文档未明确说明该内容
是否存在矛盾？—引用了两个观点相悖的文档 → 存在一致性问题（需立即修复）
验证者会自动发现矛盾——冷读者引用两个内容不同的文档，是成本最低的矛盾检测方式。

4. Fix the gaps

4. 修复缺口

  • 3+ steps / dead-end → add a question→doc pointer at the index entry point (a "Canonical answers: question → the ONE doc" block). Naming the answer is what turns 5 hops into 1.
  • Needed source code → the fact isn't in prose; write it into the one canonical doc.
  • Wrong answer → the doc is stale/incorrect; fix it (append-don't-erase if it records a decision).
  • Contradiction → reconcile the docs; keep the historical decision, mark it superseded.
  • Scattered across N docs → consolidate into ONE doc per question; make the others point to it.
  • 3步及以上跳转 / 死胡同 → 在索引入口处添加问题→文档的直接链接（一个“标准答案：问题 → 对应文档”模块）。明确标注答案能将5步跳转缩短为1步。
  • 需要查看源代码 → 该事实未在文档正文中体现；将其写入对应的标准文档。
  • 答案错误 → 文档内容过时/不正确；修复它（如果文档记录了决策过程，可追加内容而非直接删除）。
  • 存在矛盾 → 调和文档内容；保留历史决策，标记其已被取代。
  • 内容分散在N个文档中 → 将内容整合到一个对应问题的文档中；让其他文档指向该文档。

5. Re-verify (measure, don't assume)

5. 重新验证（用数据衡量，而非主观假设）

Re-run the worst question(s) against the fixed docs. Confirm the number moved (e.g. 5 steps → 2). A fix you didn't re-measure is a guess.
针对修复后的文档，重新运行问题最严重的几个问题。确认指标有所改善（例如从5步跳转变为2步）。未经过重新验证的修复只是猜测。

Output

输出结果

A short findability scorecard (question | correct | steps | needed-source | verdict) + a prioritized fix list. Report the before/after on any question you fixed and re-verified.
一份简短的可查找性评分卡（问题 | 答案是否正确 | 跳转步数 | 是否需要源代码 | 结论）+ 优先级修复列表。对于已修复并重新验证的问题，需报告前后变化。

Notes

注意事项

  • This is orthogonal to "is every doc indexed?" completeness checks — it measures the layer above: can a stranger reach the answer. Run it after any doc reorganization.
  • Keep the question set in the repo (e.g. a 
    docs-findability-questions.md
    ) so the audit is repeatable and the canonical answers stay honest as the corpus grows.
  • Scales down (1 question, 1 agent, to spot-check one fix) and up (the full canonical set).
  • 这与“是否所有文档都已索引”的完整性检查是正交的——它衡量的是更高层面：陌生人能否找到答案。在任何文档重组后都应运行该审核。
  • 将问题集保存在仓库中（例如
    docs-findability-questions.md
    ），以便审核可重复进行，且随着文档库的增长，标准答案始终真实可靠。
  • 可灵活调整规模（小到1个问题、1个Agent，用于抽查某一项修复；大到完整的标准问题集）。