Back to Details

spec-driven-eval

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Spec-Driven Implementation Evaluation

规格驱动实现评估

Evaluate a spec-driven-development (SDD) effort against a PRD case by case (acceptance criterion by acceptance criterion), scoring implementation and tests separately, and roll up to a single comparable final grade. Designed for benchmarking: the same PRD evaluated across different SDD frameworks — and the same effort evaluated twice — must yield comparable, reproducible numbers.
对照PRD逐案（逐条验收标准）评估规格驱动开发（SDD）的成果，分别对实现和测试进行评分，最终汇总为单一的可比较最终等级。本方法专为基准测试设计：同一PRD在不同SDD框架下的评估结果、以及同一成果的两次评估结果，必须具备可比性和可复现性。

Two subjects, two questions

两个评估对象，两个核心问题

This eval answers two independent questions, and keeps their verdicts separate because they fail independently:
  1. How good is the framework at respecting and extracting requirements? — Did it honor the PRD and stay in bounds (respect: 
    Final
    implementation side + Scope 
    S
    ), and did it surface the implicit requirements the PRD only implied, without noise (extract: Elicitation 
    E
    )?
  2. How good is the harness at ensuring they are all implemented? — Does the test suite prove every sanctioned requirement is actually built (
    T
    + Engineering Gates 
    G
    )?
The linkage that makes "all implemented" well-defined: the harness is held accountable for the full sanctioned requirement set = 
PRD acceptance criteria ∪ valid E-additions
. A valid requirement the framework extracted but the harness never tests is a harness miss (
T
), not a framework miss (
E
keeps the extraction credit). Extraction defines the verification target.
Scoring is checklist-based: every criterion is decomposed into atomic binary (MET / UNMET) checks, each backed by 
file:line
evidence. Binary decomposition is the design choice that makes the grade reproducible — graded/Likert scales (
is this a 3 or a 4?
) are the dominant source of evaluator-to-evaluator disagreement; binary checks raise inter-evaluator agreement to roughly human level. Partial credit is derived from the fraction of checks met, never judged on a sliding scale.
本评估回答两个独立的问题，并分别保留判定结果，因为二者的失败是相互独立的：
  1. 框架在遵守和提取需求方面的表现如何？ — 它是否遵循PRD并保持在范围内（遵守性：对应
    Final
    实现侧 + 范围
    S
    ），是否能无干扰地挖掘出PRD仅隐含的需求（提取性：对应需求启发
    E
    ）？
  2. 测试套件在确保所有需求都被实现方面的表现如何？ — 测试集是否能证明所有获批需求都已实际构建完成（
    T
    + 工程门禁
    G
    ）？
让“全部实现”具备明确定义的关联逻辑：测试套件需对**完整获批需求集 = 
PRD验收标准 ∪ 有效的E类新增需求
**负责。框架提取出的有效需求若未被测试套件覆盖，属于测试套件的遗漏（
T
），而非框架的遗漏（
E
仍保留提取得分）。需求提取定义了验证的目标范围。
评分基于检查清单：每条标准都被拆解为原子化的二元（达标/未达标）检查项，每项都有
file:line
级别的证据支撑。二元拆解是确保评分可复现的核心设计——分级/李克特量表（“这该打3分还是4分？”）是评估者之间产生分歧的主要原因；二元检查将评估者间的一致性提升到了接近人类评估的水平。部分得分由达标检查项的占比推导而来，绝不通过滑动量表主观判定。

When to use

适用场景

  • "Evaluate/score this PRD case by case and give a final grade"
  • "Was this story/feature implemented 100%?"
  • Benchmarking multiple spec-driven implementations of the same PRD
  • Auditing implementation and test coverage (unit + e2e) against acceptance criteria
  • “逐案评估/打分这个PRD，并给出最终等级”
  • “这个故事/功能是否100%实现了？”
  • 对同一PRD的多个规格驱动实现进行基准测试
  • 对照验收标准审计实现和测试覆盖率（单元测试 + e2e测试）

Inputs required

所需输入

  1. The PRD (ground-truth product intent) — the user story acceptance criteria are the unit of evaluation.
  2. The implementation (production code).
  3. The tests (unit + e2e).
  4. The SDD-derived artifacts — 
    spec.md
    and 
    tasks.md
    (refined ACs, requirement IDs, derived requirements). Required for the Elicitation 
    E
    and Scope 
    S
    axes (they grade these against the PRD). When absent, 
    Final
    /
    T
    still run, but report 
    E
    /
    S
    as 
    n/a — no derived spec
    . The PRD remains the source of truth for what counts as "expected"; the derived spec is what gets graded for respect and extraction.
Scoping the diff. Before scoring, use git to identify which files changed for this implementation. That diff surface is the primary search scope for all 
file:line
evidence in steps 4 and 8:
bash
git diff <base>..<head> --name-only   # branch or PR
git diff --name-only HEAD             # uncommitted changes
git status --short                    # include untracked new files
Record the diff surface in the report. Evidence outside it is still valid (e.g. a pre-existing file was modified), but note when a check relies on files not in the diff — that may indicate the wrong base was chosen.
  1. PRD（真实产品意图）——用户故事的验收标准是评估的基本单位。
  2. 实现代码（生产环境代码）。
  3. 测试代码（单元测试 + e2e测试）。
  4. SDD衍生产物 —— 
    spec.md
    和
    tasks.md
    （细化后的验收标准、需求ID、衍生需求）。需求启发
    E
    和范围
    S
    维度需要这些产物（它们是对照PRD进行评分的对象）。若缺少这些产物，
    Final
    /
    T
    仍可运行，但
    E
    /
    S
    会报告为
    n/a — 无衍生规格
    。PRD仍然是“预期结果”的唯一事实来源；衍生规格只是用于评估遵守性和提取能力的对象。
界定变更范围。 评分前，使用git识别本次实现变更的文件。这个变更面是步骤4和8中所有
file:line
证据的主要搜索范围：
bash
git diff <base>..<head> --name-only   # branch or PR
git diff --name-only HEAD             # uncommitted changes
git status --short                    # include untracked new files
在报告中记录变更面。变更面之外的证据仍然有效（例如修改了已有的文件），但如果某个检查项依赖的文件不在变更面内，需要注明——这可能意味着选择了错误的基准版本。

Quick start

快速开始

New to running this end-to-end? See quickstart.md for the 4-chat-session flow (freeze baseline → plan → implement → evaluate) with paste-ready prompts. Read it when you need the operational how-to; the rest of this file is the scoring methodology.
第一次运行这套端到端流程？请查看quickstart.md，了解包含4个会话阶段的流程（冻结基线 → 规划 → 实现 → 评估），附带可直接粘贴使用的提示词。需要操作指南时阅读该文件；本文件其余部分为评分方法论。

Core rules (read first)

核心规则（请先阅读）

These five rules govern every score. They exist to make the grade auditable and reproducible.
  1. Evidence or zero. Every MET check MUST cite evidence as 
    file:line
    (or 
    file:startLine-endLine
    ). No located evidence ⇒ the check is UNMET. Never award credit from assumptions or from the PRD restating intent.
  2. Search-before-zero (anti false-negative). Before marking a check UNMET for "not found", record the search actually performed — starting with the diff-surface files from the scoping step, then the grep/glob terms tried and any additional files/dirs inspected. A check scored UNMET must confirm the behavior is absent from the diff surface, not just from an ad-hoc grep across the full repo. UNMET means searched and genuinely absent, not did not look. If no search is shown, the check is not yet scored, not UNMET.
  3. Read the path end-to-end — including the data shape. A check is MET only if you traced the real code/test path, not because a symbol name matches. For an emitted/returned/persisted artifact, inspect the constructed payload object itself, not just the call site — a present 
    emit(...)
    /
    return ...
    does not prove the named field is in the payload (see the Conjunction rule). A test that exercises a behavior but does not assert it does not meet a verification check.
  4. Judge ≠ author for benchmarks. When grading to compare implementations, the evaluating model should differ from the model that wrote the code; LLM judges over-reward their own output (self-preference bias). If they are the same, flag it under Assumptions and treat borderline checks as UNMET.
  5. The evaluator is read-only over the subject. Never modify the code under evaluation — not to fix a failing gate, not to "help", not even for trivial type errors. A red gate stays red: record 
    ✗
    , apply the 
    Adjusted Final
    , and put the required correction in the report's fix list. Modifying the subject mid-evaluation contaminates the benchmark (the grade no longer measures the framework's output) and invalidates the diff surface. If a correction was accidentally applied, revert it and score the original state.
这五条规则适用于所有评分，目的是确保评分可审计、可复现。
  1. 无证据即零分。 每个达标的检查项必须引用
    file:line
    （或
    file:startLine-endLine
    ）格式的证据。未找到证据 ⇒ 检查项未达标。绝不能基于假设或PRD中重述的意图给分。
  2. 零分前先搜索（防止假阴性）。 在因“未找到”将检查项标记为未达标之前，需记录实际执行的搜索操作——从范围界定步骤得到的变更面文件开始，然后记录尝试的grep/glob关键词，以及检查过的其他文件/目录。被评为未达标的检查项必须确认该行为在变更面中确实不存在，而不仅仅是在整个仓库中随便grep了一下没找到。未达标意味着搜索过且确实不存在，而不是没去找。如果没有展示搜索过程，该检查项的状态是尚未评分，而非未达标。
  3. 端到端通读路径——包括数据结构。 只有当你追踪了真实的代码/测试路径后，才能判定检查项达标，不能仅因为符号名称匹配就判定达标。对于被发出/返回/持久化的产物，要检查实际构建的载荷对象本身，而不仅仅是调用位置——存在
    emit(...)
    /
    return ...
    并不能证明载荷中包含指定字段（见合取规则）。仅执行了某个行为但未断言结果的测试不满足验证检查项的要求。
  4. 基准测试中评判者≠作者。 当评分是为了比较不同实现时，评估所用的模型应与编写代码的模型不同；LLM评判者会高估自己生成的输出（自偏好偏差）。如果二者是同一模型，需在假设部分标记此情况，并将边界检查项视为未达标。
  5. 评估者对评估对象仅有只读权限。 绝不修改被评估的代码——哪怕是为了修复失败的门禁、为了“帮忙”，甚至是为了微不足道的类型错误。红门禁就保持红色：记录
    ✗
    ，应用
    调整后最终得分
    ，并将所需的修正项列入报告的修复清单。在评估过程中修改评估对象会污染基准测试结果（评分不再能衡量框架的输出质量），并使变更面失效。如果不小心应用了修正，需回退修改后再对原始状态进行评分。

Reproducibility rules

可复现性规则

The grade must be stable: the same PRD + same commit must yield the same 
Final
 across runs and evaluators. Obey all of:
  1. Freeze the AC list AND its checklist (the single biggest drift source). AC enumeration and check granularity are decided once and pinned, because both 
    I
    , 
    T
    , and the 
    Story_score
    denominator depend on them:
    • If 
      spec.md
      with stable requirement IDs exists, the AC list and IDs are frozen — use them verbatim; do not re-split or merge.
    • Derive the binary checklist for each AC once and persist it to 
      <spec-folder>/evaluations/_ac-baseline.md
      (IDs + verbatim AC text + the I-checks and T-checks). Every implementation of that PRD is scored against the identical checklist.
    • One AC = one testable assertion of intent; one check = one atomic, observable yes/no proposition. Do not collapse distinct behaviors into one check, nor split one behavior across several.
  2. Priority comes from the PRD, never inferred silently. Use the PRD's explicit P0/P1/P2 labels. If a story is unlabeled, mark its priority 
    ASSUMED
    in the report and list it under Assumptions; never let an inferred priority silently move the 
    3/2/0
    weights.
  3. Round once, at the end. Carry full precision through the arithmetic; round only the reported 
    AC_score
    , 
    Story_score
    , and 
    Final
    to 2 decimals. Band assignment uses the rounded 
    Final
    .
  4. Compute, don't calculate. The roll-up (
    Σw
    , every 
    Story_score
    , 
    Final
    , 
    Adjusted Final
    ) MUST be computed by executing a script (e.g. 
    node -e
    / 
    python3 -c
    ) that takes the per-AC 
    I
    /
    T
    fractions and the priority-weight table as input, with the script's output pasted into the report. Never do the arithmetic mentally or by hand — a hand-summed denominator (
    Σw
    ) is a known failure mode that silently shifts the grade band. 
    Σw
    must be derived inside the script from the same priority table shown in the report, not typed in as a literal.
  5. Self-consistency ensemble (k = 3). Evaluate the checklist three times independently at low/zero temperature and take the majority MET/UNMET per check before computing any number. Majority voting over k=3 removes most stochastic flips at low cost. If the three passes disagree on a check, that check is borderline — keep the majority verdict and note it; persistent disagreement means the check wording is ambiguous (sharpen it in the baseline, see Calibration).
评分必须稳定：同一PRD + 同一提交在不同运行和不同评估者之间必须得到相同的
Final
得分。请遵守以下所有规则：
  1. 冻结验收标准列表及其检查清单（最大的偏差来源）。 验收标准的枚举和检查粒度一次性确定并固定，因为
    I
    、
    T
    和
    故事得分
    的分母都依赖于此：
    • 如果存在带有稳定需求ID的
      spec.md
      ，则验收标准列表和ID均已冻结——直接原样使用，不得重新拆分或合并。
    • 为每条验收标准生成二元检查清单一次，并保存到
      <spec-folder>/evaluations/_ac-baseline.md
      （包含ID、验收标准原文、I类检查项和T类检查项）。该PRD的所有实现都对照同一份检查清单评分。
    • 一条验收标准 = 一个可测试的意图断言；一个检查项 = 一个原子化的、可观察的是/否命题。不要将不同的行为合并为一个检查项，也不要将单个行为拆分为多个检查项。
  2. 优先级来自PRD，绝不静默推断。 使用PRD中明确标注的P0/P1/P2标签。如果某个故事没有标注优先级，在报告中将其优先级标记为
    ASSUMED
    ，并列在假设部分；绝不能让推断的优先级静默地改变
    3/2/0
    的权重。
  3. 仅在末尾四舍五入一次。 计算过程中保留全部精度；仅将报告的
    验收标准得分
    、
    故事得分
    和
    最终得分
    四舍五入到2位小数。等级划分使用四舍五入后的
    最终得分
    。
  4. 用脚本计算，不要手动计算。 汇总得分（
    Σw
    、每个
    故事得分
    、
    最终得分
    、
    调整后最终得分
    ）必须通过执行脚本（例如
    node -e
    / 
    python3 -c
    ）来计算，脚本输入为每条验收标准的
    I
    /
    T
    占比和优先级权重表，脚本输出需粘贴到报告中。绝不能心算或手动计算——手动求和的分母（
    Σw
    ）是已知的故障模式，会静默地改变评分等级。
    Σw
    必须由脚本根据报告中展示的同一份优先级表推导得出，而不能作为字面量直接输入。
  5. 自一致性集成（k=3）。 在低/零温度下独立评估检查清单三次，在计算任何数值之前，对每个检查项取多数票的达标/未达标结果。k=3的多数投票能以较低成本消除大部分随机波动。如果三次运行对某个检查项的判定不一致，说明该检查项处于边界——保留多数票的判定结果并注明；持续不一致意味着检查项的表述存在歧义（需在基线中优化表述，见校准部分）。

Scoring model

评分模型

Unit of scoring: the acceptance criterion (AC), decomposed into binary checks

评分单位：拆解为二元检查项的验收标准（AC）

For each in-scope AC, the frozen checklist holds two sets of atomic checks, each scored MET (1) / UNMET (0):
Implementation checks (
I
-checks) — one per distinct observable behavior the AC asserts. A check is MET only if production code performs that behavior, evidenced 
file:line
. Include only behaviorally-observable clauses (a verb the AC states: creates / returns / rejects / persists / grants / defaults-to). Non-functional polish (wording, logging) is not a check — note it separately so it never moves the score.
When parsing an AC into checks (done once, at baseline-freeze time), apply both rules below so verbs aren't the only thing captured:
Conjunction / payload-field rule. When an AC enumerates multiple items — joined by "and", ",", "with [field]", "including" — especially in the subject or payload of an emitted/returned/persisted artifact, treat each named field or entity as its own I-check. Do not stop at confirming the parent method is called: for each named field, open the actual data object constructed at 
file:line
and verify the field is present in it (score against the payload shape, not the call site). Example: "emit a trigger associated with the user and the trial end date" → two I-checks: (1) trigger payload carries 
userId
, (2) trigger payload carries 
trialEndsAt
. Confirming 
emit(...)
is reached does not satisfy check (2) if the payload object omits the field.
Disjunction / product-chosen rule. When an AC presents alternatives ("A or B", "pause or cancel", "immediately or at period end"), decide the reading once and freeze it in the baseline (the reading itself must not vary per run):
  • Independent paths — both are separately reachable behaviors (e.g. "cancel immediately or at period end"): one I-check per path, each independently MET/UNMET.
  • Product-controlled / configurable — the AC frames the choice as product-driven (e.g. "apply the product-chosen behavior: pause or cancel"): two I-checks — (1) the recommended/default behavior is implemented, and (2) the alternative is reachable without a code change (config key / flag / env). Hard-coding one option with no switch to the other ⇒ check (2) UNMET. A different feature that happens to reach the other option (e.g. user-initiated cancel) does not satisfy (2) — it must be the same product-controlled decision point.
  • If "product-chosen" is itself ambiguous between runtime-configurable and design-time chosen, resolve it against the PRD/spec wording and record the resolved reading in the baseline; only require reachability of the non-default path when the AC frames the choice as runtime/product-controlled.
Wiring / ingress rule (asynchronously-delivered side-effects). When an observable side-effect is delivered asynchronously through an inbound event (e.g. a Stripe webhook → handler → persisted status), add a dedicated wiring I-check in addition to the behavior I-check: the entry endpoint actually receives, verifies, and dispatches the inbound event — by type — to the handler. This is distinct from the 
T-outcome
test that the handler produces the right state: a fully-tested handler sitting behind a dead, unregistered, or mis-routed endpoint still fails the wiring check. Without it, a correct-but-unreachable handler would earn full credit (the blind spot the 
T-outcome
entry-point-neutrality opens up — letting the test bypass HTTP means something else must prove the route exists). Score the wiring I-check against the controller/route that maps the event type to the handler (signature/authenticity verification + dispatch), not against the handler internals. Freeze a wiring I-check on every AC whose observable effect arrives via an inbound async event.
I = (# I-checks MET) / (# I-checks)
Test checks (
T
-checks) — verification checks at the required level(s). Required levels are fixed by policy, not judged per-AC:
  • pure / business logic ⇒ unit required
  • observable HTTP / contract / persistence side-effect ⇒ e2e / integration required
  • an AC with both ⇒ both required
T-outcome
— persistence / async side-effects (outcome-based, entry-point-neutral). When the AC asserts a resulting state — phrased "results in / persisted / displayed status / reflects the change / updates the status" — the verification check is on the real resulting state, not on a call. Label these checks 
T-outcome
 (distinct from a plain synchronous-response 
T-e2e
). The policy has four parts:
  • Assert the real outcome. MET requires asserting the actual persisted state (a real DB row) or the actually-returned payload, against real infra. A test that only exercises the path without asserting the resulting state is UNMET (Core rule 3).
  • Entry point is the implementation's choice (entry-point-neutral). Drive the real component via any entry point and assert the resulting state: the HTTP endpoint, OR by invoking the handler / queue-consumer / service method directly against the real DB. An async (ack-fast-then-process) design need not be tested through the full HTTP→queue→worker path — awaiting the handler and asserting the persisted row is sufficient and equivalent. Do not penalize a queue/async architecture for not being driven end-to-end through HTTP; the question is whether the real resulting state is asserted, not which door the test entered.
  • Mock-only exclusion (scoped precisely). Asserting that a method was called on a mocked repository / state-machine (
    expect(save).toHaveBeenCalled()
    / 
    expect(transition).toHaveBeenCalledWith(...)
    ) proves a call, not a persisted outcome ⇒ UNMET for any "results in / persisted / displayed status" check (it may still count toward Robustness 
    R
    ). The exclusion is scoped to outcome checks only — it must not over-fire: a check whose proposition is "invokes the external API" (e.g. "immediate cancel calls 
    StripeClient.cancel
    ", "creates the subscription in Stripe") is correctly verified by a spy/mock, because the asserted proposition is the call. Mock-only is fine there; the exclusion does not apply.
  • Minimum-level floor. The required level is a floor, not an exact match. A stronger-level test (an e2e/integration test covering a unit-required proposition) satisfies that proposition, provided the specific proposition (e.g. a named payload field, a default value) is actually asserted on the real artifact. Do not mark a unit-required check UNMET solely because the only assertion of it lives in an e2e/integration test.
For each required level, derive binary checks such as: primary behavior asserted at this level (MET only if asserted, not merely exercised), relevant negative/error case asserted, each secondary clause asserted. Mocked external systems (e.g. Stripe via nock) cannot prove the external side — write the check against the observable local contract and note the limitation.
The conjunction rule applies to T-checks too: each named payload field gets its own verification check. A test that asserts the artifact was emitted/returned but does not assert the field's presence/value is UNMET for that field's check (asserting 
emit
was called ≠ asserting the payload carries 
trialEndsAt
).
T = (# T-checks MET across all required levels) / (# T-checks)
Both 
I
and 
T
land on a fine-grained value, but every underlying atom is a binary, evidence-backed yes/no — that is what makes them reproducible. There is no "minor vs meaningful" judgment: a clause either has its own check or it does not, and that decision was frozen in the baseline.
对于每个范围内的验收标准，冻结的检查清单包含两组原子检查项，每项评分为达标（1）/ 未达标（0）：
实现检查项（I类检查项） —— 验收标准声明的每个可观察的独立行为对应一个检查项。只有当生产代码实现了该行为时，检查项才达标，需有
file:line
证据支撑。仅包含行为可观察的条款（验收标准中陈述的动词：创建 / 返回 / 拒绝 / 持久化 / 授予 / 默认值为）。非功能性的优化（措辞、日志）不作为检查项——单独记录即可，绝不影响评分。
将验收标准拆解为检查项时（在基线冻结时一次性完成），需应用以下两条规则，确保不只是捕获到动词：
合取/载荷字段规则。 当验收标准枚举了多个条目——通过*“和”、“,”、“带有[字段]”、“包括”*连接——尤其是在被发出/返回/持久化的产物的主题或载荷中时，将每个命名的字段或实体视为独立的I类检查项。不能仅确认父方法被调用就停止：对于每个命名字段，要打开
file:line
处实际构建的数据对象，验证该字段存在于其中（对照载荷结构评分，而非对照调用位置）。 示例：“发出与用户和试用结束日期关联的触发器” → 两个I类检查项：(1) 触发器载荷包含
userId
，(2) 触发器载荷包含
trialEndsAt
。如果载荷对象遗漏了该字段，仅确认执行到了
emit(...)
不能满足检查项(2)。
析取/产品选择规则。 当验收标准给出了多个选项（“A或B”、“暂停或取消”、“立即或在周期结束时”），需一次性确定解读方式并冻结在基线中（解读方式本身不能因运行而变化）：
  • 独立路径 —— 两者都是可分别到达的行为（例如“立即取消或在周期结束时取消”）：每条路径对应一个I类检查项，各自独立判定达标/未达标。
  • 产品控制/可配置 —— 验收标准将选择描述为产品驱动的（例如“应用产品选择的行为：暂停或取消”）：两个I类检查项——(1) 推荐/默认行为已实现，(2) 替代方案无需修改代码即可到达（通过配置键 / 开关 / 环境变量）。硬编码一个选项且没有切换到另一个的方式 ⇒ 检查项(2)未达标。另一个恰好能到达另一个选项的功能（例如用户主动取消）不能满足检查项(2)——必须是同一个产品控制的决策点。
  • 如果*“产品选择的”本身在运行时可配置和设计时选定*之间存在歧义，需对照PRD/规格的措辞解决，并将解决后的解读记录在基线中；只有当验收标准将选择描述为运行时/产品控制的时，才要求非默认路径可到达。
接线/入口规则（异步交付的副作用）。 当可观察的副作用通过入站事件异步交付时（例如Stripe webhook → 处理器 → 持久化状态），除了行为I类检查项之外，还需添加一个专门的接线I类检查项：入口端点确实按类型接收、验证入站事件，并将其分派给处理器。这与验证处理器产生正确状态的
T-outcome
测试不同：一个经过充分测试的处理器，如果位于失效、未注册或路由错误的端点之后，仍然无法通过接线检查。没有这项检查，一个正确但无法到达的处理器可能会得到满分（这是
T-outcome
入口点中立性带来的盲区——允许测试绕过HTTP意味着必须有其他东西证明路由存在）。接线I类检查项的评分依据是将事件类型映射到处理器的控制器/路由（签名/真实性验证 + 分派），而非处理器内部逻辑。对于所有可观察效果通过入站异步事件到达的验收标准，都要冻结一个接线I类检查项。
I = (# I-checks MET) / (# I-checks)
测试检查项（T类检查项） —— 在所需级别上的验证检查项。所需级别由策略固定，而非按每条验收标准判定：
  • 纯逻辑 / 业务逻辑 ⇒ 需要单元测试
  • 可观察的HTTP / 契约 / 持久化副作用 ⇒ 需要e2e / 集成测试
  • 同时包含两者的验收标准 ⇒ 两者都需要
T-outcome
— 持久化 / 异步副作用（基于结果，入口点中立）。 当验收标准声明了结果状态——措辞为“导致 / 持久化 / 显示状态 / 反映变更 / 更新状态”——时，验证检查针对的是真实的结果状态，而非调用。将这些检查项标记为**
T-outcome
**（区别于普通的同步响应
T-e2e
）。该策略包含四个部分：
  • 断言真实结果。 达标要求对照真实基础设施，断言实际持久化的状态（真实的数据库行）或实际返回的载荷。仅执行了路径但未断言结果状态的测试为未达标（核心规则3）。
  • 入口点由实现决定（入口点中立）。 通过任意入口点驱动真实组件并断言结果状态：可以是HTTP端点，也可以是直接调用处理器 / 队列消费者 / 服务方法并对照真实数据库。异步（快速确认后处理）设计无需通过完整的HTTP→队列→工作器路径进行测试——等待处理器执行完成并断言持久化的行就足够且等效。不要因为队列/异步架构没有通过HTTP端到端驱动而扣分；关键在于是否断言了真实的结果状态，而非测试从哪个入口进入。
  • 仅Mock排除（范围精确）。 断言在Mock的仓库 / 状态机上调用了某个方法（
    expect(save).toHaveBeenCalled()
    / 
    expect(transition).toHaveBeenCalledWith(...)
    ）只能证明发生了调用，不能证明持久化了结果 ⇒ 对于任何“导致 / 持久化 / 显示状态”的检查项，均为未达标（仍可计入健壮性
    R
    ）。该排除仅适用于结果检查项——绝不能扩大范围： 如果检查项的命题是*“调用外部API”（例如“立即取消调用
    StripeClient.cancel
    ”、“在Stripe中创建订阅”），则使用spy/mock验证是    正确的*，因为断言的命题就是调用本身。这种情况下仅用Mock没问题，排除规则不适用。
  • 最低级别下限。 所需级别是下限，而非精确匹配。 更高级别的测试（覆盖了单元测试要求命题的e2e/集成测试）满足该命题的要求，前提是该特定命题（例如命名字段、默认值）确实在真实产物上被断言了。不要仅仅因为某个单元测试要求的检查项的唯一断言存在于e2e/集成测试中，就将其标记为未达标。
对于每个所需级别，推导二元检查项，例如：该级别下断言了主要行为（仅当断言了才达标，而非仅执行了）、断言了相关的负面/错误场景、断言了每个次要条款。Mock的外部系统（例如通过nock模拟Stripe）无法证明外部侧的行为——需对照可观察的本地契约编写检查项，并注明该限制。
合取规则同样适用于T类检查项：每个命名字段都有自己的验证检查项。如果测试断言了产物已发出/返回，但未断言该字段的存在/值，则该字段的检查项为未达标（断言调用了
emit
≠ 断言载荷包含
trialEndsAt
）。
T = (# T-checks MET across all required levels) / (# T-checks)
I
和
T
最终都会得到细粒度的数值，但每个底层原子都是二元的、有证据支撑的是/否判定——这正是它们可复现的原因。不存在“轻微 vs 重大”的主观判断：条款要么有自己的检查项，要么没有，而这个决定已经冻结在基线中了。

Per-AC, per-story, final

单验收标准、单故事、最终得分

AC_score    = 0.6 * I + 0.4 * T
Story_score = mean(AC_score over the story's ACs)      # ACs equally weighted by default
Final       = Σ(w_s * Story_score) / Σ(w_s)            # weighted by story priority
Story priority weights 
w_s
:
PriorityWeightRationale
P03Core MVP
P12MVP-adjacent / launch
P2 / out-of-scope0Excluded from the grade; report separately as roadmap readiness
The 0.6/0.4 implementation/test split reflects "working software first, proven by tests". Keep these weights — and the P0/P1/P2 weights — fixed across the benchmark; changing them breaks comparability.
AC_score    = 0.6 * I + 0.4 * T
Story_score = mean(AC_score over the story's ACs)      # ACs equally weighted by default
Final       = Σ(w_s * Story_score) / Σ(w_s)            # weighted by story priority
故事优先级权重
w_s
：
优先级权重理由
P03核心MVP
P12MVP相邻 / 上线相关
P2 / 范围外0不计入评分；作为路线图准备度单独报告
实现/测试按0.6/0.4拆分，反映了“可工作的软件优先，由测试验证”的理念。在整个基准测试过程中，这些权重以及P0/P1/P2权重都要保持固定；修改它们会破坏可比性。

Grade bands

评分等级

FinalBand
≥ 0.90Spec-complete
0.75–0.89Strong (minor gaps)
0.60–0.74Partial (meaningful gaps)
0.40–0.59Weak
< 0.40Inadequate
最终得分等级
≥ 0.90规格完整
0.75–0.89良好（有轻微差距）
0.60–0.74部分实现（有明显差距）
0.40–0.59较弱
< 0.40不足

Bias controls

偏差控制

LLM judges have well-documented biases; hold these explicitly:
  • Verbosity / quantity bias — more code or more tests never raises 
    I
    or 
    T
    . Only checks that map to an AC clause count. Surplus tests go to 
    R
    /
    D
    (below), never into the grade.
  • Self-preference bias — see Core rule 4 (judge ≠ author).
  • Anchoring — score against the frozen baseline and the calibration anchors in 
    references/reference.md
    , not against the previous implementation you happened to grade.
LLM评判者存在已被充分证实的偏差；需明确控制以下偏差：
  • 冗长/数量偏差 —— 更多的代码或更多的测试绝不会提高
    I
    或
    T
    的得分。只有映射到验收标准条款的检查项才会计入。多余的测试计入
    R
    /
    D
    （见下文），绝不计入评分。
  • 自偏好偏差 —— 见核心规则4（评判者≠作者）。
  • 锚定效应 —— 对照冻结的基线和
    references/reference.md
    中的校准锚点评分，而非对照你刚评完的上一个实现。

Reported beside the grade (NOT folded into Final — keep comparable)

评分旁的报告项（不计入最终得分——保持可比性）

  • Robustness Index 
    R
     — extra tests beyond PRD cases, weighted High=1.0 / Med=0.5 / Low=0.25, summed. Signals defensive quality. Never inflates 
    Final
    .
  • Scope Adherence 
    S
     — 
    pass
    / 
    partial
    / 
    fail
    , governed by the traceability principle: every built behavior must trace to a sanctioned source (a PRD acceptance criterion → 
    Final
    , or a valid 
    E
    -addition). Three checks, each per built behavior:
    • PRD-boundary — built something on the PRD's explicit out-of-scope list ⇒ 
      fail
      .
    • Rogue build — built something that traces to neither a PRD AC nor a valid 
      E
      -addition (untraceable / invented) ⇒ 
      fail
      .
    • Plan drift — 
      spec.md
      /
      tasks.md
      sanctioned a behavior the code didn't build (or left half-built / inconsistent) ⇒ 
      partial
      . Everything traces cleanly ⇒ 
      pass
      . Report the failing/partial behaviors with 
      file:line
      . A valid requirement the framework derived into the spec but correctly did NOT build (because it is out of scope) is good discipline — it is not an 
      S
      penalty;       record it as deferred-valid under 
      E
      .
  • Engineering Gates 
    G
     — 
    build
    , 
    lint
    , 
    unit
    , 
    e2e
    . Each gate must be an actually-executed 
    ✓
    /
    ✗
    or an explicit 
    not-run
    (with reason, e.g. e2e infra unavailable).     Never report a gate as passing without running it. The 
    build
    gate is pinned mechanically to remove evaluator discretion:     pick the project's canonical build command once per benchmark (the build script the repo documents) and define the gate as that command exiting 
    0
    — nothing else decides it. Record the exact command in the report so every run uses the identical gate. A pre-existing, unrelated toolchain failure in code the change didn't touch (e.g. a known typecheck error in an untouched migration path) is a documented non-graded NOTE, not a 
    ✗
    : it does not gate 
    build
    and does not trigger 
    Adjusted Final
    . Record it in the report as a known note, but the 
    build
    verdict follows only the pinned build command's exit code. Probe-before-not-run: a gate may only be reported 
    not-run
    after a recorded real attempt — the command executed (or the prerequisite check, e.g. 
    docker ps
    for e2e infra) and its error output pasted as evidence. A reason without an attempt is not valid; the gate stays not yet scored, exactly like an UNMET without a search (Core rule 2 applied to gates). Only a confirmed-red gate (
    ✗
    ) triggers 
    Adjusted Final = Final × 0.5
    ; a 
    not-run
    gate cannot grant or deduct credit — it is reported as a known blind spot. The unadjusted 
    Final
    stays reported for comparison. A red gate is never fixed by the evaluator (Core rule 5) — record 
    ✗
    , adjust, and list the fix.
  • Test Distribution 
    D
     — every feature test classified into one of three tiers, reported as counts and % of the suite. Shows where the testing effort went; never inflates 
    Final
    . See below.
  • 健壮性指数
    R
     —— PRD场景之外的额外测试，按高=1.0 / 中=0.5 / 低=0.25加权后求和。反映防御性质量。绝不会抬高
    最终得分
    。
  • 范围遵守度
    S
     —— 
    通过
    / 
    部分通过
    / 
    未通过
    ，遵循可追溯性原则：每个已构建的行为都必须能追溯到获批来源（PRD验收标准 → 计入
    最终得分
    ，或者有效的E类新增需求）。针对每个已构建的行为，进行三项检查：
    • PRD边界 —— 构建了PRD明确列入范围外清单的内容 ⇒ 
      未通过
      。
    • ** Rogue构建** —— 构建的行为既无法追溯到PRD验收标准，也无法追溯到有效的E类新增需求（不可追溯 / 凭空发明） ⇒ 
      未通过
      。
    • 计划偏移 —— 
      spec.md
      /
      tasks.md
      批准了某个行为，但代码没有构建（或者只构建了一半 / 不一致） ⇒ 
      部分通过
      。 所有内容都能清晰追溯 ⇒ 
      通过
      。用
      file:line
      记录未通过/部分通过的行为。框架在规格中推导出来但正确地未构建（因为超出范围）的有效需求是良好的规范——这不是
      S
      的扣分项；       将其作为“有效但延后”的需求记录在
      E
      下。
  • 工程门禁
    G
     —— 
    构建
    、
    lint
    、
    单元测试
    、
    e2e测试
    。每个门禁必须是实际执行后的
    ✓
    /
    ✗
    ，或者明确的
    未运行
    （附带原因，例如e2e基础设施不可用）。     绝不能在未运行的情况下报告门禁通过。
    构建
    门禁是机械固定的，以消除评估者的自由裁量权：     每个基准测试确定一次项目的规范构建命令（仓库文档中记录的构建脚本），门禁的定义就是该命令退出码为
    0
    ——其他任何因素都不决定结果。在报告中记录确切的命令，确保每次运行都使用完全相同的门禁。变更未触及的代码中存在的、与本次变更无关的预先存在的工具链故障（例如未修改的迁移路径中已知的类型检查错误）是已记录的非评分备注，而非
    ✗
    ：它不阻塞
    构建
    ，也不会触发
    调整后最终得分
    。将其作为已知备注记录在报告中，但
    构建
    的判定仅遵循固定构建命令的退出码。未运行前先探测： 只有在记录了真实尝试之后，才能将门禁报告为
    未运行
    ——命令已执行（或已检查前置条件，例如检查e2e基础设施的
    docker ps
    ），并将错误输出粘贴为证据。没有尝试的理由是无效的；门禁状态保持尚未评分，就像没有搜索过程的未达标一样（适用于门禁的核心规则2）。只有确认红的门禁（
    ✗
    ）才会触发
    调整后最终得分 = 最终得分 × 0.5
    ；
    未运行
    的门禁不能加分或扣分——它作为已知盲区被报告。未调整的
    最终得分
    仍会保留用于比较。红门禁绝不由评估者修复（核心规则5）——记录
    ✗
    ，调整得分，并列出修复方案。
  • 测试分布
    D
     —— 每个功能测试都归入三个层级之一，以计数和占测试集的百分比形式报告。展示测试投入的方向；绝不会抬高
    最终得分
    。见下文。

Test distribution by tier 
D
(reported beside)

按层级划分的测试分布
D
（旁报告项）

Classify every added feature test (each 
it
/test case, counting 
it.each
rows individually) into exactly one tier. The classification reuses work already done in the test-check and robustness steps, so it is cheap and reproducible:
TierDefinitionMaps to
Necessary (primary happy path)Asserts the primary success path of a P0/MVP AC, or a dependency required for that path to work.A T-check MET on a P0 main flow
Secondary (important)Maps to an AC but is not the P0 primary path: P1 ACs, edge/negative/error paths, secondary clauses, security/authz, duplicate/idempotency/dedup, status-trigger emission.A T-check MET that is not Necessary
Nice-to-haveMaps to no AC: defensive/exhaustive coverage (enum mapping, unknown-status, repo/infra internals, helper methods) or unit coverage redundant with an e2e-proven path.The Robustness 
R
extras
Rule of thumb: AC-mapped tests split into Necessary + Secondary; the Robustness 
R
inventory is the Nice-to-have tier. A test belongs to exactly one tier — do not double-count.
Report 
D
as a table of 
tier → count → %
over total feature tests, plus a one-line shape read. Healthy suites cover every P0 primary path with at least one Necessary test; a suite thin on Necessary but heavy on Nice-to-have is a smell (robustness without proven core), and the opposite (no defensive tests) is fragile. 
D
is descriptive — it never changes 
Final
.
将每个新增的功能测试（每个
it
/测试用例，
it.each
的每一行单独计数）归入恰好一个层级。分类复用了测试检查和健壮性步骤中已完成的工作，因此成本低且可复现：
层级定义对应项
必要（主要 happy path）断言P0/MVP验收标准的主要成功路径，或者该路径正常工作所需的依赖项。P0主流程上达标的T类检查项
次要（重要）对应某个验收标准，但不是P0主要路径：P1验收标准、边界/负面/错误路径、次要条款、安全/授权、重复/幂等/去重、状态触发器发出。非必要的达标的T类检查项
锦上添花不对应任何验收标准：防御性/穷尽式覆盖（枚举映射、未知状态、仓库/基础设施内部、辅助方法），或者与e2e验证路径冗余的单元测试覆盖。健壮性
R
的额外测试
经验法则： 映射到验收标准的测试分为必要+次要；健壮性
R
的清单就是锦上添花层级。每个测试只属于一个层级——不要重复计数。
将
D
报告为一个
tier → count → %
的表格，基于总功能测试数，再加上一行形态解读。健康的测试集应该为每个P0主要路径至少覆盖一个必要测试；必要测试很少但锦上添花测试很多的测试集是一种坏味道（有健壮性但核心未经验证），反之（没有防御性测试）则很脆弱。
D
是描述性的——它绝不会改变
最终得分
。

Elicitation 
E
— how well the framework extracts requirements (reported beside; never folded into Final)

需求启发
E
——框架提取需求的能力如何（旁报告项；绝不计入最终得分）

E
grades the SDD-derived 
spec.md
/
tasks.md
against the PRD: did the framework surface the implicit requirements a senior engineer would insist on, and did it do so cleanly (no hallucinated or contradictory additions)? This is the "value beyond transcription" of an SDD framework. It evaluates the spec artifacts, not the code. Like 
R
/
S
/
D
, it is reported beside 
Final
so the fidelity grade stays comparable.
Three binary-checklist sub-metrics:
E
对照PRD对SDD衍生的
spec.md
/
tasks.md
进行评分：框架是否挖掘出了资深工程师会坚持要求的隐含需求，并且做到了干净无噪（没有幻觉或矛盾的新增内容）？这是SDD框架“超越转录的价值”。它评估的是规格产物，而非代码。与
R
/
S
/
D
一样，它在
最终得分
旁报告，以保证保真度评分的可比性。
包含三个基于二元检查清单的子指标：

E_recall
— did it find what it should have?

E_recall
（召回率）——它找到了应该找到的内容吗？

Run the frozen implicit-requirement category rubric below. For each category, mark Addressed (cite 
spec.md:line
), Missed, or N/A (the PRD/domain makes it irrelevant). The rubric is PRD-agnostic and reusable, so recall is reproducible without per-PRD gold labeling.
E_recall = Addressed / (Addressed + Missed)        # N/A excluded
#CategoryA requirement is expected if the feature…
1Input validation & boundsaccepts user/external input (ranges, formats, required fields)
2Error taxonomy & messagingcan fail in ways the caller must distinguish (typed errors, codes)
3AuthN / AuthZexposes an endpoint or touches user-owned data
4Idempotency & dedupcan be retried or replayed (payments, create-once resources)
5Concurrency & race conditionshas interleaving writes / check-then-act on shared state
6Data lifecycle & consistencypersists state (transactions, partial-failure rollback, retention)
7Observabilityruns in production (logs, metrics, traceability of failures)
8Limits, pagination & ratereturns lists or is callable at volume
9External-dependency failurecalls a third party (timeouts, retries, circuit-breaking, fallback)
10State-transition integrityhas a lifecycle/status machine (illegal-transition guards)
Extend the rubric per domain, but freeze it per benchmark and apply the identical list to every framework. For head-to-head benchmarks, also report pooled recall: union the valid additions across all frameworks into one adjudicated master list, then score each framework's recall against that pool.
运行下面的冻结的隐含需求类别评分标准。对于每个类别，标记为已覆盖（引用
spec.md:line
）、遗漏或不适用（PRD/领域不相关）。该评分标准与PRD无关，可复用，因此无需为每个PRD制作金标就能保证召回率的可复现性。
E_recall = Addressed / (Addressed + Missed)        # N/A excluded
序号类别如果功能满足以下条件则预期有该需求
1输入验证与边界接受用户/外部输入（范围、格式、必填字段）
2错误分类与消息可能以调用方必须区分的方式失败（类型化错误、错误码）
3认证 / 授权暴露端点或接触用户自有数据
4幂等与去重可能被重试或重放（支付、一次性创建资源）
5并发与竞态条件存在共享状态上的交错写入 / 先检查后操作
6数据生命周期与一致性持久化状态（事务、部分失败回滚、保留策略）
7可观测性在生产环境运行（日志、指标、故障可追溯性）
8限制、分页与速率返回列表或可被大量调用
9外部依赖故障调用第三方（超时、重试、熔断、降级）
10状态转换完整性有生命周期/状态机（非法转换防护）
可根据领域扩展评分标准，但每个基准测试要冻结标准，并对所有框架应用完全相同的列表。对于直接对比的基准测试，还需报告合并召回率：将所有框架的有效新增需求合并为一个经过裁定的主列表，然后对照该列表评分每个框架的召回率。

E_precision
— is the signal clean, or gold-plating?

E_precision
（准确率）——信号是干净的，还是过度镀金？

Build the added-requirement ledger: every requirement in 
spec.md
/
tasks.md
not traceable to a PRD line. Adjudicate each, binary, with a one-line warrant + 
spec.md:line
:
VerdictMeaning
Valid-necessarya real gap the PRD implied (e.g. "reject 
trialDays > 30
")
Valid-defensivereasonable hardening (observability, retries, typed errors)
Invalidhallucinated, contradicts the PRD, or scope-creep into the explicit out-of-scope list
E_precision = (Valid-necessary + Valid-defensive) / total additions
The valid additions become the 
valid E-additions
 set referenced everywhere else (the harness denominator, and 
S
traceability). Mark each valid addition built or deferred (deferred-valid = good discipline, see 
S
).
构建新增需求台账：
spec.md
/
tasks.md
中无法追溯到PRD行的所有需求。对每个需求进行二元裁定，并附上一行理由 + 
spec.md:line
：
裁定结果含义
有效-必要PRD隐含的真实缺口（例如“拒绝
trialDays > 30
”）
有效-防御合理的加固（可观测性、重试、类型化错误）
无效幻觉、与PRD矛盾，或者属于明确范围外清单的范围蔓延
E_precision = (Valid-necessary + Valid-defensive) / total additions
有效的新增需求构成了其他地方引用的**
有效E类新增需求
集合（测试套件的分母，以及
S
的可追溯性）。将每个有效新增需求标记为已构建或延后**（有效但延后 = 良好的规范，见
S
）。

E_justified
— are additions warranted?

E_justified
（合理性）——新增需求是否有依据？

E_justified = additions carrying an explicit rationale/traceability / total additions
Unjustified additions are a smell even when individually plausible. Report all three: 
E_recall / E_precision / E_justified
. A strong framework reads high on all three — found the implicit requirements, added little noise, justified what it added.
E_justified = additions carrying an explicit rationale/traceability / total additions
即使单个新增需求看起来合理，没有依据的新增也是一种坏味道。报告全部三个指标：
E_recall / E_precision / E_justified
。优秀的框架在这三个指标上都表现出色——找到了隐含需求，几乎没有噪音，新增的内容都有依据。

Calibration (do this once per PRD, before trusting a benchmark)

校准（每个PRD执行一次，在信任基准测试结果之前）

The checklist is a hypothesis about what each AC means; calibrate it so different evaluators agree:
  1. Anchor. 
    references/reference.md
    holds worked checklist anchors — at least one clearly-MET, one clearly-UNMET, and one borderline check with the verdict and reasoning. Read them before scoring; they fix where the MET/UNMET boundary sits.
  2. Agreement check. For a high-stakes benchmark, have the checklist reviewed against a human-labeled reference (or a second judge model). If verdicts disagree on more than ~20% of checks, the checklist wording is too vague — sharpen the ambiguous checks in 
    _ac-baseline.md
    and re-run. Do not "average away" disagreement; fix the operational definition.
  3. Borderline cases are the valuable ones. When you find a check humans/judges split on, add it to the anchor pool in 
    references/reference.md
    with the resolved verdict.
检查清单是关于每条验收标准含义的假设；对其进行校准，确保不同评估者达成一致：
  1. 锚点。 
    references/reference.md
    包含已完成的检查清单锚点——至少一个明确达标、一个明确未达标、一个边界检查项，附带判定结果和理由。评分前阅读它们；它们确定了达标/未达标的边界位置。
  2. 一致性检查。 对于高风险的基准测试，需对照人工标注的参考（或第二个评判模型）审核检查清单。如果超过约20%的检查项判定不一致，说明检查清单的表述太模糊——需在
    _ac-baseline.md
    中优化有歧义的检查项，然后重新运行。不要“平均掉”分歧；要修正操作定义。
  3. 边界案例是有价值的。 当你发现人类/评判者存在分歧的检查项时，将其连同最终判定结果一起添加到
    references/reference.md
    的锚点池中。

Process

流程

Copy this checklist and track progress:
FRAMEWORK — respect & extract requirements
- [ ] 1. Locate PRD, spec.md/tasks.md. Run git diff to identify changed files (diff surface). Use the diff surface as the primary search scope for implementation and test evidence throughout steps 4 and 8
- [ ] 2. Enumerate stories + ACs; tag priority; mark out-of-scope items
- [ ] 3. Build/load the frozen binary checklist per AC (I-checks + T-checks) → _ac-baseline.md
- [ ] 4. Score I-checks MET/UNMET with file:line evidence (record searches before any UNMET)
- [ ] 5. Elicitation E: run the category rubric (E_recall) + added-requirement ledger (E_precision, E_justified) → freeze the `valid E-additions` set
- [ ] 6. Scope S: trace every built behavior (PRD-boundary / rogue-build / plan-drift)

HARNESS — ensure they are all implemented
- [ ] 7. Define the sanctioned set = PRD ACs ∪ valid E-additions; ensure each has a T-check
- [ ] 8. Score T-checks MET/UNMET with file:line evidence (unit + e2e per fixed level policy)
- [ ] 9. Inventory extra tests → Robustness Index R
- [ ] 10. Classify every feature test into tiers → Test Distribution D (counts + %)
- [ ] 11. Run/record Engineering Gates G (✓/✗/not-run)

ROLL-UP
- [ ] 12. Repeat the scoring steps three times (k=3); take majority verdict per check
- [ ] 13. Compute I, T, AC/Story/Final; apply bands; Adjusted Final only if a gate is ✗
- [ ] 14. Emit the report (see references/reference.md template): Final + E + S + R/G/D + ranked gaps + fixes
Step rules:
  • Step 1 — Identify the diff surface before any evidence search. Pick the base that matches what you are evaluating (branch tip, last commit, or working tree):
bash
git diff <base>..<head> --name-only   # for a branch/PR
git diff --name-only HEAD~1           # for the last commit
git diff --name-only HEAD             # for uncommitted changes
git status --short                    # for untracked new files
Record the file list in the report. Steps 4 and 8 use it as the first set of paths to inspect.
  • Step 2 — Out-of-scope items from the PRD's explicit out-of-scope / non-goals section (whatever heading the PRD uses) get 
    w=0
    . Absence is not a defect. Do not score them.
  • Step 3 — Reuse the existing 
    _ac-baseline.md
    if present; otherwise create it and treat it as the contract for all future runs of this PRD. Include only observable-behavior clauses as checks.
  • Step 5 — 
    E
    grades 
    spec.md
    /
    tasks.md
    , not code. The ledger's valid additions define the 
    valid E-additions
    set used by Steps 6 and 7. Mark each valid addition built/deferred.
  • Step 6 — Traceability: a built behavior with no PRD AC and no valid 
    E
    -addition is a rogue build (
    fail
    ). A deferred-valid out-of-scope addition is not a penalty.
  • Step 7 — The harness denominator is the sanctioned set (PRD ACs ∪ valid E-additions). A valid extracted requirement with no test is a 
    T
    miss, not an 
    E
    miss.
  • Step 8 — Binary only: MET (evidence) or UNMET (searched, absent). No partial per-check values.
  • Step 9 — Extra tests are a positive signal but never substitute for missing coverage. A high 
    R
    with low 
    T
    is still a low grade.
  • Step 10 — Each test lands in exactly one tier; counts must sum to the total feature-test count. Exclude pre-existing tests not added by the feature (note them separately).
  • Step 11 — A gate is 
    ✓
    /
    ✗
    only if actually executed; otherwise 
    not-run
    with the attempt evidence (command + error output — see Probe-before-not-run). Do not assume infra is unavailable: check it (e.g. 
    docker ps
    , or whatever the project's own docs say the e2e suite needs running). If a gate is red, do NOT fix the subject code (Core rule 5).
  • Step 12 — Majority over k=3. Note any check where the three passes disagreed.
  • Step 13 — Show the arithmetic: list MET/total per check set so 
    I
    and 
    T
    are recomputable. Execute the roll-up as a script and paste its output (Reproducibility rule 4) — never hand-compute 
    Σw
    or 
    Final
    .
复制此检查清单并跟踪进度：
框架层 —— 遵守并提取需求
- [ ] 1. 定位PRD、spec.md/tasks.md。运行git diff识别变更文件（变更面）。在步骤4和8中，将变更面作为实现和测试证据的主要搜索范围
- [ ] 2. 枚举故事 + 验收标准；标记优先级；标记范围外项
- [ ] 3. 为每条验收标准构建/加载冻结的二元检查清单（I类检查项 + T类检查项）→ _ac-baseline.md
- [ ] 4. 用file:line证据为I类检查项评分达标/未达标（任何未达标前先记录搜索过程）
- [ ] 5. 需求启发E：运行类别评分标准（E_recall） + 新增需求台账（E_precision、E_justified）→ 冻结`有效E类新增需求`集合
- [ ] 6. 范围S：追溯每个已构建行为（PRD边界 / rogue构建 / 计划偏移）

测试套件层 —— 确保所有需求都已实现
- [ ] 7. 定义获批集合 = PRD验收标准 ∪ 有效E类新增需求；确保每个需求都有对应的T类检查项
- [ ] 8. 用file:line证据为T类检查项评分达标/未达标（根据固定级别策略的单元测试 + e2e测试）
- [ ] 9. 盘点额外测试 → 健壮性指数R
- [ ] 10. 将每个功能测试分类到层级 → 测试分布D（计数 + 百分比）
- [ ] 11. 运行/记录工程门禁G（✓/✗/未运行）

汇总
- [ ] 12. 重复评分步骤三次（k=3）；每个检查项取多数票判定结果
- [ ] 13. 计算I、T、验收标准/故事/最终得分；应用等级；仅当门禁为✗时计算调整后最终得分
- [ ] 14. 生成报告（见references/reference.md模板）：最终得分 + E + S + R/G/D + 排名差距 + 修复方案
步骤规则：
  • 步骤1 —— 在任何证据搜索之前确定变更面。选择与你正在评估的内容匹配的基准（分支顶端、上一次提交、或工作树）：
bash
git diff <base>..<head> --name-only   # for a branch/PR
git diff --name-only HEAD~1           # for the last commit
git diff --name-only HEAD             # for uncommitted changes
git status --short                    # for untracked new files
在报告中记录文件列表。步骤4和8将其作为第一组要检查的路径。
  • 步骤2 —— 来自PRD明确的范围外/非目标部分（不管PRD用什么标题）的范围外项权重为
    w=0
    。未实现这些不是缺陷。不要对它们评分。
  • 步骤3 —— 如果存在已有的
    _ac-baseline.md
    则复用；否则创建它，并将其作为该PRD所有未来运行的契约。仅将可观察行为条款作为检查项。
  • 步骤5 —— 
    E
    评估的是
    spec.md
    /
    tasks.md
    ，而非代码。台账中的有效新增需求定义了步骤6和7使用的
    有效E类新增需求
    集合。将每个有效新增需求标记为已构建/延后。
  • 步骤6 —— 可追溯性：没有PRD验收标准且没有有效E类新增需求的已构建行为属于rogue构建（
    未通过
    ）。延后的有效范围外新增需求不扣分。
  • 步骤7 —— 测试套件的分母是获批集合（PRD验收标准 ∪ 有效E类新增需求）。有效的提取需求如果没有测试，属于
    T
    遗漏，而非
    E
    遗漏。
  • 步骤8 —— 仅二元判定：达标（有证据）或未达标（已搜索，不存在）。每个检查项没有部分得分。
  • 步骤9 —— 额外测试是积极信号，但绝不能替代缺失的覆盖。
    R
    高但
    T
    低的情况仍然是低评分。
  • 步骤10 —— 每个测试恰好属于一个层级；计数之和必须等于功能测试总数。排除不是本次功能新增的预先存在的测试（单独记录）。
  • 步骤11 —— 门禁只有在实际执行后才是
    ✓
    /
    ✗
    ；否则为
    未运行
    ，并附带尝试证据（命令 + 错误输出——见未运行前先探测）。不要假设基础设施不可用：去检查（例如
    docker ps
    ，或者项目自己的文档中说明的e2e测试套件需要运行的东西）。如果门禁是红的，不要修改被评估代码（核心规则5）。
  • 步骤12 —— k=3的多数投票。记录任何三次运行存在分歧的检查项。
  • 步骤13 —— 展示计算过程：列出每组检查项的达标数/总数，以便
    I
    和
    T
    可重新计算。通过脚本执行汇总并粘贴输出（可复现性规则4）——绝不要手动计算
    Σw
    或
    最终得分
    。

Output

输出

Produce a markdown report following the template in reference.md. It must separate the two subjects and contain: per-AC implementation checklist table (I-checks, MET/UNMET, evidence); the Elicitation 
E
block (category-rubric recall table + added-requirement ledger with verdicts → 
E_recall
/ 
E_precision
/ 
E_justified
); the Scope 
S
traceability verdict; per-requirement unit/e2e test checklist table (T-checks over the sanctioned set = PRD ACs ∪ valid E-additions); extra-tests inventory; the test-distribution 
D
table (tiers with counts + %); the computed 
Final
(+ 
Adjusted Final
if a gate is ✗); 
R
/ 
G
/ 
D
; a ranked gap list; and concrete fixes to reach 1.00.
Save the report to 
<spec-folder>/evaluations/<priority>-<story-slug>-<timestamp>.md
when a 
.specs
/spec folder exists; otherwise present it inline. The 
<timestamp>
is a UTC instant in 
YYYYMMDDTHHMMSSZ
form (e.g. 
20260606T143012Z
), obtained from the system clock at write time (
date -u +%Y%m%dT%H%M%SZ
). The timestamp makes every report filename unique so multiple agents evaluating the same story in parallel never overwrite each other — each run produces its own file. If a same-second collision is still possible (many parallel runs), append a short run id: 
<priority>-<story-slug>-<timestamp>-<run-id>.md
.
Note: only the report filename is timestamped. The frozen 
_ac-baseline.md
is deliberately not timestamped — it is a single shared artifact that every run reads from to stay comparable; never fork it per run.
按照reference.md中的模板生成markdown报告。报告必须区分两个评估对象，并包含：每条验收标准的实现检查清单表格（I类检查项、达标/未达标、证据）；需求启发
E
模块（类别评分标准召回率表格 + 带裁定结果的新增需求台账 → 
E_recall
/ 
E_precision
/ 
E_justified
）；范围
S
的可追溯性判定；每个需求的单元/e2e测试检查清单表格（针对获批集合 = PRD验收标准 ∪ 有效E类新增需求的T类检查项）；额外测试清单；测试分布
D
表格（各层级的计数 + 百分比）；计算出的
最终得分
（如果门禁为✗则加上
调整后最终得分
）；
R
/ 
G
/ 
D
；排名的差距列表；以及达到1.00分的具体修复方案。
如果存在
.specs
/spec文件夹，将报告保存到
<spec-folder>/evaluations/<优先级>-<故事slug>-<时间戳>.md
；否则直接内联展示。
<时间戳>
是UTC时间，格式为
YYYYMMDDTHHMMSSZ
（例如
20260606T143012Z
），在写入时从系统时钟获取（
date -u +%Y%m%dT%H%M%SZ
）。时间戳让每个报告的文件名都是唯一的，因此并行评估同一故事的多个Agent绝不会互相覆盖——每次运行生成自己的文件。如果仍然可能出现同一秒的冲突（大量并行运行），则附加一个短的运行ID：
<优先级>-<故事slug>-<时间戳>-<运行ID>.md
。
注意：只有报告的文件名带有时间戳。冻结的
_ac-baseline.md
故意不带时间戳——它是一个单一的共享产物，每次运行都读取它以保持可比性；绝不要每次运行都分叉出一个版本。

Worked example

实操示例

A complete applied evaluation (an example billing service's P0 "Start Free Trial Without a Card") showing the binary checklist per AC with 
file:line
evidence, plus calibration anchors, is in reference.md. Mirror its structure and rigor.
reference.md中有一个完整的应用评估示例（示例计费服务的P0“Start Free Trial Without a Card”），展示了每条验收标准的二元检查清单和
file:line
证据，以及校准锚点。请参照其结构和严谨性。

Anti-patterns

反模式

  • Scoring 
    I
    /
    T
    on a sliding "feels like a 0.75" judgment instead of counting binary checks.
  • Re-deriving the AC list or checklist per implementation (breaks comparability) instead of reusing the frozen baseline.
  • Marking a check UNMET without showing the search performed; or MET without 
    file:line
    evidence.
  • Searching the full codebase for evidence without first running 
    git diff
    to bound the diff surface — this produces unreliable searches and inflates "not found" confidence.
  • Awarding implementation credit without reading the production code path end-to-end.
  • Counting extra/defensive tests toward an AC's required coverage, or letting more code/tests raise the grade.
  • Reporting an Engineering Gate as passing without running it; or applying 
    Adjusted Final
    for a 
    not-run
    gate.
  • Marking a gate 
    not-run
    without a recorded attempt (command + error output). "Infra not available" without a probe is fabricated evidence.
  • Fixing the code under evaluation to turn a red gate green — the evaluator is read-only over the subject; a red gate is scored 
    ✗
    and the fix goes in the report.
  • Hand-computing the roll-up (
    Σw
    , 
    Final
    ) instead of executing it as a script and pasting the output.
  • Penalizing missing out-of-scope features.
  • Folding 
    E
    or 
    S
    into 
    Final
    — the headline grade must mean only "fidelity to the given PRD" or comparability breaks.
  • Penalizing a valid requirement the framework extracted but correctly deferred as out-of-scope (that is good discipline, not a scope failure).
  • Scoring harness completeness (
    T
    ) against the PRD alone — the denominator is the sanctioned set (PRD ACs ∪ valid E-additions).
  • Rewarding requirement volume in 
    E
    : hallucinated/scope-creep additions lower 
    E_precision
    , they don't raise the score.
  • Changing the weights (
    0.6/0.4
    , P0/P1/P2) between implementations of the same PRD.
  • 用滑动式的“感觉像0.75分”的判断来给
    I
    /
    T
    评分，而不是计数二元检查项。
  • 每个实现都重新推导验收标准列表或检查清单（破坏可比性），而不是复用冻结的基线。
  • 没有展示执行的搜索过程就将检查项标记为未达标；或者没有
    file:line
    证据就标记为达标。
  • 没有先运行
    git diff
    界定变更面就搜索整个代码库找证据——这会导致不可靠的搜索结果，并高估“未找到”的可信度。
  • 没有端到端通读生产代码路径就给予实现得分。
  • 将额外/防御性测试计入验收标准的所需覆盖，或者让更多的代码/测试提高评分。
  • 没有运行就报告工程门禁通过；或者对
    未运行
    的门禁应用
    调整后最终得分
    。
  • 没有记录尝试过程（命令 + 错误输出）就将门禁标记为
    未运行
    。没有探测就说“基础设施不可用”属于伪造证据。
  • 修改被评估的代码以让红门禁变绿——评估者对评估对象仅有只读权限；红门禁就评
    ✗
    ，修复方案写在报告里。
  • 手动计算汇总得分（
    Σw
    、
    最终得分
    ），而不是通过脚本执行并粘贴输出。
  • 因为缺少范围外的功能而扣分。
  • 将
    E
    或
    S
    计入
    最终得分
    ——headline评分必须仅代表“对给定PRD的保真度”，否则可比性会被破坏。
  • 因为框架提取但正确地延后了一个有效需求（因为超出范围）而扣分——这是良好的规范，不是范围失败。
  • 仅对照PRD给测试套件完整性（
    T
    ）评分——分母是获批集合（PRD验收标准 ∪ 有效E类新增需求）。
  • 在
    E
    中奖励需求数量：幻觉/范围蔓延的新增需求会降低
    E_precision
    ，而不是提高得分。
  • 在同一PRD的不同实现之间修改权重（
    0.6/0.4
    、P0/P1/P2）。