evals-clarify

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

evals-clarify

evals-clarify

What this skill does

这个 Skill 的作用

Conducts axial coding following EDD Principles III & IX to cluster related failure patterns, refine evaluation criteria, generate adversarial examples, and accept validated drafts into the published goldset.
Output:
  1. Clustered Criteria - Related patterns grouped into coherent evaluation themes
  2. Adversarial Examples - Generated attack scenarios and edge cases for robustness
  3. Published Goldset - Accepted criteria in
    evals/{system}/goldset.md
    with full documentation
  4. Holdout Dataset - Reserved test set (20%) for unbiased evaluation validation
  5. JSON Configuration - Auto-generated
    goldset.json
    for system consumption
  6. Auto-handoff to
    /evals-implement
    for grader generation
Key EDD Principles Applied:
  • Principle III: Error Analysis & Pattern Discovery - Axial coding → theoretical relationships
  • Principle IX: Test Data as Code - Adversarial generation, holdout splits, version control
  • Principle II: Binary Pass/Fail - Maintain strict binary evaluation throughout
  • Principle I: Spec-Driven Contracts - Criteria validate spec compliance
根据 EDD 原则 III 和 IX 执行轴向编码(axial coding),用于聚类相关失败模式、细化评估标准、生成对抗性示例,并将通过验证的草案验收到已发布的 goldset 中。
输出:
  1. 聚类后的标准 - 将相关模式归入连贯的评估主题
  2. 对抗性示例 - 生成用于验证稳健性的攻击场景和边缘用例
  3. 已发布的 Goldset - 位于
    evals/{system}/goldset.md
    的已验收标准,并附完整文档
  4. 留出数据集 - 保留的测试集(20%),用于无偏评估验证
  5. JSON 配置 - 自动生成供系统使用的
    goldset.json
  6. 自动移交给
    /evals-implement
    ,用于生成评分器
应用的 EDD 关键原则:
  • 原则 III:错误分析与模式发现 - 轴向编码 → 理论关联
  • 原则 IX:测试数据即代码(Test Data as Code)- 对抗性生成、留出划分、版本控制
  • 原则 II:二元通过/失败(Binary Pass/Fail)- 始终保持严格的二元评估
  • 原则 I:规范驱动契约(Spec-Driven Contracts)- 使用标准验证规范符合性

When to use

何时使用

  • After
    /evals-specify
    : Refine and accept draft criteria into goldset
  • Dataset maintenance: Balance pass/fail examples or add adversarial cases
  • Adding holdout split: Isolate validation data from training data
  • 执行
    /evals-specify
    之后
    :将草案标准细化并验收到 goldset 中
  • 维护数据集:平衡通过/失败示例或添加对抗性用例
  • 添加留出划分:将验证数据与训练数据隔离

When NOT to use

何时不要使用

  • No draft criteria exist: Run
    /evals-specify
    to discover patterns first
  • Grader generation: Use
    /evals-implement
    to convert accepted goldset into code
  • 不存在草案标准:先运行
    /evals-specify
    以发现模式
  • 生成评分器:使用
    /evals-implement
    将已验收的 goldset 转换为代码

Process

流程

User Input

用户输入

text
$ARGUMENTS
  • --accept IDS
    — Accept specific draft IDs (e.g., "EVAL-001,EVAL-003")
  • --merge IDS
    — Merge related criteria (e.g., "EVAL-001+EVAL-002")
  • --split ID
    — Split complex criterion into multiple focused criteria
  • --holdout-ratio RATIO
    — Holdout percentage (default: 0.2, range: 0.1-0.3)
text
$ARGUMENTS
  • --accept IDS
    — 验收特定的草案 ID(例如,"EVAL-001,EVAL-003")
  • --merge IDS
    — 合并相关标准(例如,"EVAL-001+EVAL-002")
  • --split ID
    — 将复杂标准拆分为多个聚焦的标准
  • --holdout-ratio RATIO
    — 留出集百分比(默认值:0.2,范围:0.1-0.3)

Execution Steps

执行步骤

Step 0: Detect Lightweight Draft Format

步骤 0:检测轻量级草案格式

Check if the draft being reviewed uses the lightweight draft template (indicated by presence of
type
,
evidence
,
source
,
revisit-when
fields in frontmatter and
## Rejected Alternatives
/
## Reason
body sections without the full formal template sections).
If lightweight:
  1. Read the draft's captured fields (Context, Decision, Rejected Alternatives, Reason)
  2. Transform to full formal template:
    • ADR: full MADR format with Decision Drivers, Considered Options, Pros/Cons, Constitution Alignment, Related ADRs
    • PDR: full PDR format with Market Forces, Consequences, Alternatives Considered, Links
    • ChDR: full ChDR format with Issue Links, Commits, Consequences, Evidence
    • CDR: full CDR format with Context Type, Target Module, Descriptor, Evidence
    • EVAL: full eval format with Error Analysis, Pass/Fail Examples, Implementation Notes
  3. Enrich from session context (add details the lightweight draft may have omitted)
  4. Present the enriched draft for review
If already full format, proceed with normal review.
检查正在审查的草案是否使用轻量级草案模板。判断依据是:frontmatter 中包含
type
、
evidence
、
source
、
revisit-when
字段,正文中包含
## Rejected Alternatives
/
## Reason
章节,但缺少完整的正式模板章节。
如果采用轻量级格式:
  1. 读取草案中已捕获的字段(Context、Decision、Rejected Alternatives、Reason)
  2. 转换为完整的正式模板:
    • ADR:完整的 MADR 格式,包含 Decision Drivers、Considered Options、Pros/Cons、Constitution Alignment、Related ADRs
    • PDR:完整的 PDR 格式,包含 Market Forces、Consequences、Alternatives Considered、Links
    • ChDR:完整的 ChDR 格式,包含 Issue Links、Commits、Consequences、Evidence
    • CDR:完整的 CDR 格式,包含 Context Type、Target Module、Descriptor、Evidence
    • EVAL:完整的评估格式,包含 Error Analysis、Pass/Fail Examples、Implementation Notes
  3. 根据会话上下文补充轻量级草案可能遗漏的细节
  4. 提交补充后的草案供审查
如果已经是完整格式,则继续正常审查。

Phase 1: Axial Coding & Clustering

阶段 1:轴向编码与聚类

  • Group related draft patterns into coherent themes.
  • Resolve any overlaps or duplicate criteria.
  • 将相关草案模式归入连贯的主题。
  • 解决标准之间的重叠或重复问题。

Phase 2: Refinement & Adversarial Generation

阶段 2:细化与对抗性生成

  • Generate 3-5 adversarial (attack) examples per criterion to test robustness.
  • Balance pass/fail examples (~50/50 ratio).
  • Write every clarification point and criterion revision in Simplified Technical English (short sentences, one idea each, active voice, imperative asks, named referents, plain words). Read the contract at
    skills/authoring/references/ste-rules.md
    before drafting. Count crossed boundaries (
    record
    ,
    lane
    ,
    team
    ,
    now
    ): 0–1 boundaries use plain STE prose, however severe the gap; 2 or more boundaries draw the full ASCII decision map from the reference (WHAT/WHY, crossed boundaries, reach fan-out, candidates × discriminating axes, IF-UNDECIDED). Render unknown gaps as
    ?
    slots, never prose complaints.
  • 为每条标准生成 3–5 个对抗性(攻击)示例,以测试稳健性。
  • 平衡通过/失败示例(约 50/50)。
  • 使用 Simplified Technical English(STE) 编写每个澄清点和标准修订(句子简短、每句只表达一个观点、采用主动语态、使用祈使句、明确指代对象、使用平实词语)。起草前,读取
    skills/authoring/references/ste-rules.md
    中的编写契约。统计跨越的边界(
    record
    、
    lane
    、
    team
    、
    now
    ):0–1 个边界时使用简明 STE 文本,无论缺口多么严重;2 个及以上边界时,从参考文档绘制完整的 ASCII 决策图(WHAT/WHY、跨越的边界、触达扩散、候选项 × 区分轴、IF-UNDECIDED)。将未知缺口呈现为
    ?
    占位符,绝不以文字抱怨的形式呈现。

Phase 3: Holdout Isolation

阶段 3:留出集隔离

  • Isolate exactly 20% of examples as a reserved holdout set (saved to
    .adlc/memory/evals/holdout.json
    — machine data stays under
    .adlc/
    per the shared layout).
  • Ensure holdout set is never used in implementation or training.
  • 准确隔离 20% 的示例作为保留的 holdout 集(保存到
    .adlc/memory/evals/holdout.json
    ——按照共享布局,机器数据始终位于
    .adlc/
    下)。
  • 确保 holdout 集绝不用于实现或训练。

Phase 4: Publish Goldset

阶段 4:发布 Goldset

  • Copy accepted drafts to
    docs/adlc/memory/evals/
    (the shared layout memory root;
    holdout.json
    alone stays at
    .adlc/memory/evals/holdout.json
    ) and update status to
    accepted
    .
  • Compile published goldset to
    evals/{system}/goldset.md
    (human-readable) and
    evals/{system}/goldset.json
    (machine-readable).
  • 将已验收的草案复制到
    docs/adlc/memory/evals/
    (共享布局的 memory 根目录;仅
    holdout.json
    保留在
    .adlc/memory/evals/holdout.json
    ),并将状态更新为
    accepted
    。
  • 将已发布的 goldset 编译为
    evals/{system}/goldset.md
    (人类可读格式)和
    evals/{system}/goldset.json
    (机器可读格式)。

Phase 5: Auto-Handoff

阶段 5:自动移交

Trigger
/evals-implement
to generate code.
触发
/evals-implement
生成代码。

Verification

验证

  • Accepted drafts stored in
    docs/adlc/memory/evals/EVAL-*.md
  • evals/{system}/goldset.md
    and
    goldset.json
    exist
  • Holdout set
    .adlc/memory/evals/holdout.json
    isolated and populated
  • All criteria are strictly binary (no confidence scores or Likert scales)
  • Handover summary lists accepted criteria and adversarial counts
  • 已验收的草案存储在
    docs/adlc/memory/evals/EVAL-*.md
    中
  • evals/{system}/goldset.md
    和
    goldset.json
    存在
  • 留出集
    .adlc/memory/evals/holdout.json
    已隔离并完成填充
  • 所有标准均采用严格的二元评估(不包含置信度分数或 Likert 量表)
  • 移交摘要列出已验收的标准和对抗性示例数量