evals-clarify
Compare original and translation side by side
🇺🇸
Original
English🇨🇳
Translation
Chineseevals-clarify
evals-clarify
What this skill does
这个 Skill 的作用
Conducts axial coding following EDD Principles III & IX to cluster related failure patterns, refine evaluation criteria, generate adversarial examples, and accept validated drafts into the published goldset.
Output:
- Clustered Criteria - Related patterns grouped into coherent evaluation themes
- Adversarial Examples - Generated attack scenarios and edge cases for robustness
- Published Goldset - Accepted criteria in with full documentation
evals/{system}/goldset.md - Holdout Dataset - Reserved test set (20%) for unbiased evaluation validation
- JSON Configuration - Auto-generated for system consumption
goldset.json - Auto-handoff to for grader generation
/evals-implement
Key EDD Principles Applied:
- Principle III: Error Analysis & Pattern Discovery - Axial coding → theoretical relationships
- Principle IX: Test Data as Code - Adversarial generation, holdout splits, version control
- Principle II: Binary Pass/Fail - Maintain strict binary evaluation throughout
- Principle I: Spec-Driven Contracts - Criteria validate spec compliance
根据 EDD 原则 III 和 IX 执行轴向编码(axial coding),用于聚类相关失败模式、细化评估标准、生成对抗性示例,并将通过验证的草案验收到已发布的 goldset 中。
输出:
- 聚类后的标准 - 将相关模式归入连贯的评估主题
- 对抗性示例 - 生成用于验证稳健性的攻击场景和边缘用例
- 已发布的 Goldset - 位于 的已验收标准,并附完整文档
evals/{system}/goldset.md - 留出数据集 - 保留的测试集(20%),用于无偏评估验证
- JSON 配置 - 自动生成供系统使用的
goldset.json - 自动移交给 ,用于生成评分器
/evals-implement
应用的 EDD 关键原则:
- 原则 III:错误分析与模式发现 - 轴向编码 → 理论关联
- 原则 IX:测试数据即代码(Test Data as Code)- 对抗性生成、留出划分、版本控制
- 原则 II:二元通过/失败(Binary Pass/Fail)- 始终保持严格的二元评估
- 原则 I:规范驱动契约(Spec-Driven Contracts)- 使用标准验证规范符合性
When to use
何时使用
- After : Refine and accept draft criteria into goldset
/evals-specify - Dataset maintenance: Balance pass/fail examples or add adversarial cases
- Adding holdout split: Isolate validation data from training data
- 执行 之后:将草案标准细化并验收到 goldset 中
/evals-specify - 维护数据集:平衡通过/失败示例或添加对抗性用例
- 添加留出划分:将验证数据与训练数据隔离
When NOT to use
何时不要使用
- No draft criteria exist: Run to discover patterns first
/evals-specify - Grader generation: Use to convert accepted goldset into code
/evals-implement
- 不存在草案标准:先运行 以发现模式
/evals-specify - 生成评分器:使用 将已验收的 goldset 转换为代码
/evals-implement
Process
流程
User Input
用户输入
text
$ARGUMENTS- — Accept specific draft IDs (e.g., "EVAL-001,EVAL-003")
--accept IDS - — Merge related criteria (e.g., "EVAL-001+EVAL-002")
--merge IDS - — Split complex criterion into multiple focused criteria
--split ID - — Holdout percentage (default: 0.2, range: 0.1-0.3)
--holdout-ratio RATIO
text
$ARGUMENTS- — 验收特定的草案 ID(例如,"EVAL-001,EVAL-003")
--accept IDS - — 合并相关标准(例如,"EVAL-001+EVAL-002")
--merge IDS - — 将复杂标准拆分为多个聚焦的标准
--split ID - — 留出集百分比(默认值:0.2,范围:0.1-0.3)
--holdout-ratio RATIO
Execution Steps
执行步骤
Step 0: Detect Lightweight Draft Format
步骤 0:检测轻量级草案格式
Check if the draft being reviewed uses the lightweight draft template (indicated
by presence of , , , fields in frontmatter
and / body sections without the full
formal template sections).
typeevidencesourcerevisit-when## Rejected Alternatives## ReasonIf lightweight:
- Read the draft's captured fields (Context, Decision, Rejected Alternatives, Reason)
- Transform to full formal template:
- ADR: full MADR format with Decision Drivers, Considered Options, Pros/Cons, Constitution Alignment, Related ADRs
- PDR: full PDR format with Market Forces, Consequences, Alternatives Considered, Links
- ChDR: full ChDR format with Issue Links, Commits, Consequences, Evidence
- CDR: full CDR format with Context Type, Target Module, Descriptor, Evidence
- EVAL: full eval format with Error Analysis, Pass/Fail Examples, Implementation Notes
- Enrich from session context (add details the lightweight draft may have omitted)
- Present the enriched draft for review
If already full format, proceed with normal review.
检查正在审查的草案是否使用轻量级草案模板。判断依据是:frontmatter 中包含 、、、 字段,正文中包含 / 章节,但缺少完整的正式模板章节。
typeevidencesourcerevisit-when## Rejected Alternatives## Reason如果采用轻量级格式:
- 读取草案中已捕获的字段(Context、Decision、Rejected Alternatives、Reason)
- 转换为完整的正式模板:
- ADR:完整的 MADR 格式,包含 Decision Drivers、Considered Options、Pros/Cons、Constitution Alignment、Related ADRs
- PDR:完整的 PDR 格式,包含 Market Forces、Consequences、Alternatives Considered、Links
- ChDR:完整的 ChDR 格式,包含 Issue Links、Commits、Consequences、Evidence
- CDR:完整的 CDR 格式,包含 Context Type、Target Module、Descriptor、Evidence
- EVAL:完整的评估格式,包含 Error Analysis、Pass/Fail Examples、Implementation Notes
- 根据会话上下文补充轻量级草案可能遗漏的细节
- 提交补充后的草案供审查
如果已经是完整格式,则继续正常审查。
Phase 1: Axial Coding & Clustering
阶段 1:轴向编码与聚类
- Group related draft patterns into coherent themes.
- Resolve any overlaps or duplicate criteria.
- 将相关草案模式归入连贯的主题。
- 解决标准之间的重叠或重复问题。
Phase 2: Refinement & Adversarial Generation
阶段 2:细化与对抗性生成
- Generate 3-5 adversarial (attack) examples per criterion to test robustness.
- Balance pass/fail examples (~50/50 ratio).
- Write every clarification point and criterion revision in Simplified Technical English (short sentences, one idea each, active voice, imperative asks, named referents, plain words). Read the contract at before drafting. Count crossed boundaries (
skills/authoring/references/ste-rules.md,record,lane,team): 0–1 boundaries use plain STE prose, however severe the gap; 2 or more boundaries draw the full ASCII decision map from the reference (WHAT/WHY, crossed boundaries, reach fan-out, candidates × discriminating axes, IF-UNDECIDED). Render unknown gaps asnowslots, never prose complaints.?
- 为每条标准生成 3–5 个对抗性(攻击)示例,以测试稳健性。
- 平衡通过/失败示例(约 50/50)。
- 使用 Simplified Technical English(STE) 编写每个澄清点和标准修订(句子简短、每句只表达一个观点、采用主动语态、使用祈使句、明确指代对象、使用平实词语)。起草前,读取 中的编写契约。统计跨越的边界(
skills/authoring/references/ste-rules.md、record、lane、team):0–1 个边界时使用简明 STE 文本,无论缺口多么严重;2 个及以上边界时,从参考文档绘制完整的 ASCII 决策图(WHAT/WHY、跨越的边界、触达扩散、候选项 × 区分轴、IF-UNDECIDED)。将未知缺口呈现为now占位符,绝不以文字抱怨的形式呈现。?
Phase 3: Holdout Isolation
阶段 3:留出集隔离
- Isolate exactly 20% of examples as a reserved holdout set (saved to — machine data stays under
.adlc/memory/evals/holdout.jsonper the shared layout)..adlc/ - Ensure holdout set is never used in implementation or training.
- 准确隔离 20% 的示例作为保留的 holdout 集(保存到 ——按照共享布局,机器数据始终位于
.adlc/memory/evals/holdout.json下)。.adlc/ - 确保 holdout 集绝不用于实现或训练。
Phase 4: Publish Goldset
阶段 4:发布 Goldset
- Copy accepted drafts to (the shared layout memory root;
docs/adlc/memory/evals/alone stays atholdout.json) and update status to.adlc/memory/evals/holdout.json.accepted - Compile published goldset to (human-readable) and
evals/{system}/goldset.md(machine-readable).evals/{system}/goldset.json
- 将已验收的草案复制到 (共享布局的 memory 根目录;仅
docs/adlc/memory/evals/保留在holdout.json),并将状态更新为.adlc/memory/evals/holdout.json。accepted - 将已发布的 goldset 编译为 (人类可读格式)和
evals/{system}/goldset.md(机器可读格式)。evals/{system}/goldset.json
Phase 5: Auto-Handoff
阶段 5:自动移交
Trigger to generate code.
/evals-implement触发 生成代码。
/evals-implementVerification
验证
- Accepted drafts stored in
docs/adlc/memory/evals/EVAL-*.md - and
evals/{system}/goldset.mdexistgoldset.json - Holdout set isolated and populated
.adlc/memory/evals/holdout.json - All criteria are strictly binary (no confidence scores or Likert scales)
- Handover summary lists accepted criteria and adversarial counts
- 已验收的草案存储在 中
docs/adlc/memory/evals/EVAL-*.md - 和
evals/{system}/goldset.md存在goldset.json - 留出集 已隔离并完成填充
.adlc/memory/evals/holdout.json - 所有标准均采用严格的二元评估(不包含置信度分数或 Likert 量表)
- 移交摘要列出已验收的标准和对抗性示例数量