improve-skill-quality

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Improve Skill Quality

提升技能质量

Turn a failing or unconvincing evaluation into a targeted fix. The single most common mistake in this repo is rewriting skill prose in response to a verdict whose real cause was the eval, the fixtures, or the harness. Classify first, then fix.

将失败或缺乏说服力的评估结果转化为针对性修复。本仓库中最常见的错误是：针对某个verdict就重写技能文本，但该verdict的真实原因其实是评估逻辑（eval）、测试夹具（fixture）或测试框架（harness）问题。请先分类问题，再进行修复。

When to Use

适用场景

An evaluation verdict is a regression, underpowered, or "no credible improvement".
A skill wins in the isolated arm but not in the plugin arm, or is reported "not activated".
```
/evaluate
```
reports "Evaluation ran but produced no results".
A skill scores well but costs too much (tokens, turns, wall time, plugin menu budget).
Deciding whether to strengthen or retire a persistently weak skill.

评估verdict为回归、测试力度不足或“无可靠改进”。
技能在独立测试组中表现合格，但在插件测试组中失败，或被标记为“未激活”。
```
/evaluate
```
命令返回“评估已运行但未生成结果”。
技能得分良好，但成本过高（令牌消耗、交互轮次、耗时、插件菜单预算）。
决定是否强化或淘汰持续表现薄弱的技能。

When Not to Use

不适用场景

Creating a new skill from scratch — use
```
create-skill
```
.
Creating a new
```
eval.yaml
```
from scratch — use
```
create-skill-test
```
.

Changing the harness itself (

eng/skill-validator

eng/vally-adapter

evaluation*.yml

从零创建新技能——请使用
```
create-skill
```
。
从零创建新的
```
eval.yaml
```
——请使用
```
create-skill-test
```
。

修改测试框架本身（

eng/skill-validator

、

eng/vally-adapter

、

evaluation*.yml

）。

Inputs

输入项

Input	Required	Description
Verdict evidence	Yes	The `/evaluate` PR comment, or `results.json` from the run artifacts
Losing trial transcripts	Yes for content fixes	Baseline vs. skilled output plus the judge's stated reason
W/T/L record and trial count	Yes	Distinguishes a real regression from an underpowered eval
Activation status per arm	Yes	Isolated and plugin activation are different failures

输入项	是否必填	描述
Verdict证据	是	`/evaluate` 的PR评论，或运行产物中的 `results.json` 文件
失败测试的转录记录	内容修复时必填	基线与技能输出对比，以及评判方给出的明确理由
胜负平记录与测试次数	是	区分真实回归与测试力度不足的情况
各测试组的激活状态	是	独立组与插件组的激活失败原因不同

Workflow

工作流程

Step 1: Get the evidence before forming a hypothesis

步骤1：先收集证据，再形成假设

Read InvestigatingResults.md for how to download artifacts and read

results.json

. Extract, per failing stimulus:

win / tie / loss record and total trials (
```
trials = stimuli × runs
```
)
activation status in the isolated and plugin arms, separately
the judge's verbatim reason on each losing trial
whether any trial errored, timed out, or produced empty output

Do not change skill content until you can quote a losing trial and the judge's reason for it. For the other cause classes the evidence is different: harness failures are diagnosed from the job log and the spec, and power problems from the trial record — neither has a losing trial to quote, and demanding one is what sends people rewriting prose instead.

阅读InvestigatingResults.md了解如何下载产物并读取

results.json

。针对每个失败的测试用例，提取以下信息：

胜负平记录与总测试次数（
```
trials = stimuli × runs
```
）
独立组和插件组各自的激活状态
评判方针对每次失败测试给出的原话理由
是否存在测试报错、超时或输出为空的情况

在能引用失败测试案例及评判理由之前，请勿修改技能内容。其他类型问题的证据形式不同：框架故障需从作业日志和规格文件诊断，测试力度问题需从测试记录分析——这两类问题都没有可引用的失败测试案例，若强行寻找只会导致错误地重写文本。

Step 2: Classify the failure

步骤2：对失败类型进行分类

Work down this table and stop at the first row that matches. Rows are ordered by how often the symptom has been misdiagnosed as a skill-content problem — the fixture row is first because a fixture failure also presents as a setup or reliability failure and gets misfiled as one.

Symptom	Real cause class	Go to
A fixture does not build, is untracked by git, breaks for the wrong reason, or contradicts itself	Fixture	Step 4
No `results.json` , "produced no results", or the spec never loaded	Harness / spec-load	Step 3
Trials errored, timed out, or returned empty output	Reliability	Step 3
Trajectories unmatched, a trial errored, or the summary disagrees — verdict reported inconclusive	Reliability (not power)	Step 3
Positive record (e.g. 16W/8T/1L), comparison conclusive, verdict still not a pass	Statistical power	Step 5
Skilled arm equals baseline arm by construction	Eval design	Step 6
Activated and lost on quality, judge names a concrete defect	Skill content	Step 7
Activated in isolation, not in plugin	Activation / routing	Step 8
Not activated in either arm	Frontmatter description	Step 8
Wins but costs far more than baseline	Scope and cost	Step 7

A verdict is only a measured result when the comparison was conclusive:

adapt.mjs

requires zero errored trials, zero unmatched trajectories, and an agreeing summary before it will report a pass or a regression. Confirm that before reading a record as a power problem.

按以下表格从上到下匹配，找到第一个符合的行即可停止。行的排序依据是：该症状被误判为技能内容问题的频率——夹具问题排在最前面，因为夹具故障也会表现为配置或可靠性问题，常被错误归类。

症状	真实原因类别	跳转至步骤
测试夹具构建失败、未被git追踪、因错误原因失效或自相矛盾	Fixture（测试夹具）	步骤4
无 `results.json` 、“未生成结果”或规格文件从未加载	Harness / 规格加载	步骤3
测试报错、超时或返回空输出	可靠性	步骤3
轨迹不匹配、测试报错或总结不一致——verdict显示无结论	可靠性（非测试力度问题）	步骤3
记录为正（如16胜/8平/1负）、对比结论明确，但verdict仍未通过	统计测试力度	步骤5
技能组与基线组从设计上完全一致	评估设计	步骤6
已激活但因质量失败，评判方指出具体缺陷	技能内容	步骤7
在独立组中激活，但在插件组中未激活	激活/路由	步骤8
在两组中均未激活	前置描述（Frontmatter）	步骤8
获胜但成本远高于基线	范围与成本	步骤7

只有当对比结论明确时，verdict才是可衡量的结果：

adapt.mjs

要求无报错测试、无轨迹不匹配、总结一致，才会报告通过或回归。在将记录视为测试力度问题前，请先确认这一点。

Step 3: Rule out harness and reliability causes

步骤3：排除框架与可靠性问题

See references/eval-triage.md for the full catalogue. The recurring ones:

A spec declaring both
```
config:
```
and
```
defaults:
```
is rejected by vally, the job still exits 0, and the PR comment blames "transient infrastructure". Merge them into one
```
defaults:
```
block.
An errored trial is not automatically a fixture problem — judge-side auth and
```
session.idle
```
failures look identical from the verdict and need harness fixes, not SDK pins.
```
expect_tools: [bash]
```
on an advisory question forces a restore or build and turns an answer into a timeout with no quality gain.
Genuine code-generation stimuli need roughly 360s; a timeout yields empty output, which fails every grader and hides the real quality signal.
Unmatched trajectories, an errored trial, or a summary that disagrees make the comparison inconclusive: the remaining matched trials are biased, so the record is not a measured null and must not be read as a power or content problem.

完整问题列表请见references/eval-triage.md。常见问题包括：

同时声明
```
config:
```
和
```
defaults:
```
的规格文件会被vally拒绝，但作业仍会以0状态退出，PR评论会将问题归咎于“临时基础设施故障”。需将两者合并为一个
```
defaults:
```
块。
报错的测试并不一定是夹具问题——评判端的认证故障和
```
session.idle
```
故障从verdict看表现相同，需要修复框架而非SDK依赖。
在咨询类问题中设置
```
expect_tools: [bash]
```
会强制执行恢复或构建操作，导致回答超时且无质量提升。
真正的代码生成测试用例大约需要360秒；超时会导致空输出，进而使所有评分器失败，掩盖真实的质量信号。
轨迹不匹配、测试报错或总结不一致会使对比无结论：剩余匹配的测试存在偏差，因此记录不能视为可衡量的无效结果，也不能被当作测试力度或内容问题处理。

Step 4: Verify the fixtures before touching the skill

步骤4：修改技能前先验证测试夹具

Run

python eng/eval-quality/check_eval_quality.py

— it blocks ten defect classes that each already cost a real result here. Then confirm by hand:

every fixture behaves as its stimulus assumes — a fixture meant to be healthy builds, and one meant to be broken fails for the exact reason the stimulus is about and no other;
every referenced fixture is in the git index (
```
git ls-files
```
), not merely on disk —
```
.gitignore
```
has silently swallowed committed coverage fixtures;
a fixture never states the same fact in two places that disagree — a Cobertura report whose declared
```
line-rate
```
, summary totals and
```
<line>
```
elements differ is the canonical case — or the two arms legitimately read different truths.

运行

python eng/eval-quality/check_eval_quality.py

——该工具会阻止10类已导致过实际问题的缺陷。然后手动确认：

每个夹具的行为符合测试用例的假设——预期正常的夹具能构建成功，预期故障的夹具会因测试用例指定的原因失效，且无其他失效原因；
所有引用的夹具都在git索引中（使用
```
git ls-files
```
查看），而非仅存在于磁盘上——
```
.gitignore
```
曾静默忽略过已提交的覆盖率夹具；
夹具不会在两处陈述相互矛盾的事实——典型案例是Cobertura报告中声明的
```
line-rate
```
、汇总总数和
```
<line>
```
元素不一致——或者两组确实读取的是不同的真实数据。

Step 5: Check whether the eval could ever have passed

步骤5：检查评估是否有可能通过

The gate has two independent bars, and confusing them is the usual misdiagnosis:

Counted trials ≥ 5 (
```
trials = stimuli × runs
```
). Below that the verdict is reported
```
underpowered
```
— never a pass, never a regression.
The sign test must reach p ≤ 0.05 over the discordant (non-tie) trials. Ties are not discarded silently; they hold the discordant count down.

discordant trials	records that pass	p
≤ 4	none, however good the skill	≥ 0.0625
5–7	zero losses only (5W/0L)	0.031
8	one loss survivable (7W/1L)	0.035

So at exactly 5 counted trials a single tie is fatal — it leaves 4 discordant. At 6 counted trials one tie is survivable (5W/1T/0L); at 7, up to two are (5W/2T/0L). A loss is not.

So a positive record with a failing verdict is a power problem, not a content problem. Fix it by adding discriminating stimuli (cross-task evidence) rather than raising

runs

(repetition only) — except where each stimulus drives an expensive pipeline. Record the reasoning in a comment above

defaults:

, as

tests/dotnet-test/grade-tests/eval.yaml

does.

评估标准有两个独立条件，混淆这两个条件是常见的误判原因：

有效测试次数 ≥ 5（
```
trials = stimuli × runs
```
）。低于此数时，verdict会显示
```
underpowered
```
（测试力度不足）——既不会通过，也不会判定为回归。
**符号检验必须在不一致（非平局）的测试中达到p ≤ 0.05。平局不会被自动忽略；它们会减少不一致测试的数量。

不一致测试次数	可通过的记录	p值
≤ 4	无，无论技能表现多好	≥ 0.0625
5–7	仅零失败（5胜/0负）	0.031
8	可容忍1次失败（7胜/1负）	0.035

因此，当有效测试次数恰好为5时，一次平局就会导致失败——此时不一致测试次数为4。当有效测试次数为6时，一次平局是可接受的（5胜/1平/0负）；当次数为7时，最多可接受两次平局（5胜/2平/0负），但失败是不可接受的。

因此，记录为正但verdict失败的情况是测试力度问题，而非内容问题。修复方法是添加有区分度的测试用例（跨任务证据），而非增加

runs

（仅重复测试）——除非每个测试用例都会触发昂贵的流水线。请在

defaults:

上方的注释中记录推理过程，参考

tests/dotnet-test/grade-tests/eval.yaml

的写法。

Step 6: Check whether the two arms differ at all

步骤6：检查两组是否存在差异

An eval that compares the skill against itself measures judge noise:

A dormancy guard (
```
expect_activation: false
```
) must not also set
```
constraints.reject_skills
```
. That makes the skilled arm skill-free, i.e. identical to baseline. Across four evals the same guard scored −0.4, +0.4, +0.4 and 0, twice costing a skill its pass.
A skill with
```
disable-model-invocation: true
```
cannot self-activate, so an eval graded on activation compares two identical arms. Cover it through a consumer skill, or grade the answer content instead, as
```
tests/dotnet-test/filter-syntax/eval.yaml
```
and
```
tests/dotnet-test/platform-detection/eval.yaml
```
do.
A grader whose
```
config
```
is missing its required key enforces nothing, so the stimulus has one fewer assertion than it appears to.

将技能与自身对比的评估仅能衡量评判噪声：

休眠防护（
```
expect_activation: false
```
）绝对不能同时设置
```
constraints.reject_skills
```
。这会使技能组变为无技能状态，即与基线组完全相同。在四次评估中，同一防护的得分分别为-0.4、+0.4、+0.4和0，两次导致技能未通过。
设置
```
disable-model-invocation: true
```
的技能无法自激活，因此基于激活情况评分的评估会对比两个完全相同的组。请通过消费型技能覆盖此类场景，或改为基于回答内容评分，参考
```
tests/dotnet-test/filter-syntax/eval.yaml
```
和
```
tests/dotnet-test/platform-detection/eval.yaml
```
的写法。
缺少必填配置项的评分器无法执行任何校验，因此测试用例的断言数量会比看起来少一个。

Step 7: Fix skill content against the losing trial

步骤7：针对失败测试修复技能内容

Only now change the skill. Apply the patterns in references/writing-for-baseline-delta.md; the ones that most often flip a loss:

Replace reference prose the model already knows with decisions it would otherwise get wrong.
Add stop-conditions so a strong skill does not over-apply — but do not over-correct into answering more narrowly than the baseline did.
Scale output structure to input size; a dashboard for an 8-test suite loses to a direct answer.
Require truthful validation reporting; claiming "Build succeeded" after a failed restore is an automatic loss.
Verify load-bearing API claims by compiling or probing, not by reading source.
For cost regressions, gate rare or expensive paths behind
```
references/
```
reads and size any orchestration to the user's scope.

只有到这一步才可以修改技能。应用references/writing-for-baseline-delta.md中的模式，其中最常能扭转失败结果的模式包括：

替换模型已熟知的参考文本，改为明确模型原本会出错的决策点。
添加停止条件，避免优秀技能过度应用——但不要过度修正，导致回答范围比基线更窄。
根据输入规模调整输出结构；针对8个测试用例的套件，直接回答比生成仪表盘更优。
要求真实的验证报告；在恢复失败后声称“构建成功”会直接导致失败。
通过编译或探测验证关键API声明，而非仅读取源代码。
针对成本回归问题，将罕见或昂贵的路径限制在
```
references/
```
读取操作之后，并根据用户范围调整编排规模。

Step 8: Fix activation

步骤8：修复激活问题

Activation failures are frontmatter and routing failures, not body failures. See references/eval-triage.md. Summary:

Failure	Fix
Not activated in any arm	Put the user's own words in `description` : symptoms, error codes, artifact names, quoted requests
A sibling skill wins the prompt	Claim the exact ambiguous words in `description` , and add matching exclusions on both siblings
Model answers with no skill at all	Raise the stakes in the description, de-crowd the plugin menu, verify with the plugin arm
Boundary excludes real scenarios	Re-read every "do not use for" clause against every eval prompt and real workflow phase
Description at the 1,024-char ceiling	Cut restated body content, not trigger phrases; check the plugin menu budget too

激活失败是前置描述和路由问题，而非内容问题。详情请见references/eval-triage.md。总结如下：

失败类型	修复方案
在任何组中都未激活	将用户的原话加入 `description` ：症状、错误代码、产物名称、引用的请求内容
兄弟技能抢占了提示	在 `description` 中声明明确的模糊词汇，并在两个兄弟技能中添加匹配的排除规则
模型直接回答，未调用任何技能	在描述中提高优先级，精简插件菜单，并通过插件组验证
边界规则排除了真实场景	对照每个评估提示和真实工作流阶段，重新阅读所有“请勿用于”条款
描述达到1024字符上限	删除重复的内容文本，而非触发短语；同时检查插件菜单预算

Step 9: Re-validate

步骤9：重新验证

bash

dotnet run --project eng/skill-validator/src/SkillValidator.csproj -- check --plugin ./plugins/<plugin>
python eng/eval-quality/check_eval_quality.py
./eng/run-skill-evals.sh <plugin> <skill>

Then request the official run by submitting a PR review containing

/evaluate

(Files changed → Review changes), which binds the run to the reviewed commit. Before declaring a regression on the result, confirm the skill payload actually changed — reruns on byte-identical content have shifted 7W/2T/2L to 4W/5T/2L.

bash

dotnet run --project eng/skill-validator/src/SkillValidator.csproj -- check --plugin ./plugins/<plugin>
python eng/eval-quality/check_eval_quality.py
./eng/run-skill-evals.sh <plugin> <skill>

然后提交包含

/evaluate

的PR评审（Files changed → Review changes）来请求官方运行，该运行会绑定到评审的提交记录。在根据结果判定回归之前，请确认技能 payload 确实已变更——对字节完全相同的内容重新运行，结果曾从7胜/2平/2负变为4胜/5平/2负。

Validation

验证清单

For a content fix, a losing trial and the judge's stated reason are quoted in the PR description.
The failure was classified before any content was edited.

check_eval_quality.py

and

skill-validator check

both pass.

Trial count clears the power bar for the observed tie rate, not just the floor of 5.
Isolated and plugin activation are both reported.
The PR body records root cause, fix, and validation so the lesson is reusable.

若为内容修复，PR描述中需引用失败测试案例及评判方给出的理由。
在编辑内容前已完成失败类型分类。

check_eval_quality.py

和

skill-validator check

均通过。

测试次数根据观察到的平局率满足测试力度要求，而非仅达到最低的5次。
已报告独立组和插件组的激活状态。
PR正文中记录了根本原因、修复方案和验证过程，以便经验复用。

Common Pitfalls

常见误区

Pitfall	Solution
Rewriting skill prose in response to an underpowered verdict	Underpowered means too few discordant trials; add discriminating stimuli instead
Adding `defaults: runs:` to a spec that already has `config:`	Merge into a single `defaults:` block; vally rejects specs with both
Padding `runs` to clear the trial floor	Five repeats of one stimulus measure one task; add stimuli
Treating an errored trial as fixture nondeterminism	Read the stderr first; judge-side auth failures need harness fixes
Fixing a "wrong" answer that the fixture actually made wrong	Check fixture self-consistency before blaming the response
Strengthening a skill nobody uses and nothing passes	Weak eval signal plus thin telemetry is a valid retirement case
Landing a fix without re-running	Verify the invoked payload contains the fix; judge noise is real

误区	解决方案
针对测试力度不足的verdict重写技能文本	测试力度不足意味着不一致测试次数太少；应添加有区分度的测试用例
向已有 `config:` 的规格文件添加 `defaults: runs:`	合并为单个 `defaults:` 块；vally会拒绝同时包含两者的规格文件
增加 `runs` 以达到测试次数下限	重复一个测试用例五次仅能衡量一项任务；应添加新的测试用例
将报错测试视为夹具非确定性问题	先查看stderr；评判端的认证故障需要修复框架
修复由夹具本身导致的“错误”回答	在指责响应之前，先检查夹具的自一致性
强化无人使用且无通过记录的技能	薄弱的评估信号加上有限的遥测数据是合理的淘汰理由
未重新运行就发布修复	验证调用的payload包含修复内容；评判噪声是真实存在的

References

参考文档

references/writing-for-baseline-delta.md — content patterns that beat the unskilled model
references/eval-triage.md — symptom, cause and fix catalogue with PR citations
eng/eval-quality/README.md — the ten structural gate checks and why each exists
eng/vally-adapter/InvestigatingResults.md — downloading artifacts and reading
```
results.json
```
. This is the current guide; the similarly-named
```
eng/skill-validator/src/docs/InvestigatingResults.md
```
documents the retired
```
skill-validator evaluate
```
schema and does not describe today's results.

references/writing-for-baseline-delta.md——优于无技能模型的内容编写模式
references/eval-triage.md——包含PR引用的症状、原因和修复清单
eng/eval-quality/README.md——10项结构性检查及各自的存在理由
eng/vally-adapter/InvestigatingResults.md——下载产物及读取
```
results.json
```
的指南。这是当前的有效指南；名称相似的
```
eng/skill-validator/src/docs/InvestigatingResults.md
```
记录的是已弃用的
```
skill-validator evaluate
```
schema，无法描述当前的结果。