bmad-eval

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Skill Eval Runner

Skill 评测运行器

You run a skill's evals and report what they say. Cite specific findings, flag evals that pass for trivial reasons, and never widen a tolerance to make a run look like it succeeded. No model or harness is named in this skill: the harness a project's evals run through is recorded in this skill's customization at the first run.

你负责运行 Skill 的评测并报告结果。引用具体发现，标记因琐碎原因而通过的评测，绝不通过放宽容差让运行结果看起来像是成功。此 Skill 未指定任何 model 或 harness：项目评测所使用的 harness 会在首次运行时记录到此 Skill 的 customization 中。

The four modes

四种模式

Mode	Question it answers	Script / reference
baseline	Does the skill beat the bare model on the same input?	`references/eval-format.md` , `scripts/run_evals.py`
variant	Does a section earn its place, or does a stripped version do as well?	`references/eval-format.md` , `scripts/run_evals.py`
quality	Does the output meet the named rubric?	`references/grader.md` , `references/eval-format.md`
trigger	Does the description fire on the right queries and stay quiet on the rest?	`references/harness.md` , `scripts/run_triggers.py`

Baseline runs every case with the skill staged and with nothing staged. Variant runs the skill against the one at

--variant-path

. Quality grades one config against its rubric. Trigger measures real firing;

references/description-optimization.md

improves the description across rounds.

模式	回答的问题	脚本 / 参考资料
baseline	Skill 在相同输入下是否优于裸模型？	`references/eval-format.md` 、 `scripts/run_evals.py`
variant	某个章节是否真正发挥作用，还是精简版效果同样好？	`references/eval-format.md` 、 `scripts/run_evals.py`
quality	输出是否符合指定的 rubric？	`references/grader.md` 、 `references/eval-format.md`
trigger	description 是否会在正确的查询上触发，并在其他查询上保持安静？	`references/harness.md` 、 `scripts/run_triggers.py`

Baseline 会将 Skill 加载后以及不加载任何内容时分别运行每个 case。Variant 会将 Skill 与

--variant-path

指定的版本进行对比。Quality 会依据其 rubric 为一个 config 评分。Trigger 衡量真实触发情况；

references/description-optimization.md

可通过多轮优化改进 description。

Case format

Case 格式

A case is

input + rubric + optional state_prefix + optional fixture files

; the

state_prefix

is a bracketed prime prepended to the input that places the skill mid-workflow in one shot. Cases live in

<skill>/evals/cases.json

, trigger queries as

{query, should_trigger}

<skill>/evals/triggers.json

. The full shape and the strong-versus-weak expectation taxonomy are in

references/eval-format.md

. A skill-creator

evals.json

converts with

uv run {skill-root}/scripts/convert_cases.py <evals.json> --output <cases.json>

(

--to skill-creator

reverses it).

一个 case 由

input + rubric + optional state_prefix + optional fixture files

构成；

state_prefix

是一个带括号的 prime，会前置到 input 上，从而一次性将 Skill 置于工作流中。Case 位于

<skill>/evals/cases.json

，trigger query 使用

{query, should_trigger}

格式并位于

<skill>/evals/triggers.json

。完整的结构以及强预期与弱预期分类体系见

references/eval-format.md

。skill-creator 的

evals.json

可通过

uv run {skill-root}/scripts/convert_cases.py <evals.json> --output <cases.json>

转换（

--to skill-creator

可执行反向转换）。

On activation

激活时执行的操作

Read config with

uv run {project-root}/_bmad/scripts/resolve_config.py --project-root {project-root} --key core.output_folder --key core.communication_language --key core.user_name

and customization with

uv run {project-root}/_bmad/scripts/resolve_customization.py --skill {skill-root} --project-root {project-root} --key workflow

. Absent keys are fine.

{reports_folder}

{workflow.reports_folder}

with

{output_folder}

filled in; set

{communication_language}

and

{user_name}

from what comes back.

Verify
```
<skill-path>/SKILL.md
```
exists; halt with a clear error if not.
When
```
workflow.harness.command
```
came back empty, work out the harness facts for the CLI you are running in per
```
references/harness.md
```
, prove them on one case, and record them by invoking the
```
bmad-customize
```
skill (install:
```
npx skills add bmad-code-org/BMAD-METHOD --skill bmad-customize
```
); the runner reads them from there. When nobody is at the keyboard, stop and show the table to record instead.
Find the cases file: the one the user named, then
```
<skill-path>/evals/
```
, then
```
<project-root>/evals/<skill-name>/
```
, then anywhere under
```
<project-root>/evals/
```
. If nothing is found, halt and say so; the runner does not invent cases.
Confirm the run summary unless the user asked for a non-interactive run, then execute. The summary names the skill, cases, modes and output dir, shows the harness command, and says that it runs with permission prompts off and is not contained unless the command wraps it in a sandbox. A non-interactive run needs the harness already recorded.

使用

uv run {project-root}/_bmad/scripts/resolve_config.py --project-root {project-root} --key core.output_folder --key core.communication_language --key core.user_name

读取 config，并使用

uv run {project-root}/_bmad/scripts/resolve_customization.py --skill {skill-root} --project-root {project-root} --key workflow

读取 customization。缺少 key 是允许的。

{reports_folder}

为将

{output_folder}

代入后的

{workflow.reports_folder}

；根据返回结果设置

{communication_language}

和

{user_name}

。

确认
```
<skill-path>/SKILL.md
```
存在；如果不存在，则停止并给出清晰的错误信息。
如果返回的
```
workflow.harness.command
```
为空，根据
```
references/harness.md
```
推导当前 CLI 的 harness 事实，在一个 case 上验证这些事实，然后调用
```
bmad-customize
```
skill 将其记录下来（安装：
```
npx skills add bmad-code-org/BMAD-METHOD --skill bmad-customize
```
）；runner 会从该处读取这些信息。如果当前没有人工操作，则停止并展示待记录的表格。
查找 case 文件：先查找用户指定的文件，然后查找
```
<skill-path>/evals/
```
，接着查找
```
<project-root>/evals/<skill-name>/
```
，最后查找
```
<project-root>/evals/
```
下的任何位置。如果没有找到，则停止并说明情况；runner 不会自行创建 case。
除非用户要求 non-interactive run，否则先确认运行摘要，然后执行。摘要应列明 Skill、case、模式和输出目录，展示 harness command，并说明运行时关闭权限提示；除非命令本身使用 sandbox 包裹，否则运行不受隔离限制。Non-interactive run 要求 harness 已被记录。

Run execution

运行执行

The user asks for a run in

<mode>

mode on

<skill>

(the directory holding

SKILL.md

), with a variant skill for variant mode. The project root is the first ancestor of the skill holding

_bmad/

.git/

; the output dir is

{reports_folder}

when BMad is set up, else

~/bmad-evals/

. Every run is its own timestamped folder there, so runs never collide. Each case runs in a clean room, a temporary folder outside the project with a copy of the skill staged into its workspace and an environment built from scratch, so host config, prior runs, the project's instruction files and its installed skills cannot bias the result. The runner reads the harness from customization itself; in a project without BMad, pass

--harness <json>

with the same keys.

Baseline, variant and quality:

uv run {skill-root}/scripts/run_evals.py \
  --cases <cases-file> --skill-path <skill> --output-dir <dir> \
  --mode quality|baseline|variant [--variant-path <skill>] \
  --label <skill-name>-<mode> [--runs N]

--runs

defaults to 1 for a quick look; a result counts as settled only at 3 or more. Case subsets, timeouts and workers are in the script docstrings. Exit 0 is a complete run; 1 means a case errored or timed out or the command was not found, 3 that no harness is recorded; a trigger query failing its threshold is a result, not an error.

Trigger:

uv run {skill-root}/scripts/run_triggers.py \
  --skill-path <skill> --queries <queries-file> --output-dir <dir> \
  [--runs-per-query N]

For quality, spawn the grader in

references/grader.md

per case with its rubric, transcript path, artifacts dir (the case's

cwd/

) and

grading_path

<case-folder>/grading.json

. Relay its rubric feedback. If a grader errors, mark the case

grading_error

, never a default verdict.

When

--runs

is greater than one,

uv run {skill-root}/scripts/aggregate_benchmark.py --baseline <run-dir>/<config-a> --variant <run-dir>/<config-b>

gives mean, sample standard deviation, min, max and delta (

--runs <run-dir>/<config>

for one config).

When a run comes back weak and the user wants the skill improved from it, follow

references/self-improvement.md

用户会要求以

<mode>

模式对

<skill>

（包含

SKILL.md

的目录）运行一次，并可能为 variant mode 提供 variant skill。项目根目录是 Skill 路径中第一个包含

_bmad/

或

.git/

的祖先目录；设置 BMad 时，输出目录为

{reports_folder}

，否则为

~/bmad-evals/

。每次运行都会在该目录下创建独立的带时间戳文件夹，因此运行之间不会冲突。每个 case 都在 clean room 中运行，即项目外的临时文件夹，其中将 Skill 的副本加载到 workspace，并从头构建环境，从而避免 host config、先前运行、项目的 instruction file 和已安装 Skill 对结果造成影响。Runner 自行从 customization 读取 harness；对于未设置 BMad 的项目，使用相同的 key 传入

--harness <json>

。

Baseline、variant 和 quality：

uv run {skill-root}/scripts/run_evals.py \
  --cases <cases-file> --skill-path <skill> --output-dir <dir> \
  --mode quality|baseline|variant [--variant-path <skill>] \
  --label <skill-name>-<mode> [--runs N]

对于快速查看，

--runs

默认为 1；只有运行次数达到 3 次或更多时，结果才视为 settled。Case 子集、超时和 worker 配置记录在脚本 docstring 中。退出码 0 表示完整运行；1 表示某个 case 出错、超时或未找到命令；3 表示未记录 harness。Trigger query 未达到阈值属于结果，而不是错误。

Trigger：

uv run {skill-root}/scripts/run_triggers.py \
  --skill-path <skill> --queries <queries-file> --output-dir <dir> \
  [--runs-per-query N]

对于 quality，根据

references/grader.md

为每个 case 启动 grader，并向其提供 rubric、transcript 路径、artifacts 目录（该 case 的

cwd/

）以及

<case-folder>/grading.json

的

grading_path

。转发其 rubric 反馈。如果 grader 出错，将该 case 标记为

grading_error

，绝不能给出默认 verdict。

当

--runs

大于 1 时，

uv run {skill-root}/scripts/aggregate_benchmark.py --baseline <run-dir>/<config-a> --variant <run-dir>/<config-b>

会给出均值、样本标准差、最小值、最大值和差值（对于单个 config，使用

--runs <run-dir>/<config>

）。

当运行结果较弱且用户希望据此改进 Skill 时，遵循

references/self-improvement.md

。

Artifacts

产物

Each run is a dated folder under the output dir, with

run.json

naming the harness command; each case folder, and each trigger attempt, holds its prompt, the transcript (what the harness printed),

stderr.txt

cwd/

(the workspace after the run),

timing.json

(written the moment the invocation ends) and

grading.json

when quality ran. Never delete, overwrite or rotate a run folder. Tell the user where it is when you finish.

每次运行都会在输出目录下创建一个带日期的文件夹，其中

run.json

记录 harness command；每个 case 文件夹以及每次 trigger attempt 都包含其 prompt、transcript（harness 输出的内容）、

stderr.txt

、

cwd/

（运行后的 workspace）、

timing.json

（在调用结束时立即写入），以及运行 quality 时生成的

grading.json

。绝不删除、覆盖或轮转运行文件夹。完成后告知用户其所在位置。

Outcomes

结果判定

The run reflects the skill in a clean working directory, not the host shell.
Failures cite specific expectations with evidence; a superficial pass is flagged.
A result from fewer than 3 runs per case (or per trigger query) is reported as a quick look, never as settled.
A baseline the skill no longer wins points to retiring the skill, not patching it.

运行结果反映的是 Skill 在干净工作目录中的表现，而非 host shell 中的表现。
失败项需要引用具体预期和证据；表面上的通过会被标记。
每个 case（或每个 trigger query）的运行次数少于 3 次时，结果仅报告为 quick look，绝不视为 settled。
如果 Skill 在 baseline 中不再胜出，就应考虑淘汰该 Skill，而不是继续修补。