skill-upper

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

use-skill-up-cli

使用skill-up CLI

Help the user evaluate and evolve Agent Skills through the

skill-up

CLI.

Manual: https://alibaba.github.io/skill-up/

帮助用户通过

skill-up

CLI评测和优化Agent Skill。

手册：https://alibaba.github.io/skill-up/

Language Policy

语言规范

Default to English when responding to the user. If the user writes in Chinese (or any other language), switch to that language and stay consistent with the user's input throughout the session.

Detection rules (highest priority first):

The user explicitly specifies a language in the current message (e.g. "answer in English" / "用中文回答") → follow the user's instruction.
The natural language used in the user's current message → match it.
None of the above → use English (default).

Regardless of the response language, technical identifiers in this SKILL — CLI commands,

eval.yaml

case.yaml

field names, report field names, etc. — MUST stay in their original English form. Do not translate them.

默认使用英文回复用户。若用户使用中文（或其他语言）提问，则切换为对应语言，并在整个会话中保持与用户输入语言一致。

检测规则（优先级从高到低）：

用户在当前消息中明确指定语言（例如“answer in English” / “用中文回答”）→ 遵循用户指示。
用户当前消息使用的自然语言→ 匹配该语言。
以上均不满足→ 使用英文（默认）。

无论回复语言如何，本SKILL中的技术标识符——CLI命令、

eval.yaml

case.yaml

字段名、报告字段名等——必须保持原始英文形式，不得翻译。

Language Rules for Generated Artifacts

生成产物的语言规则

When creating or editing

eval.yaml

case.yaml

, grading scripts, README snippets, final replies, or any other user-visible artifact, treat the language of the user's current message as the output language for this turn:

If the user asks in Chinese, write the final response and all generated natural-language content in Chinese, including YAML comments,
```
title
```
,
```
description
```
,
```
input.prompt
```
,
```
expect
```
keywords, and
```
judge.criteria
```
.
If the user asks in English, write the final response and all generated natural-language content in English, including YAML comments,
```
title
```
,
```
description
```
,
```
input.prompt
```
,
```
expect
```
keywords, and
```
judge.criteria
```
; do not leave Chinese or CJK characters in generated case files.
If the target Skill itself is written in Chinese but the user asks in English, translate the Skill's functional intent into English test prompts and assertions instead of copying Chinese prose from the target Skill or templates.
In an English context, deterministic keywords in
```
rule_based
```
cases, including
```
expect.must_contain
```
and
```
judge.success.output_contains
```
, must also be English keywords. Translate terms such as
```
资源泄漏
```
,
```
关闭
```
, and
```
异常处理
```
into
```
resource leak
```
,
```
close
```
, and
```
exception handling
```
; do not write bilingual parentheticals like
```
"资源" (resources)
```
.
Keep technical identifiers unchanged, such as
```
schema_version
```
,
```
environment.type
```
,
```
engine.name
```
,
```
rule_based
```
,
```
agent_judge
```
,
```
script_path
```
, file paths, and commands.
Treat
```
assets/*.tmpl
```
as structural references only. Rewrite placeholder prose and comments into the current output language; in an English context, translate or remove every Chinese comment and Chinese placeholder before writing generated files.
In an English context, after generating all files but BEFORE submitting the final reply, you MUST perform a CJK self-check: open every
```
evals/cases/*.yaml
```
and
```
evals/eval.yaml
```
and scan for CJK characters (Unicode ranges
```
\u4e00-\u9fff\u3400-\u4dbf\uf900-\ufaff\u3000-\u303f\uff00-\uffef
```
), including but not limited to
```
title
```
,
```
description
```
,
```
input.prompt
```
,
```
expect
```
keywords,
```
judge.criteria
```
, and YAML comments. If any CJK character is found, replace it with an equivalent English expression before finishing the task. This step is mandatory and must not be skipped.

创建或编辑

eval.yaml

、

case.yaml

、评分脚本、README片段、最终回复或其他用户可见产物时，将用户当前消息的语言作为本轮输出语言：

若用户用中文提问，最终回复及所有生成的自然语言内容均使用中文，包括YAML注释、
```
title
```
、
```
description
```
、
```
input.prompt
```
、
```
expect
```
关键字和
```
judge.criteria
```
。
若用户用英文提问，最终回复及所有生成的自然语言内容均使用英文，包括YAML注释、
```
title
```
、
```
description
```
、
```
input.prompt
```
、
```
expect
```
关键字和
```
judge.criteria
```
；生成的用例文件中不得保留中文或CJK字符。
若目标Skill本身为中文但用户用英文提问，需将Skill的功能意图翻译成英文测试提示和断言，而非直接复制目标Skill或模板中的中文文本。
在英文场景下，
```
rule_based
```
用例中的确定性关键字（包括
```
expect.must_contain
```
和
```
judge.success.output_contains
```
）也必须为英文关键字。将“资源泄漏”“关闭”“异常处理”等术语翻译为
```
resource leak
```
、
```
close
```
和
```
exception handling
```
；不得使用“资源(resources)”这类双语括号形式。
技术标识符保持不变，例如
```
schema_version
```
、
```
environment.type
```
、
```
engine.name
```
、
```
rule_based
```
、
```
agent_judge
```
、
```
script_path
```
、文件路径和命令。
将
```
assets/*.tmpl
```
仅视为结构参考。将占位文本和注释重写为当前输出语言；在英文场景下，生成文件前需翻译或移除所有中文注释和中文占位符。
在英文场景下，生成所有文件后、提交最终回复前，必须执行CJK自检：打开所有
```
evals/cases/*.yaml
```
和
```
evals/eval.yaml
```
，扫描CJK字符（Unicode范围
```
\u4e00-\u9fff\u3400-\u4dbf\uf900-\ufaff\u3000-\u303f\uff00-\uffef
```
），包括但不限于
```
title
```
、
```
description
```
、
```
input.prompt
```
、
```
expect
```
关键字、
```
judge.criteria
```
和YAML注释。若发现任何CJK字符，需替换为等效英文表达后再完成任务。此步骤为必填项，不得跳过。

What is skill-up

什么是skill-up

skill-up

is an evaluation CLI for Agent Skill authors. It installs the Skill into a real Agent Engine (Claude Code, Codex, qodercli, etc.), spins up an execution environment for each case, runs the prompt, then grades the result via declared rules / LLM judges / custom scripts, and finally produces a report.

Typical layout:

my-skill/
  SKILL.md
  evals/
    eval.yaml
    cases/
      <case-id>.yaml
    fixtures/

skill-up

是面向Agent Skill开发者的评测CLI工具。它会将Skill安装到真实的Agent引擎（Claude Code、Codex、qodercli等）中，为每个用例启动执行环境，运行提示词，然后通过声明式规则/LLM评判器/自定义脚本对结果评分，最终生成报告。

典型目录结构：

my-skill/
  SKILL.md
  evals/
    eval.yaml
    cases/
      <case-id>.yaml
    fixtures/

When to trigger

触发场景

Use this skill in any of the following situations:

The user asks to "run / evaluate / verify / test this skill".
The user asks to "fix / improve / iterate / evolve this skill" from eval failures.
The user wants to "add evals, test cases, or regression cases to a skill".
The user wants to edit
```
eval.yaml
```
/
```
case.yaml
```
, or asks you to choose an appropriate
```
judge
```
type.

The user mentions

skill-up run/validate/list-cases/report/import/init

The user wants to migrate from Anthropic
```
evals.json
```
to skill-up.
The current working directory contains
```
evals/eval.yaml
```
or
```
evals/evals.json
```
and the user wants to run it.

在以下任一场景中使用本Skill：

用户要求“运行/评测/验证/测试这个Skill”。
用户要求从评测失败中“修复/改进/迭代/升级这个Skill”。
用户希望“为Skill添加评测、测试用例或回归用例”。
用户希望编辑
```
eval.yaml
```
/
```
case.yaml
```
，或询问如何选择合适的
```
judge
```
类型。

用户提及

skill-up run/validate/list-cases/report/import/init

命令。

用户希望从Anthropic
```
evals.json
```
迁移到skill-up。
当前工作目录包含
```
evals/eval.yaml
```
或
```
evals/evals.json
```
且用户希望运行它。

Main flow (follow this order strictly)

主流程（严格遵循此顺序）

Step 0: Make sure skill-up is installed

步骤0：确保skill-up已安装

Before doing anything, verify

skill-up

is available:

bash

command -v skill-up && skill-up --version

If a version is printed, continue. If you see

command not found

, on macOS / Linux:

bash

curl -fsSL https://raw.githubusercontent.com/alibaba/skill-up/main/install.sh | bash

export SKILL_UP_VERSION=v0.1.0
curl -fsSL https://raw.githubusercontent.com/alibaba/skill-up/main/install.sh | bash

export INSTALL_DIR="$HOME/bin"
curl -fsSL https://raw.githubusercontent.com/alibaba/skill-up/main/install.sh | bash

Platform:
skill-up
currently supports macOS / Linux only; Windows is not supported.

After installing, run

skill-up --version

again. If the command is still missing, add

~/.local/bin

PATH

More details:

references/install.md

执行任何操作前，先验证

skill-up

是否可用：

bash

command -v skill-up && skill-up --version

若打印出版本信息，继续操作。若显示

command not found

，在macOS/Linux系统中执行：

bash

curl -fsSL https://raw.githubusercontent.com/alibaba/skill-up/main/install.sh | bash

export SKILL_UP_VERSION=v0.1.0
curl -fsSL https://raw.githubusercontent.com/alibaba/skill-up/main/install.sh | bash

export INSTALL_DIR="$HOME/bin"
curl -fsSL https://raw.githubusercontent.com/alibaba/skill-up/main/install.sh | bash

平台支持：
skill-up
目前仅支持macOS/Linux；暂不支持Windows。

安装完成后，再次运行

skill-up --version

。若仍找不到命令，需将

~/.local/bin

添加到

PATH

中。

更多细节：

references/install.md

。

Step 0.5 (optional): User config and telemetry

步骤0.5（可选）：用户配置与遥测

For OTLP defaults,

runtime_kwargs

(e.g. OpenSandbox

base_url

), etc.:

bash

skill-up init
skill-up init --local
skill-up init --print
skill-up init --force

Precedence (low → high): embedded empty defaults < user config < project

.skill-up.yaml

--config

SKILL_UP_CONFIG

can point at the user config file (env var name is historical). See the upstream README "User config".

针对OTLP默认值、

runtime_kwargs

（例如OpenSandbox

base_url

）等配置：

bash

skill-up init
skill-up init --local
skill-up init --print
skill-up init --force

优先级（从低到高）：内置空默认值 < 用户配置 < 项目

.skill-up.yaml

--config

参数。

SKILL_UP_CONFIG

可指向用户配置文件（环境变量名称为历史遗留）。详见上游README中的“User config”部分。

Step 1: Locate the target Skill

步骤1：定位目标Skill

Identify the root directory of the target Skill (the directory containing
```
SKILL.md
```
). Search in this priority: user path → nearest
```
SKILL.md
```
upward from CWD → recently viewed files.
Read the target
```
SKILL.md
```
for scope, triggers, and dependencies. If the Skill is Chinese but the user writes in English, translate capabilities into English for prompts and assertions.

Check

evals/

```
evals/eval.yaml
```
exists → Step 4 (optionally Step 3).

Only

evals/evals.json

→

references/migrate-anthropic.md

(

skill-up run --auto

skill-up import

Nothing → Step 2.

确定目标Skill的根目录（包含
```
SKILL.md
```
的目录）。按以下优先级搜索：用户指定路径 → 从当前工作目录向上查找最近的
```
SKILL.md
```
→ 最近查看的文件。
读取目标
```
SKILL.md
```
以了解范围、触发条件和依赖关系。若Skill为中文但用户用英文提问，需将功能翻译成英文用于提示词和断言。
检查
```
evals/
```
目录：
- 存在
```
evals/eval.yaml
```
  → 执行步骤4（可选择先执行步骤3）。
- 仅存在
```
evals/evals.json
```
  → 参考
```
references/migrate-anthropic.md
```
  （使用
```
skill-up run --auto
```
  或
```
skill-up import
```
  ）。
- 无任何文件 → 执行步骤2。

Step 2: Scaffold the evals (only when none exist)

步骤2：搭建评测框架（仅当无现有评测时）

Copy

assets/eval.yaml.tmpl

<skill-root>/evals/eval.yaml

Copy

assets/case.yaml.tmpl

<skill-root>/evals/cases/<case-id>.yaml

Adapt language per "Language Rules for Generated Artifacts". In an English context, it is prohibited to copy Chinese placeholder text from the templates into generated files — all prose must be rewritten in English. The Chinese in the templates is for structural reference only, not to be carried over.

Selection guidelines:

```
environment.type
```
: use
```
none
```
for pure-text Skills; use
```
opensandbox
```
when you need a remote sandbox (set
```
OPENSANDBOX_API_KEY
```
, put non-secrets in
```
environment.kwargs
```
).

engine.name

engine.model

: default

claude_code

;

model

is optional. For

qodercli

, often omit

model

judge.type

rule_based

(preferred),

script

agent_judge

(expensive) — see

references/judge-types.md

Case ID = filename without
```
.yaml
```
; prompts should exercise real Skill value.

See

references/eval-yaml.md

and

references/case-yaml.md

将

assets/eval.yaml.tmpl

复制到

<skill-root>/evals/eval.yaml

。

将

assets/case.yaml.tmpl

复制到

<skill-root>/evals/cases/<case-id>.yaml

。

根据“生成产物的语言规则”调整语言。在英文场景下，禁止将模板中的中文占位文本复制到生成文件中——所有文本必须重写为英文。模板中的中文仅用于结构参考，不得沿用。

选择指南：

```
environment.type
```
：纯文本Skill使用
```
none
```
；需要远程沙箱时使用
```
opensandbox
```
（需设置
```
OPENSANDBOX_API_KEY
```
，将非敏感信息放入
```
environment.kwargs
```
）。
```
engine.name
```
+
```
engine.model
```
：默认
```
claude_code
```
；
```
model
```
为可选参数。对于
```
qodercli
```
，通常省略
```
model
```
。
```
judge.type
```
：优先选择
```
rule_based
```
，其次是
```
script
```
，最后是
```
agent_judge
```
（成本较高）——详见
```
references/judge-types.md
```
。
用例ID = 文件名（不含
```
.yaml
```
）；提示词应体现Skill的实际价值。

详见

references/eval-yaml.md

和

references/case-yaml.md

。

Step 3: Fill the gaps (when evals already exist)

步骤3：补充完善（当已有评测时）

```
skill-up list-cases <path>
```
Review
```
eval.yaml
```
and representative cases; avoid
```
agent_judge
```
abuse.
Add or edit YAML under
```
cases/
```
as needed.

```
skill-up list-cases <path>
```
检查
```
eval.yaml
```
和代表性用例；避免滥用
```
agent_judge
```
。
根据需要添加或编辑
```
cases/
```
下的YAML文件。

Step 4: Validate the configuration

步骤4：验证配置

bash

skill-up validate <skill-root>/evals/eval.yaml

Expect:

✓ eval.yaml is valid (loaded N case(s))

bash

skill-up validate <skill-root>/evals/eval.yaml

预期结果：

✓ eval.yaml is valid (loaded N case(s))

。

Step 5: Prepare credentials

步骤5：准备凭证

Priority:

--api-key

> env (

ANTHROPIC_API_KEY

OPENAI_API_KEY

QODER_PERSONAL_ACCESS_TOKEN

) >

~/.skill-up/credentials.yaml

bash

printenv | grep -E 'ANTHROPIC_API_KEY|OPENAI_API_KEY|QODER_PERSONAL_ACCESS_TOKEN'

If missing, stop and ask; do not write secrets into YAML without consent.

For

opensandbox

, also ensure

OPENSANDBOX_API_KEY

(and related env) as needed.

优先级：

--api-key

参数 > 环境变量（

ANTHROPIC_API_KEY

、

OPENAI_API_KEY

、

QODER_PERSONAL_ACCESS_TOKEN

） >

~/.skill-up/credentials.yaml

。

bash

printenv | grep -E 'ANTHROPIC_API_KEY|OPENAI_API_KEY|QODER_PERSONAL_ACCESS_TOKEN'

若缺失，停止操作并询问用户；未经同意不得将密钥写入YAML文件。

对于

opensandbox

，还需确保

OPENSANDBOX_API_KEY

（及相关环境变量）已配置。

Step 6: Run the evaluation

步骤6：运行评测

bash

skill-up run <skill-root>/evals/eval.yaml

Scenario	Command
Subset	`--include-case-name "basic-*"`
Exclude	`--exclude-case-name "*-flaky"`
HTML report	`--format html`
Engine override	`--engine codex --model openai/gpt-4`
Parallelism	`--parallelism 4` (1–256)
Anthropic JSON	`--auto`
Stability/flakiness sampling	`--iteration 3`
Auto-append after last iteration	`--iteration 0` (default behavior)
Verbose	`-v` , `-vv`

Exit

= all passed;

= failure or error — suitable for CI. When an explicit positive

--iteration N

runs more than one sample, inspect the terminal's simple current-command summary for lines like

case_a: 3 trials, 2 PASS, 1 FAIL -> flaky

bash

skill-up run <skill-root>/evals/eval.yaml

场景	命令
运行子集用例	`--include-case-name "basic-*"`
排除指定用例	`--exclude-case-name "*-flaky"`
生成HTML报告	`--format html`
覆盖引擎配置	`--engine codex --model openai/gpt-4`
并行运行	`--parallelism 4` （范围1–256）
兼容Anthropic JSON格式	`--auto`
稳定性/波动性抽样测试	`--iteration 3`
在最后一次迭代后自动追加	`--iteration 0` （默认行为）
Verbose模式	`-v` , `-vv`

退出码

表示全部通过；

表示失败或出错——适用于CI流程。当指定正整数

--iteration N

运行多次抽样时，查看终端中的命令摘要，例如

case_a: 3 trials, 2 PASS, 1 FAIL -> flaky

。

Step 7: Interpret the report

步骤7：解读报告

Artifacts under

<skill-root>/<skill-name>-workspace/iteration-N/

```
result.json
```
,
```
benchmark.json
```
, optional
```
report.html
```

<case-id>/with_skill/grading.json

outputs/

Summarize: pass rate and timing; for failures, case id, assertion

text

, and

evidence

; benchmark deltas if enabled; offer HTML path or

skill-up report result.json --format html

产物位于

<skill-root>/<skill-name>-workspace/iteration-N/

目录下：

```
result.json
```
、
```
benchmark.json
```
、可选
```
report.html
```

<case-id>/with_skill/grading.json

、

outputs/

总结内容：通过率和耗时；失败用例的ID、断言文本和证据；启用基准测试时的差异；提供HTML报告路径或

skill-up report result.json --format html

命令。

Step 8: Evolve the Skill when requested

步骤8：按需优化Skill

Only enter this loop when the user asks to fix, improve, iterate, or evolve the target Skill. If the user only asks to evaluate or report results, stop after Step 7 without modifying it.

Diagnose failures from
```
result.json
```
,
```
grading.json
```
, and output evidence.
Fix
```
SKILL.md
```
or supporting files when the Skill behavior is incorrect.
Add or refine eval cases when coverage is missing.
Do not weaken valid assertions merely to make a failure pass.
Rerun failed cases first, then run the full eval suite.
Continue until the evals pass or clearly report what remains blocked.

仅当用户要求修复、改进、迭代或升级目标Skill时进入此循环。若用户仅要求评测或报告结果，完成步骤7后停止操作，不得修改Skill。

从
```
result.json
```
、
```
grading.json
```
和输出证据中诊断失败原因。
若Skill行为不正确，修改
```
SKILL.md
```
或相关支持文件。
若覆盖范围不足，添加或优化评测用例。
不得为了让失败用例通过而削弱有效断言。
先重新运行失败用例，再运行完整评测套件。
持续操作直到评测通过，或明确报告剩余阻塞问题。

Command quick reference

命令速查

Command	Purpose
`skill-up validate <eval.yaml>`	Validate before `run` .
`skill-up list-cases <eval.yaml>`	List cases.
`skill-up run [eval.yaml]`	Run evals.
`skill-up run --auto`	Run from `evals/evals.json` .
`skill-up report <result.json> --format html`	Re-render reports.
`skill-up import <evals.json>`	Convert Anthropic format to YAML.
`skill-up init`	Write user-config template.
`skill-up debug judge <input.json>`	Debug judge.
`skill-up debug report <input.json>`	Debug report.

Full flags:

references/cli.md

命令	用途
`skill-up validate <eval.yaml>`	运行前验证配置
`skill-up list-cases <eval.yaml>`	列出所有用例
`skill-up run [eval.yaml]`	运行评测
`skill-up run --auto`	从 `evals/evals.json` 运行评测
`skill-up report <result.json> --format html`	重新生成报告
`skill-up import <evals.json>`	将Anthropic格式转换为YAML格式
`skill-up init`	生成用户配置模板
`skill-up debug judge <input.json>`	调试评判器
`skill-up debug report <input.json>`	调试报告

完整参数说明：

references/cli.md

。

Common pitfalls

常见陷阱

Model IDs vs proxy aliases — preserve what works for the user's
```
base_url
```
.
```
opensandbox
```
without
```
OPENSANDBOX_API_KEY
```
— auth failures.
Chinese
```
expect.must_contain
```
vs English model output — align language in prompts/assertions.
Abusing
```
agent_judge
```
.
Anthropic
```
evals.json
```
expectations → default
```
agent_judge
```
; use
```
import
```
+ hand edits for deterministic checks.
Paths relative to Skill root (
```
SKILL.md
```
directory).
```
--iteration 0
```
appends one run after the latest existing iteration without summarizing history; positive
```
--iteration N
```
runs N samples of the selected cases and, when N > 1, prints a simple stability/flakiness summary covering only samples from the current command.

模型ID与代理别名冲突——保留对用户
```
base_url
```
有效的配置。
使用
```
opensandbox
```
但未配置
```
OPENSANDBOX_API_KEY
```
——导致认证失败。
中文
```
expect.must_contain
```
与英文模型输出不匹配——确保提示词和断言语言一致。
滥用
```
agent_judge
```
。
Anthropic
```
evals.json
```
中的预期默认使用
```
agent_judge
```
；使用
```
import
```
命令后需手动编辑以实现确定性检查。
路径需相对于Skill根目录（
```
SKILL.md
```
所在目录）。
```
--iteration 0
```
会在最新迭代后追加一次运行，但不汇总历史记录；正整数
```
--iteration N
```
会对选定用例运行N次抽样，当N>1时，仅打印当前命令中抽样的稳定性/波动性简要总结。

References

参考资料

```
references/install.md
```
```
references/eval-yaml.md
```
```
references/case-yaml.md
```
```
references/judge-types.md
```
```
references/cli.md
```
```
references/migrate-anthropic.md
```

assets/eval.yaml.tmpl

assets/case.yaml.tmpl

```
references/install.md
```
```
references/eval-yaml.md
```
```
references/case-yaml.md
```
```
references/judge-types.md
```
```
references/cli.md
```
```
references/migrate-anthropic.md
```

assets/eval.yaml.tmpl

、

assets/case.yaml.tmpl