Back to Details

add-runner-eval

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Add a Runner Eval

添加 Runner Eval

Use this skill for the Runner Evals family: a real Runner/provider session against a seeded mock control plane. Product browser/server/database/Daytona coverage belongs in add-product-e2e-eval.
Locate the Paperclip checkout using 
PAPERCLIP_ROOT
when supplied, or 
git rev-parse --show-toplevel
from a checkout. From outside Git, inspect the workspace roots (for example 
~/paperclipai/paperclip
) and verify that the selected root contains 
packages/paperclip-runner
and 
tests/runner-e2e
. Locate 
paperclip-evals
using 
PAPERCLIP_EVALS_ROOT
or a discovered sibling; a worktree's parent directory need not contain that repository. Read 
doc/evals.md
and 
packages/paperclip-runner/docs/runner-protocol-live-evals.md
, then inspect the nearest existing case, roster, schema, and report test before editing. Definitions and authored cases belong in the sibling 
paperclip-evals/evals/paperclip-runner
; Runner integration, aggregation, viewer, and publication behavior belongs in 
packages/paperclip-runner
. Keep the control-plane boundary explicit in names and documentation.
The sibling eval README is the concrete map: cases live under 
cases/
, company fixtures under 
fixtures/
, runtime/model settings under 
configs/
, selections under 
rosters/
, and maintained campaign membership under 
campaigns/live-direct-full.json
. Update inventory/coverage mappings when the program requires them; a new file alone does not join the maintained campaign. From the Evals repository root, adapt these provider-free checks to the case and roster you changed. Run the reliability-plan validator only when that separate plan changes:
sh
python3 evals/paperclip-runner/tools/eval_program.py validate \
  --case evals/paperclip-runner/cases/get-task-context.json \
  --config evals/paperclip-runner/configs/live-codex-pinned.json
python3 evals/paperclip-runner/tools/run_live_roster.py validate \
  --roster evals/paperclip-runner/rosters/live-mini.json --run-id validate-new-case
python3 evals/paperclip-runner/tools/run_live_campaign.py validate \
  --campaign evals/paperclip-runner/campaigns/live-direct-full.json
python3 evals/paperclip-runner/tools/reliability_campaign.py validate \
  --plan evals/paperclip-runner/campaigns/paperclip-runner-reliability.json
Use nearby positive and negative grader cases/fixtures to calibrate the new assertion, including malformed or missing evidence where the grader must fail closed. Preserve the existing machine disposition and grade; product, model/provider, grading, and infrastructure labels are analytical annotations, not instructions to rewrite classifiers.
Author one bounded case with a deterministic semantic assertion and an inspectable result. Declare its expected operation, state effect, provider lane/profile, timeout and retry policy, and any required evidence. Do not grade hidden reasoning, infer success from a provider terminal message, or invent conversation/tool evidence. Public output follows the reviewed projection: sanitized fixture conversation and allowlisted tool outcomes may be published; raw trusted artifacts, credentials, secrets, private references, and hidden reasoning may not.
Validate without provider calls first using the commands above and the relevant report/render validation documented in the Runner docs. When a live run is authorized, pin the Paperclip commit and exact 40-character 
paperclip-evals
commit, select the smallest useful roster, and retain the complete provenance and cost record.
Update authoritative detailed docs when the contract or command changes, then link from 
doc/evals.md
rather than duplicating the Runner runbook. Keep public reports immutable and use the reviewed projection; sanitized fixture conversation and allowlisted tool outcomes may be public, while credentials, secrets, private references, raw trusted payloads, and hidden reasoning must not be exposed.
将此 skill 用于 Runner Evals 系列：针对已预置数据的 mock control plane 执行真实的 Runner/provider 会话。产品级浏览器、服务器、数据库和 Daytona 覆盖测试属于 add-product-e2e-eval。
如果提供了 
PAPERCLIP_ROOT
，则使用它定位 Paperclip checkout；否则从 checkout 中执行 
git rev-parse --show-toplevel
。如果不在 Git 仓库中，则检查 workspace roots（例如 
~/paperclipai/paperclip
），并确认所选根目录包含 
packages/paperclip-runner
和 
tests/runner-e2e
。 使用 
PAPERCLIP_EVALS_ROOT
或发现的同级仓库定位 
paperclip-evals
；worktree 的父目录不一定需要包含该仓库。阅读 
doc/evals.md
和 
packages/paperclip-runner/docs/runner-protocol-live-evals.md
，然后在编辑前检查最近的现有 case、roster、schema 和报告测试。Definitions 和编写的 cases 应放在同级仓库 
paperclip-evals/evals/paperclip-runner
中；Runner 集成、聚合、viewer 和发布行为应放在 
packages/paperclip-runner
中。 在名称和文档中明确体现 control-plane 边界。
同级 eval README 是具体映射：cases 位于 
cases/
，company fixtures 位于 
fixtures/
，runtime/model 设置位于 
configs/
，selections 位于 
rosters/
，维护中的 campaign 成员关系位于 
campaigns/live-direct-full.json
。如果计划要求，则更新 inventory/coverage 映射；仅新增文件不会自动加入维护中的 campaign。 从 Evals 仓库根目录出发，针对你修改的 case 和 roster 调整以下不调用 provider 的检查命令。仅当独立的 reliability plan 发生变化时，才运行 reliability-plan validator：
sh
python3 evals/paperclip-runner/tools/eval_program.py validate \
  --case evals/paperclip-runner/cases/get-task-context.json \
  --config evals/paperclip-runner/configs/live-codex-pinned.json
python3 evals/paperclip-runner/tools/run_live_roster.py validate \
  --roster evals/paperclip-runner/rosters/live-mini.json --run-id validate-new-case
python3 evals/paperclip-runner/tools/run_live_campaign.py validate \
  --campaign evals/paperclip-runner/campaigns/live-direct-full.json
python3 evals/paperclip-runner/tools/reliability_campaign.py validate \
  --plan evals/paperclip-runner/campaigns/paperclip-runner-reliability.json
使用附近正向和负向的 grader cases/fixtures 来校准新断言，包括 grader 必须 fail closed 时的格式错误或证据缺失场景。保留现有的 machine disposition 和 grade；产品、模型/provider、评分和基础设施标签仅作为分析注释，不是重写 classifiers 的指令。
编写一个有边界的 case，其中包含确定性的语义断言和可检查的结果。声明预期的操作、状态影响、provider lane/profile、timeout 和 retry policy，以及任何必需的证据。不要对隐藏推理进行评分，不要从 provider 的终端消息推断成功，也不要虚构 conversation/tool 证据。公开输出应遵循经过审查的 projection：经过清理的 fixture conversation 和允许列入白名单的工具结果可以发布；原始可信工件、凭据、密钥、私有引用和隐藏推理不得发布。
首先使用上述命令以及 Runner 文档中列出的相关 report/render validation，在不调用 provider 的情况下进行验证。获得 live run 授权后，固定 Paperclip commit 和确切的 40 字符 
paperclip-evals
commit，选择最小且有用的 roster，并保留完整的来源追踪和成本记录。
当 contract 或命令发生变化时，更新权威的详细文档，然后从 
doc/evals.md
链接该文档，而不是重复 Runner runbook。公开报告必须保持不可变并使用经过审查的 projection；经过清理的 fixture conversation 和允许列入白名单的工具结果可以公开，但不得暴露凭据、密钥、私有引用、原始可信载荷和隐藏推理。