fable-eval

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Fable Eval

Measure whether an agent-control change improves the behavior it claims to improve without quietly overfitting the benchmark or breaking neighboring behavior.

衡量 Agent 控制变更是否真正改善了其声称要改善的行为，同时没有悄然过拟合 benchmark 或破坏相邻行为。

Mission

目标

An eval is a decision instrument, not a scoreboard. It needs a frozen comparison point, representative semantic families, oracle isolation, explicit failure costs, and a rollback decision.

A candidate should not win because the prompts resemble its instructions, because holdouts leaked into authoring, or because one average score hides a severe regression.

eval 是一种决策工具，而不是记分牌。它需要冻结的比较点、有代表性的语义族、oracle 隔离、明确的失败代价和回滚决策。

候选方案不应仅因 prompt 与其指令相似、holdout 泄漏到编写过程，或某个平均分掩盖了严重回归而胜出。

Activate When

启用条件

changing Skills, prompts, routers, hooks, agent profiles, policies, or model-control logic;
comparing candidate prompt/agent configurations;
measuring trigger precision/recall or action compliance;
validating a new behavioral maturity claim;
investigating whether an apparent improvement is robust or benchmark-specific.

变更 Skill、prompt、Router、Hook、Agent profile、policy 或模型控制逻辑；
比较候选 prompt 或 Agent 配置；
衡量触发精确率/召回率或 action 遵循度；
验证新的行为成熟度声明；
调查表观改进是否稳健，还是仅对特定 benchmark 有效。

Do Not Activate When

不启用条件

verifying ordinary application behavior (
```
fable-verify
```
);
authoring a Skill before its intended behavior is clear (
```
skill-creator
```
);
running a one-off subjective prompt demo with no acceptance decision.

验证普通应用行为（
```
fable-verify
```
）；
在预期行为尚不明确时编写 Skill（
```
skill-creator
```
）；
进行一次性主观 prompt 演示，且不存在明确的验收决策。

Evaluation Classification

评估分类

Change	Primary eval risk
Router/trigger	false positives, false negatives, precedence
Skill instruction	action correctness, forbidden shortcuts, boundary behavior
Spark/next-action	top-1 action, unsafe suggestion, silence precision
Hook/guard	enforcement, false blocking, bypasses
Prompt/persona	task quality + regressions + instruction conflicts
Tool policy	correct tool choice, unsafe/missing action
Model/config	quality/latency/cost variance across representative tasks

变更	主要 eval 风险
Router/trigger	误报、漏报、优先级冲突
Skill instruction	action 正确性、禁止的捷径、边界行为
Spark/next-action	top-1 action、不安全建议、保持静默的精确率
Hook/guard	执行力度、错误阻断、绕过
Prompt/persona	任务质量、回归、指令冲突
Tool policy	正确的工具选择、不安全或缺失的 action
Model/config	代表性任务之间的质量、延迟和成本差异

Protocol

协议

Stage 1 — Define the decision before running tests

阶段 1 — 在运行测试前定义决策

State:

candidate being evaluated;
baseline/control;
exact behavior expected to improve;
metrics and thresholds;
unacceptable regressions;
rollback action.

Avoid inventing metrics after seeing results.

明确：

被评估的候选方案；
baseline/control；
确切预期改善的行为；
指标和阈值；
不可接受的回归；
回滚操作。

避免在看到结果后临时编造指标。

Stage 2 — Build semantic scenario families

阶段 2 — 构建语义场景族

Cover distinct decisions, not wording variants. Include as applicable:

straightforward positive case;
non-trigger/boundary case;
ambiguous competing action;
adversarial shortcut pressure;
partial/contradictory evidence;
failure/recovery path;
legacy/constrained environment;
unseen holdout.

Record family coverage separately from raw prompt count.

覆盖不同的决策，而非仅变换措辞。根据实际情况包括：

直接的正例；
不触发/边界用例；
存在竞争 action 的歧义用例；
对抗性捷径压力；
部分或矛盾的证据；
失败/恢复路径；
旧版/受限环境；
未见 holdout。

分别记录语义族覆盖情况和原始 prompt 数量。

Stage 3 — Freeze baseline and corpus identity

阶段 3 — 冻结 baseline 和 corpus 身份

Bind the run to:

corpus hash/version;
candidate/baseline identity;
evaluator/scorer version;
provider/model/config where external;
timestamp/repository revision.

Changing the subject, oracle, corpus, or scoring logic invalidates direct comparability unless explicitly normalized.

将本次运行绑定到：

corpus 哈希/版本；
候选方案/baseline 身份；
evaluator/scorer 版本；
外部 provider/model/config；
时间戳/repository revision。

除非明确完成归一化，否则更改被评估对象、oracle、corpus 或评分逻辑后，证据将失去直接可比性。

Stage 4 — Protect the oracle

阶段 4 — 保护 oracle

Provider-facing requests must not reveal expected actions, forbidden actions, category labels, holdout identity, scoring implementation, or answer-bearing metadata.

Do not draft the candidate while repeatedly reading holdout failures. Promote discovered cases into a future checked corpus and preserve a new unseen holdout.

面向 provider 的请求不得泄露预期 action、禁止 action、category 标签、holdout 身份、评分实现或含答案的元数据。

不要在反复阅读 holdout 失败项时编写候选方案。将发现的用例纳入未来的受检 corpus，并保留一个新的未见 holdout。

Stage 5 — Execute baseline and candidate consistently

阶段 5 — 一致地执行 baseline 和候选方案

Use the same task inputs, tool availability, context budget, temperature/configuration, timeout policy, and scoring rules where comparison requires them.

Capture provider errors/timeouts as failures or explicit unavailable states; do not fill missing outputs from the oracle.

在需要比较时，使用相同的任务输入、工具可用性、context 预算、temperature/config、超时 policy 和评分规则。

将 provider 错误或超时记录为失败或明确的不可用状态；不要使用 oracle 填补缺失的输出。

Stage 6 — Score by slices, not average alone

阶段 6 — 按切片评分，而非只看平均值

Inspect:

overall metric;
each semantic family;
negative/adversarial forbidden violations;
high-cost regressions;
variance/repeated-run stability where stochasticity is material;
routing confusion pairs where applicable.

A 1% average gain is not acceptable if it introduces a severe release/security/recovery regression.

检查：

总体指标；
每个语义族；
负例/对抗样本中的禁止违规；
高成本回归；
存在实质随机性时的方差/重复运行稳定性；
适用情况下的路由混淆对。

如果 1% 的平均提升引入了严重的发布、安全或恢复回归，则该提升不可接受。

Stage 7 — Investigate suspicious gains

阶段 7 — 调查可疑的改进

Check for:

prompt leakage;
duplicated/near-duplicate scenarios;
benchmark-specific keyword matching;
changed tool/context budget;
scorer drift;
cherry-picked seeds/runs;
examples copied into the candidate.

检查：

prompt 泄漏；
重复或近似重复的场景；
针对特定 benchmark 的关键词匹配；
工具/context 预算发生变化；
scorer 漂移；
挑选有利的 seed 或运行结果；
示例被复制到候选方案中。

Stage 8 — Decide and preserve rollback

阶段 8 — 决策并保留回滚能力

Verdict:

ACCEPT: thresholds met, no prohibited regression, evidence representative/fresh;
REJECT: candidate regresses required behavior or fails threshold;
INCONCLUSIVE: evidence lacks breadth/stability/holdout integrity.

Record baseline artifact so rollback remains possible.

结论：

ACCEPT：达到阈值、不存在禁止的回归，且证据具有代表性和时效性；
REJECT：候选方案导致必需行为回归或未达到阈值；
INCONCLUSIVE：证据缺少广度、稳定性或 holdout 完整性。

记录 baseline artifact，以确保始终可以回滚。

Decision Rules

决策规则

Semantic family breadth matters more than raw scenario count.
Surface rewrites of one case do not create independent coverage.
Holdouts stop being holdouts once used repeatedly to tune the candidate.
Compare slices before averages; safety-critical forbidden violations can veto a higher average score.
If stochastic variance could change the decision, repeat enough runs to estimate stability rather than cherry-picking one seed.
If provider/runtime errors differ between candidate and baseline, separate infrastructure failure from behavior score.
Never preserve an old maturity result after the evaluated Skill/corpus/oracle changes unless freshness validation proves identity.
Do not lower thresholds after a candidate fails simply to ship it.

语义族的广度比原始场景数量更重要。
对同一个案例进行表面改写，不会形成独立覆盖。
holdout 一旦被反复用于调优候选方案，就不再是 holdout。
先比较各个切片，再比较平均值；安全关键型禁止违规可以否决更高的平均分。
如果随机方差可能改变决策，应重复运行足够多次以评估稳定性，而不是挑选单个 seed。
如果候选方案和 baseline 的 provider/runtime 错误不同，应将基础设施故障与行为评分分开。
被评估的 Skill、corpus 或 oracle 发生变化后，绝不能继续沿用旧的成熟度结果，除非时效性验证证明其身份一致。
候选方案失败后，不得仅仅为了发布而降低阈值。

Invariants

不变量

Baseline is reproducible/frozen before candidate judgment.
Oracle/holdout data remains hidden from the evaluated agent.
Candidate and baseline are compared under equivalent conditions where claimed.
Missing/failed provider outputs are never replaced by expected answers.
Every acceptance decision has a rollback path.
High-cost regressions remain visible even when aggregate score improves.

在判断候选方案之前，baseline 必须可复现且已冻结。
oracle/holdout 数据必须对被评估的 Agent 保持隐藏。
在所声称的比较范围内，候选方案和 baseline 必须采用等价条件进行比较。
缺失或失败的 provider 输出绝不能用预期答案替代。
每个接受决策都必须有回滚路径。
即使聚合分数有所改善，高成本回归仍需保持可见。

Failure Taxonomy

失败分类

Benchmark overfit

Benchmark 过拟合

Candidate improves checked cases but fails unseen family/holdout. Increase semantic breadth and reject/generalize candidate.

候选方案在已检查用例上有所提升，却在未见语义族/holdout 上失败。应增加语义广度，并拒绝候选方案或推动其泛化。

Oracle leakage

Oracle 泄漏

Expected/forbidden/category data reaches provider/candidate authoring loop. Discard contaminated evidence and create fresh blind cases.

预期、禁止或 category 数据进入 provider/候选方案的编写循环。应丢弃受污染的证据，并创建全新的盲测用例。

Metric blindness

指标盲区

Average improves while important slice worsens. Use per-family and veto metrics.

平均值有所改善，但重要切片发生恶化。应使用逐语义族指标和否决指标。

Non-comparable runs

运行不可比

Different model/tool/context/scorer settings produce apparent gain. Re-run under controlled conditions.

不同的 model、tool、context 或 scorer 设置产生了表观提升。应在受控条件下重新运行。

High variance

高方差

Repeated runs change verdict. Increase samples/control nondeterminism or mark inconclusive.

重复运行会改变结论。应增加样本、控制非确定性，或将其标记为 INCONCLUSIVE。

Corpus drift

Corpus 漂移

Skill/scenario/oracle changed after evidence capture. Mark evidence stale and rerun.

Skill、场景或 oracle 在证据采集后发生变化。应将证据标记为过期并重新运行。

Anti-Patterns

反模式

five paraphrases counted as five independent tests;
reading holdouts while tuning every candidate;
accepting on average score alone;
changing thresholds after seeing failure;
treating provider timeout as skipped rather than failed/incomplete;
evaluating only positive examples;
copying expected action vocabulary in a way that reveals the answer per case;
claiming M4 because an evidence JSON file exists even though corpus hash changed.

将同一案例的五个改写版本视为五项独立测试；
在调优每个候选方案时阅读 holdout；
仅依据平均分作出接受决定；
在看到失败后修改阈值；
将 provider 超时视为跳过，而不是失败或不完整；
仅评估正例；
复制预期 action 词汇，以逐用例泄露答案；
仅因存在 evidence JSON 文件就声称达到 M4，即使 corpus 哈希已经发生变化。

Eval Report

Eval 报告

text

Candidate / baseline:
Corpus + hashes:
Provider/config:
Semantic families:
Metrics + thresholds:
Per-family results:
Forbidden/high-cost regressions:
Variance/repeats:
Leakage/comparability checks:
Verdict: ACCEPT | REJECT | INCONCLUSIVE
Rollback:
Evidence freshness:

text

候选方案 / baseline：
Corpus + 哈希：
Provider/config：
语义族：
指标 + 阈值：
各语义族结果：
禁止项/高成本回归：
方差/重复运行：
泄漏/可比性检查：
结论：ACCEPT | REJECT | INCONCLUSIVE
回滚：
证据时效性：

Completion Criteria

完成标准

Evaluation completes when:

baseline and candidate identities are explicit;
semantic coverage and blind holdout integrity are credible;
metrics are inspected by meaningful slices;
regressions/forbidden actions are not hidden by averages;
verdict and rollback are evidence-backed;
evidence freshness is tied to the exact evaluated corpus/control.

当满足以下条件时，评估即告完成：

baseline 和候选方案的身份明确；
语义覆盖和盲测 holdout 的完整性可信；
按有意义的切片检查了指标；
回归和禁止 action 未被平均值掩盖；
结论和回滚方案均有证据支持；
证据时效性与被评估的确切 corpus/control 绑定。

Progressive Resources

延伸资源

Deep guide:

references/benchmark-design-and-overfit-control.md

Existing protocol:
```
references/eval-harness-protocol.md
```
Example:
```
examples/eval-regression-run.md
```

深度指南：

references/benchmark-design-and-overfit-control.md

现有 protocol：
```
references/eval-harness-protocol.md
```
示例：
```
examples/eval-regression-run.md
```