wikieval

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Skill: wikieval

Turn wiki goldens into a CI gate: cheap deterministic asserts decide most goldens, a committed baseline catches regressions, and the one expensive thing (an LLM judge) is quarantined behind a single adapter file.

将 wiki goldens 转换为 CI gate：低成本的确定性 asserts 判定大多数 goldens，已提交的 baseline 捕获回归，而唯一昂贵的能力（LLM judge）被隔离在单个 adapter 文件之后。

WHAT

内容

Purpose và context

目的与上下文

Purpose: make wiki golden pages a regression suite checked in pure code (no model), with a committed baseline and a CI exit code that fires only when a metric drops below it.
Trigger (when to use):
- You want wiki pages to act as a regression suite an agent (or PR) cannot silently break.
- You have golden input→expected pairs and want them checked in pure code, no model needed.
- You need a CI step that exits non-zero only when a metric drops below a committed baseline.
- User says "wikieval", "eval suite for the wiki", "regression gate for goldens", "assertion cascade", "block CI on eval drop", or
```
/wikieval
```
  (model-invocation disabled — user invokes).
Non-goals: never calls an LLM (tier 3 is reported
```
needs-judge
```
, not judged); does not generate candidate outputs (they come from
```
--outputs
```
); does not author goldens' content for you.

**目的：**将 wiki golden 页面构建成通过纯代码检查（无需模型）的回归套件，并配备已提交的 baseline 和 CI exit code，仅在某个 metric 低于基线时触发。
触发条件（何时使用）：
- 你希望 wiki 页面充当回归套件，使 agent（或 PR）无法在无感知的情况下将其破坏。
- 你有 golden input→expected 对，希望通过纯代码检查，无需模型。
- 你需要一个 CI step，只有当某个 metric 低于已提交的 baseline 时才以非零状态退出。
- 用户说出 "wikieval"、"eval suite for the wiki"、"regression gate for goldens"、"assertion cascade"、"block CI on eval drop" 或
```
/wikieval
```
  （model invocation 已禁用——由用户调用）。
**非目标：**绝不调用 LLM（tier 3 会被报告为
```
needs-judge
```
，而不会实际评判）；不生成候选输出（它们来自
```
--outputs
```
）；不会替你编写 goldens 的内容。

Mental model

心智模型

golden (input · expected · rubric? · asserts?) + candidate output → cascade: tier 1 asserts → [tier 2 similarity] → [tier 3 needs-judge] → per-golden pass+score → baseline (write) | regression diff (check)

. Cheap tiers short-circuit; the expensive one sits behind one adapter file.

golden (input · expected · rubric? · asserts?) + candidate output → cascade：tier 1 asserts → [tier 2 similarity] → [tier 3 needs-judge] → 每个 golden 的 pass+score → baseline（写入） | regression diff（检查）

。低成本的 tiers 会短路；昂贵的 tier 位于单个 adapter 文件之后。

Input và output contract

输入与输出契约

	Field	Required?	Ý nghĩa
In	goldens `llmwiki/wiki/sources/evals/*.md`	có	YAML frontmatter `input` , `expected` , optional `rubric` , optional `asserts` , optional `id` ; also R9 `type` + R2 `## Origin`
In	`--outputs <outputs.json>`	có	candidate outputs per golden — makes the suite runnable without any model
In	mode	không	none = report; `--write-baseline` ; `--check`
In	`harness/wikieval.config.yaml`	không	adapter config ( `verified: false` ): judge model, rubric prompt, `embedding.backend`
Out	report	có	per-golden pass/fail/score/ `needs-judge`
Out	`harness/metrics/eval-baseline.json`	với `--write-baseline`	committed per-golden pass + score
Out	exit code	với `--check`	0 = no regression; 2 = a metric dropped below baseline (+ regression diff)

	字段	是否必需	含义
输入	goldens `llmwiki/wiki/sources/evals/*.md`	是	YAML frontmatter： `input` 、 `expected` 、可选的 `rubric` 、可选的 `asserts` 、可选的 `id` ；同时遵循 R9 `type` + R2 `## Origin`
输入	`--outputs <outputs.json>`	是	每个 golden 的候选输出，使整个套件无需任何模型即可运行
输入	mode	否	none = report； `--write-baseline` ； `--check`
输入	`harness/wikieval.config.yaml`	否	adapter 配置（ `verified: false` ）：judge model、rubric prompt、 `embedding.backend`
输出	report	是	每个 golden 的 pass/fail/score/ `needs-judge`
输出	`harness/metrics/eval-baseline.json`	使用 `--write-baseline` 时	已提交的每个 golden 的 pass + score
输出	exit code	使用 `--check` 时	0 = 无回归；2 = 某个 metric 低于 baseline（附 regression diff）

Rules và capabilities

规则与能力

RULE-01 (MUST): If a golden declares
```
asserts
```
, tier 1 alone decides pass/fail. Unknown ops fail closed.
RULE-02 (MUST): tier 2 similarity is only reached when a golden has no asserts; skipped + noted unless
```
embedding.backend
```
is set (
```
difflib
```
= stdlib lexical, offline; a real model id stays skipped until the adapter is wired).
RULE-03 (MUST): the
```
judge(output, rubric) -> {score, reason}
```
stub is NEVER called in deterministic mode; escalated goldens are reported
```
needs-judge
```
.
RULE-04 (MUST):
```
--check
```
is a deterministic diff against the committed baseline; exit 2 only on a drop.
RULE-05 (MUST): the adapter boundary is one file — finalizing the judge changes
```
harness/wikieval.config.yaml
```
+
```
judge()
```
only; the rest of WikiEval does not change.
Capabilities: read goldens + candidate outputs + adapter config; write baseline file; no network, no model.

RULE-01 (MUST)：若 golden 声明了
```
asserts
```
，则仅由 tier 1 决定 pass/fail。未知 ops 按 fail closed 处理。
RULE-02 (MUST)：只有在 golden 没有 asserts 时才会进入 tier 2 similarity；除非设置了
```
embedding.backend
```
，否则会 skipped + noted（
```
difflib
```
= stdlib lexical、offline；真实 model id 在 adapter 接线完成前仍保持 skipped）。
RULE-03 (MUST)：
```
judge(output, rubric) -> {score, reason}
```
stub 在 deterministic mode 下绝不会被调用；升级的 goldens 会被报告为
```
needs-judge
```
。
RULE-04 (MUST)：
```
--check
```
是与已提交 baseline 进行的 deterministic diff；仅在指标下降时返回 exit 2。
RULE-05 (MUST)：adapter boundary 只有一个文件——完成 judge 接入时只需更改
```
harness/wikieval.config.yaml
```
+
```
judge()
```
；WikiEval 的其余部分不变。
能力：读取 goldens + candidate outputs + adapter 配置；写入 baseline 文件；无网络访问、无模型调用。

Failure boundaries

失败边界

An assert fails → golden failed (report); with
```
--check
```
, drop vs baseline → exit 2 + regression diff.
Unknown assert op → failed closed (not skipped).
Golden without asserts and no
```
embedding.backend
```
→ tier 2 skipped + noted; rubric escalation →
```
needs-judge
```
(not a pass, not a fail).
Missing/invalid
```
--outputs
```
file → blocked: cannot run without candidates.

assert 失败 → golden failed（写入报告）；使用
```
--check
```
时，与 baseline 相比出现下降 → exit 2 + regression diff。
未知 assert op → failed closed（不会 skipped）。
Golden 没有 asserts 且未设置
```
embedding.backend
```
→ tier 2 skipped + noted；rubric escalation →
```
needs-judge
```
（既不是 pass，也不是 fail）。
缺失或无效的
```
--outputs
```
文件 → blocked：没有候选输出就无法运行。

HOW

使用方法

Main workflow

主要工作流

Step	Type	Inputs	Action	Outputs/exit	Failure/next
W01	deterministic	goldens, `--outputs`	Load goldens + candidate outputs	pairs per golden	missing outputs → blocked
W02	deterministic	pair, `asserts`	Tier 1 deterministic asserts	pass/fail if asserts declared	unknown op → fail closed
W03	deterministic	pair, config	Tier 2 similarity (only if no asserts)	score or "skipped + noted"	backend unset → skip
W04	deterministic	rubric	Tier 3 → report `needs-judge` (judge never called)	`needs-judge`	—
W05	deterministic	results, mode	Report / `--write-baseline` / `--check` diff vs baseline	report, baseline file, exit 0/2	drop → exit 2

Chi tiết từng bước (nguồn chân lý cho W01–W05):

步骤	类型	输入	操作	输出/退出	失败/后续
W01	deterministic	goldens、 `--outputs`	加载 goldens + candidate outputs	每个 golden 的 pair	outputs 缺失 → blocked
W02	deterministic	pair、 `asserts`	Tier 1 deterministic asserts	声明 asserts 时返回 pass/fail	未知 op → fail closed
W03	deterministic	pair、config	Tier 2 similarity（仅在没有 asserts 时）	score 或“skipped + noted”	backend 未设置 → skip
W04	deterministic	rubric	Tier 3 → 报告 `needs-judge` （绝不调用 judge）	`needs-judge`	—
W05	deterministic	results、mode	Report / `--write-baseline` / `--check` diff vs baseline	report、baseline 文件、exit 0/2	下降 → exit 2

各步骤的详细信息（W01–W05 的唯一事实来源）：

The assertion cascade (cheap → expensive, short-circuits)

Assertion cascade（低成本 → 高成本，短路执行）

tier 1 — deterministic asserts (build-now core):
```
equals
```
/
```
contains
```
/
```
icontains
```
/
```
not-contains
```
/
```
regex
```
/
```
is-json
```
/
```
is-sql-ish
```
run on the candidate output. If a golden declares
```
asserts
```
, tier 1 alone decides pass/fail. Unknown ops fail closed.
tier 2 — similarity (OPTIONAL): only reached when a golden has no asserts. Skipped + noted unless
```
embedding.backend
```
is set in the config (
```
difflib
```
= stdlib lexical, offline; a real model id stays skipped until the adapter is wired).
tier 3 — LLM-rubric judge = THE ADAPTER (
```
verified: false
```
). The stub
```
judge(output, rubric) -> {score, reason}
```
is NEVER called in deterministic mode; escalated goldens are reported
```
needs-judge
```
.

tier 1 — deterministic asserts（build-now 核心）：对 candidate output 执行
```
equals
```
/
```
contains
```
/
```
icontains
```
/
```
not-contains
```
/
```
regex
```
/
```
is-json
```
/
```
is-sql-ish
```
。若 golden 声明了 asserts，则仅由 tier 1 决定 pass/fail。未知 ops 按 fail closed 处理。
tier 2 — similarity（OPTIONAL）：只有在 golden 没有 asserts 时才会进入。除非在配置中设置了
```
embedding.backend
```
，否则会 skipped + noted（
```
difflib
```
= stdlib lexical、offline；真实 model id 在 adapter 接线完成前仍保持 skipped）。
tier 3 — LLM-rubric judge = THE ADAPTER（
```
verified: false
```
）。
```
judge(output, rubric) -> {score, reason}
```
stub 在 deterministic mode 下绝不会被调用；升级的 goldens 会被报告为
```
needs-judge
```
。

Goldens

Markdown under

llmwiki/wiki/sources/evals/

with YAML frontmatter:

input

expected

, optional

rubric

, optional

asserts

(quote each:

'contains:foo'

'regex:^...$'

'is-json'

'equals:bar'

'is-sql-ish'

). Optional

id

(defaults to filename stem). Each golden also obeys R9 (frontmatter

type

) and R2 (

## Origin

llmwiki/wiki/sources/evals/

下的 Markdown 文件，使用 YAML frontmatter：

input

、

expected

、可选的

rubric

、可选的

asserts

（每项都加引号，例如

'contains:foo'

、

'regex:^...$'

、

'is-json'

、

'equals:bar'

、

'is-sql-ish'

）。可选的

id

默认取文件名 stem。每个 golden 还需遵循 R9（frontmatter

type

）与 R2（

## Origin

）。

Run it (no model required)

运行（无需模型）

undefined

undefined

run the cascade with a fixed candidate set, print a report

python3 harness/scripts/wikieval.py --outputs harness/evals/wikieval-outputs.example.json

write / refresh the committed baseline (per-golden pass + score)

python3 harness/scripts/wikieval.py --outputs <outputs.json> --write-baseline

CI gate: re-run and exit 2 if any metric dropped below baseline (deterministic diff)

python3 harness/scripts/wikieval.py --outputs <outputs.json> --check

Baseline: `harness/metrics/eval-baseline.json`. Adapter config: `harness/wikieval.config.yaml`.

python3 harness/scripts/wikieval.py --outputs <outputs.json> --check

Baseline：`harness/metrics/eval-baseline.json`。Adapter 配置：`harness/wikieval.config.yaml`。

Branches

分支

ID	Kind	Guard	Hành vi	Skip / failure	Rejoin
B01	capability_optional	golden has no `asserts` AND `embedding.backend` set	tier 2 similarity ( `difflib` offline)	backend unset or real model id not wired → skipped + noted	W04
B02	user_optional	`--write-baseline`	write/refresh `harness/metrics/eval-baseline.json`	—	end
B03	user_optional	`--check`	deterministic diff vs baseline, exit 2 on drop	no baseline → run `--write-baseline` first	end

ID	种类	条件	行为	跳过/失败	汇合点
B01	capability_optional	golden 没有 `asserts` 且设置了 `embedding.backend`	Tier 2 similarity（ `difflib` offline）	backend 未设置，或真实 model id 尚未接线 → skipped + noted	W04
B02	user_optional	`--write-baseline`	写入/刷新 `harness/metrics/eval-baseline.json`	—	end
B03	user_optional	`--check`	与 baseline 进行 deterministic diff，下降时 exit 2	没有 baseline → 先运行 `--write-baseline`	end

Validation và stopping

验证与停止条件

Everything is code-checked: asserts, cascade, baseline diff and exit code are deterministic. Only tier 3 needs review (

needs-judge

) and stays out of the pass/fail decision until the adapter is finalized.

所有内容都通过代码检查：asserts、cascade、baseline diff 和 exit code 都是 deterministic。只有 tier 3 需要审查（

needs-judge

），在 adapter 完成前不参与 pass/fail 决策。

Self-test (build-now slice)

Self-test（build-now slice）

OUT=harness/evals/wikieval-outputs.example.json
python3 harness/scripts/wikieval.py --outputs $OUT --write-baseline   # both goldens PASS
python3 harness/scripts/wikieval.py --outputs $OUT --check            # exit 0 — no regression

OUT=harness/evals/wikieval-outputs.example.json
python3 harness/scripts/wikieval.py --outputs $OUT --write-baseline   # both goldens PASS
python3 harness/scripts/wikieval.py --outputs $OUT --check            # exit 0 — no regression

flip one output to break an assert, then:

python3 harness/scripts/wikieval.py --outputs <broken.json> --check # exit 2 + regression diff

undefined

python3 harness/scripts/wikieval.py --outputs <broken.json> --check # exit 2 + regression diff

undefined

Examples

示例

Positive:

python3 harness/scripts/wikieval.py --outputs harness/evals/wikieval-outputs.example.json --check

after writing baseline → both goldens PASS → exit 0.

Boundary/failure: flip one candidate so
```
'contains:foo'
```
no longer matches →
```
--check
```
exits 2 with the regression diff naming that golden.
Boundary: golden with only
```
rubric
```
, no asserts, no
```
embedding.backend
```
→ tier 2 skipped + noted, reported
```
needs-judge
```
; judge not called.

**正向：**写入 baseline 后运行

python3 harness/scripts/wikieval.py --outputs harness/evals/wikieval-outputs.example.json --check

→ 两个 goldens 均 PASS → exit 0。

**边界/失败：**翻转一个 candidate，使
```
'contains:foo'
```
不再匹配 →
```
--check
```
返回 exit 2，并提供指出对应 golden 的 regression diff。
**边界：**golden 只有
```
rubric
```
，没有 asserts，也没有
```
embedding.backend
```
→ tier 2 skipped + noted，并报告为
```
needs-judge
```
；不调用 judge。

Reference — Adapter boundary (build-now-adapt-later)

参考——Adapter boundary（build-now-adapt-later）

Built + tested now: tier-1 asserts, the cascade, baseline write +
```
--check
```
regression diff.
Quarantined (
verified: false
):
```
harness/wikieval.config.yaml
```
(judge model + rubric prompt, embedding backend — each a flagged
```
# ASSUMPTION
```
) and the
```
judge()
```
stub.
Finalize: wire
```
judge()
```
to the configured model, run the self-test, flip
```
verified: true
```
. One file changes; the rest of WikiEval does not.

**当前已构建并经过测试：**tier-1 asserts、cascade、baseline 写入 +
```
--check
```
regression diff。
已隔离（
verified: false
）：
```
harness/wikieval.config.yaml
```
（judge model + rubric prompt、embedding backend——每项均以带标记的
```
# ASSUMPTION
```
呈现）和
```
judge()
```
stub。
**完成接入：**将
```
judge()
```
接到配置的 model，运行 self-test，并将
```
verified
```
改为
```
true
```
。只有一个文件会发生变化；WikiEval 的其余部分不变。

Reference — Files

参考——文件

```
harness/scripts/wikieval.py
```
— the eval engine.

harness/wikieval.config.yaml

— the one adapter (

verified: false

```
harness/metrics/eval-baseline.json
```
— committed regression baseline (generated).
```
llmwiki/wiki/sources/evals/*.md
```
— goldens.

harness/evals/wikieval-outputs.example.json

— self-test candidate outputs.

```
harness/scripts/wikieval.py
```
— eval 引擎。

harness/wikieval.config.yaml

— 唯一的 adapter（

verified: false

）。

```
harness/metrics/eval-baseline.json
```
— 已提交的 regression baseline（生成文件）。
```
llmwiki/wiki/sources/evals/*.md
```
— goldens。

harness/evals/wikieval-outputs.example.json

— self-test candidate outputs。