trace-grader

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Skill: trace-grader

Grade an agent's trajectory, not just its output. A run can return the right answer via a bad path (force-pushed, retry-stormed, edited a file it never read) — "corrupt success" — and it can be flaky (passes once, fails on rerun). An answer-only checker is blind to both; TraceGrader inspects the path.

评估 Agent 的 trajectory（执行轨迹），而不仅是输出。一次运行可能通过不良路径返回正确答案（force-push、retry storm、编辑了从未读取的文件）——即“corrupt success”——也可能是 flaky（一次通过，重跑失败）。仅检查答案的检查器对这两种情况都无能为力；TraceGrader 会检查执行路径。

WHAT

WHAT（是什么）

Purpose và context

目的与上下文

Purpose: chấm ĐƯỜNG ĐI của agent (tool choice, ordering, retries, repeatability, grounding) chứ không chỉ đáp án, để bắt "corrupt success" và flakiness bằng tín hiệu tất định gate được trong CI.
Trigger (when to use):
You have one or more agent runs (traces) and want to know how the answer was reached, not only whether it was reached.
You need a repeatability gate: "does this task pass on ALL k reruns?" (
```
pass^k
```
).
You want a deterministic, CI-gateable signal on tool choice / ordering / retries / step budget — before reaching for an LLM judge.
Non-goals: không chấm chất lượng đáp án cuối; không bật trục agent-as-judge (STUB, chưa gọi) khi config còn
```
verified:false
```
; không sửa run bị gắn cờ.

目的： 对 Agent 的执行路径（工具选择、顺序、重试、可重复性、grounding）评分，而不仅是答案，以通过可在 CI 中作为 gate 的确定性信号捕获“corrupt success”和 flakiness。
触发条件（何时使用）：
你拥有一次或多次 Agent 运行记录（traces），想知道答案是如何得出的，而不只是是否得出了答案。
你需要一个可重复性 gate：“这个 task 是否在全部 k 次重跑中都通过？”（
```
pass^k
```
）。
你希望获得关于工具选择、顺序、重试或步骤预算的确定性、可用于 CI gate 的信号，而不是直接依赖 LLM judge。
非目标： 不评估最终答案的质量；只要 config 仍为
```
verified:false
```
，就不启用 agent-as-judge 轴（STUB，不会调用）；不修改被标记的 run。

Mental model

心智模型

trace nguồn (traces.json · audit jsonl · session transcript) → parser → run = [{step, tool, args, observation, ok}] → rule checks (config) + pass^k → verdict per run (clean-pass · pass-with-warnings · corrupt-success · fail) → exit 3 nếu corrupt/flaky

. Tham số luật nằm DUY NHẤT trong adapter

harness/trace-grader.config.yaml

(chi tiết ở mục Reference dưới).

trace 来源（traces.json · audit jsonl · session transcript）→ parser → run = [{step, tool, args, observation, ok}] → 规则检查（config）+ pass^k → 每次运行的 verdict（clean-pass · pass-with-warnings · corrupt-success · fail）→ 若存在 corrupt/flaky 则 exit 3

。规则参数只存在于 adapter

harness/trace-grader.config.yaml

中（详见下方“参考”）。

Input và output contract

输入与输出契约

	Field	Required?	Ý nghĩa
In	`--traces` / `--audit` / `--transcript`	có (một trong ba)	canonical `traces.json` ; audit jsonl (một run mỗi `session_id` ); transcript phiên Claude Code
In	`--task`	khi dùng audit/transcript	gán run vào task để tính `pass^k`
In	`--json`	không	output máy đọc
In	`harness/trace-grader.config.yaml`	có	forbidden tools, `retry_threshold` , `step_budget` , `order_constraints` , `grounding.enabled`
Out	verdict per run + flags + khối `pass^k`	có	báo cáo
Out	exit code	có	`3` nếu có `corrupt-success` hoặc task flaky

	字段	必填？	含义
输入	`--traces` / `--audit` / `--transcript`	是（三者之一）	规范的 `traces.json` ；audit jsonl（每个 `session_id` 对应一次 run）；Claude Code transcript
输入	`--task`	使用 audit/transcript 时必填	将 run 关联到 task，以计算 `pass^k`
输入	`--json`	否	机器可读输出
输入	`harness/trace-grader.config.yaml`	是	forbidden tools、 `retry_threshold` 、 `step_budget` 、 `order_constraints` 、 `grounding.enabled`
输出	每次运行的 verdict + flags + `pass^k` 块	是	报告
输出	exit code	是	若存在 `corrupt-success` 或 task flaky，则为 `3`

Rules và capabilities

规则与能力

RULE-01 (MUST): The config file is the ONLY place rule parameters live. Same guess in two places = a leak.
RULE-02 (MUST): Never present a guessed threshold as verified — the file stays
```
verified:false
```
until validated.
RULE-03 (MUST): The agent-as-judge axis is OFF and uncalled until configured + verified; never fake an inferential score.
RULE-04 (MUST): The audit parser cannot see per-step
```
ok
```
/observations, so
```
retry_storm
```
won't fire from a pure audit log — use full
```
traces.json
```
when you need failure-aware checks.
Capabilities: đọc file trace/log/transcript cục bộ; đọc một file config adapter; không gọi model (judge tắt), không ghi gì ngoài báo cáo.

RULE-01 (MUST)：配置文件是规则参数唯一存放的位置。在两处采用相同的猜测 = 泄漏。
RULE-02 (MUST)：绝不能将猜测的阈值表述为已验证；在完成验证前，文件保持
```
verified:false
```
。
RULE-03 (MUST)：在完成配置与验证前，agent-as-judge 轴保持关闭且不会被调用；绝不能伪造推断性评分。
RULE-04 (MUST)：audit parser 无法看到每一步的
```
ok
```
/observation，因此纯 audit log 不会触发
```
retry_storm
```
；需要 failure-aware 检查时，应使用完整的
```
traces.json
```
。
能力：读取本地 trace/log/transcript 文件；读取一个 config adapter；不调用 model（judge 已关闭），除报告外不写入任何内容。

Failure boundaries

失败边界

Có run
```
corrupt-success
```
hoặc
```
pass^k = FAIL
```
→ exit
```
3
```
(failed cho CI gate) — đó là tín hiệu, không phải lỗi tool.
Nguồn chỉ là audit log → không thấy
```
ok
```
/observation từng bước,
```
retry_storm
```
không bắn → kết quả partial, dùng
```
traces.json
```
đầy đủ nếu cần.
Transcript không có task-verdict →
```
run.ok
```
bị ép True → tin FLAGS, không tin verdict delivered.
Ngưỡng chưa kiểm trên trace thật → giữ
```
verified:false
```
, không trình bày như đã xác minh.

若存在
```
corrupt-success
```
run 或
```
pass^k = FAIL
```
→ exit
```
3
```
（对 CI gate 而言为 failed）——这是信号，而不是工具错误。
来源仅为 audit log → 看不到每一步的
```
ok
```
/observation，
```
retry_storm
```
不会触发 → 结果为 partial；如需 failure-aware 检查，请使用完整的
```
traces.json
```
。
Transcript 没有 task-verdict →
```
run.ok
```
被强制设为 True → 应信任 FLAGS，而不要信任已交付 verdict。
阈值尚未在真实 trace 上验证 → 保持
```
verified:false
```
，不要将其表述为已验证。

HOW

HOW（如何使用）

Main workflow

主流程

Step	Type	Inputs	Action	Outputs/exit	Failure/next
W01	deterministic	runs	Collect traces ( `traces.json` hoặc `--audit` / `--transcript` )	nguồn trace	chỉ audit → partial (B01)
W02	judgment	dự án	Set the boundary: chỉ sửa `harness/trace-grader.config.yaml`	config	—
W03	deterministic	trace + config	Grade: chạy `trace-grader.py`	verdict + flags + `pass^k` , rc	rc 3 → W04
W04	judgment	báo cáo	Act on signal: điều tra bước bị cờ / coi task flaky là không tin được	kết luận	—
W05	judgment	trace thật	(Later) verify adapter rồi flip `verified:true`	config verified	chưa đủ bằng chứng → giữ false (B02)

Chi tiết từng bước (nguồn chân lý cho W01–W05):

Collect traces. Either emit
```
traces.json
```
in the schema above, or point
```
--audit
```
at a
```
hooklib.audit()
```
jsonl (one run per
```
session_id
```
).
Set the boundary. Edit
```
harness/trace-grader.config.yaml
```
ONLY — forbidden tools,
```
retry_threshold
```
,
```
step_budget
```
,
```
order_constraints
```
. Keep it the single place these values live.
Grade. Run the grader; read per-run verdicts + flags + the
```
pass^k
```
block.
Act on signal.
```
corrupt-success
```
= right answer, bad/unsafe path → investigate the flagged step.
```
pass^k = FAIL
```
= flaky → not safe to rely on.
(Later) verify the adapter. Validate thresholds against real traces, then flip
```
verified:true
```
. Only then consider enabling the agent-as-judge axis.

步骤	类型	输入	操作	输出/退出	失败/下一步
W01	deterministic	runs	收集 traces（ `traces.json` 或 `--audit` / `--transcript` ）	trace 来源	仅有 audit → partial（B01）
W02	judgment	项目	设定边界：只修改 `harness/trace-grader.config.yaml`	config	—
W03	deterministic	trace + config	评分：运行 `trace-grader.py`	verdict + flags + `pass^k` 、rc	rc 3 → W04
W04	judgment	报告	根据信号采取行动：调查被标记的步骤，或将 flaky task 视为不可信	结论	—
W05	judgment	真实 trace	（稍后）验证 adapter，然后切换至 `verified:true`	config verified	证据不足 → 保持 false（B02）

各步骤详情（W01–W05 的唯一事实来源）：

收集 traces。 根据上述 schema 生成
```
traces.json
```
，或让
```
--audit
```
指向
```
hooklib.audit()
```
生成的 jsonl（每个
```
session_id
```
对应一次 run）。
设定边界。 只能编辑
```
harness/trace-grader.config.yaml
```
——用于设置 forbidden tools、
```
retry_threshold
```
、
```
step_budget
```
、
```
order_constraints
```
。这些值只能保存在这一处。
评分。 运行 grader；读取每次运行的 verdict + flags +
```
pass^k
```
块。
根据信号采取行动。
```
corrupt-success
```
= 答案正确，但路径不良或不安全 → 调查被标记的步骤。
```
pass^k = FAIL
```
= flaky → 不可依赖。
（稍后）验证 adapter。 对照真实 traces 验证阈值，然后切换至
```
verified:true
```
。只有在此之后，才考虑启用 agent-as-judge 轴。

How to run

如何运行

bash

undefined

bash

undefined

bundled 3-case self-test (clean / corrupt / flaky)

python3 harness/scripts/trace-grader.py --self-test

grade canonical traces, an audit log, or a full session transcript; --json for machine output

python3 harness/scripts/trace-grader.py --traces traces.json python3 harness/scripts/trace-grader.py --audit .claude/audit/<date>.jsonl --task my-task python3 harness/scripts/trace-grader.py --transcript ~/.claude/projects/<proj>/<session>.jsonl --task my-task

Exit code is `3` if any run is `corrupt-success` or any task is flaky (so CI can gate).

如果任何 run 为 `corrupt-success`，或任何 task 存在 flaky，退出码将为 `3`（因此可用于 CI gate）。

Branches

分支

ID	Kind	Guard	Hành vi	Skip / failure	Rejoin
B01	conditional_required	nguồn là audit jsonl thuần	chấm được nhưng `retry_storm` không bắn; báo giới hạn	cần failure-aware → chuyển `traces.json` hoặc `--transcript`	W03
B02	capability_optional	judge model đã cấu hình VÀ config `verified:true`	mới xem xét bật trục agent-as-judge ( `judge_trajectory()` )	chưa đủ → STUB, không gọi	W04

ID	类型	条件	行为	跳过/失败	汇合
B01	conditional_required	来源为纯 audit jsonl	可以评分，但 `retry_storm` 不会触发；需报告该限制	需要 failure-aware → 改用 `traces.json` 或 `--transcript`	W03
B02	capability_optional	judge model 已配置且 config 为 `verified:true`	只有此时才考虑启用 agent-as-judge 轴（ `judge_trajectory()` ）	证据不足 → 保持 STUB，不调用	W04

Validation và stopping

验证与停止条件

Tất định:

python3 harness/scripts/trace-grader.py --self-test

(3 case clean / corrupt / flaky), verdict và exit code do engine quyết. Cần review: việc flip

verified:true

sau khi so ngưỡng với trace thật. Dừng sau W04; W05 là bước làm sau, không chạy trong cùng lượt nếu chưa có trace thật.

确定性验证：运行

python3 harness/scripts/trace-grader.py --self-test

（3 个 clean/corrupt/flaky case），verdict 和退出码由引擎决定。需要人工审查：将阈值与真实 trace 对比后，再切换至

verified:true

。在 W04 后停止；如果尚无真实 trace，W05 属于后续步骤，不应在同一轮中执行。

Examples

示例

Positive:

python3 harness/scripts/trace-grader.py --transcript ~/.claude/projects/<proj>/<session>.jsonl --task my-task

trên phiên có Edit vào file chưa từng Read → flag

edited_without_read

(medium), verdict

pass-with-warnings

, chỉ ra bước bị cờ.

Boundary/failure: 3 lần chạy cùng task, 1 lần fail →
```
pass^k = FAIL
```
→ exit
```
3
```
, task bị coi là flaky; run trả đúng đáp án nhưng có
```
forbidden_tool
```
(force-push, high) →
```
corrupt-success
```
, exit
```
3
```
.

正面示例： 在一个包含 Edit、但文件从未 Read 的会话上运行

python3 harness/scripts/trace-grader.py --transcript ~/.claude/projects/<proj>/<session>.jsonl --task my-task

→ 标记

edited_without_read

（medium），verdict 为

pass-with-warnings

，并指出被标记的步骤。

边界/失败： 同一 task 运行 3 次，其中 1 次失败 →
```
pass^k = FAIL
```
→ exit
```
3
```
，该 task 被视为 flaky；某次 run 返回了正确答案，但使用了
```
forbidden_tool
```
（force-push，high）→
```
corrupt-success
```
，exit
```
3
```
。

Reference — What it is (build-now vs adapter)

参考 — 它是什么（当前构建与 adapter）

Built now — deterministic, no unknowns:

Trace schema (the contract): a run = list of steps
```
{step, tool, args, observation, ok}
```
.
Parsers: canonical
```
traces.json
```
; the audit jsonl shape written by
```
llmwiki/.claude/hooks/hooklib.audit()
```
(and code-logger
```
events.jsonl
```
); OR a full Claude Code session transcript (
```
--transcript
```
) — the richest source: it recovers retrieval args (Grep
```
pattern
```
, Read
```
file_path
```
),
```
observation
```
, and per-step
```
ok
```
, so grounding + outcome are observable. Sidechain (sub-agent) lines are skipped; a transcript carries no task-verdict so its
```
run.ok
```
is forced True — trust the FLAGS.
pass^k
: k runs of one task → require ALL k to deliver; report
```
pass^k
```
.
Rule checks (config-driven):
```
forbidden_tool
```
(high),
```
out_of_order
```
(medium, via
```
must_precede
```
),
```
retry_storm
```
(medium, >N consecutive same-tool failures),
```
excessive_steps
```
(low, > budget),
```
edited_without_read
```
(medium — grounding proxy: an Edit/MultiEdit to a file never Read/Grep'd/Write-authored first; gated by
```
grounding.enabled
```
).
Verdict per run:
```
clean-pass
```
/
```
pass-with-warnings
```
/
```
corrupt-success
```
(delivered and a high-severity flag) /
```
fail
```
(not delivered).

Quarantined behind ONE adapter —

harness/trace-grader.config.yaml

verified:false

the rule PARAMETERS (forbidden list, retry threshold, step budget, order constraints) — project-specific best-guesses, each
```
# ASSUMPTION
```
;
the inferential agent-as-judge axis (task-completion / tool-rationale / planning rubric) —
```
judge_trajectory()
```
is a STUB, not called until a judge model is configured and the file is flipped
```
verified:true
```
.

Adapting later = edit that one file, never the engine.

当前已构建——确定性，无未知项：

Trace schema（契约）： 一次 run = 步骤列表
```
{step, tool, args, observation, ok}
```
。
解析器： 规范的
```
traces.json
```
；由
```
llmwiki/.claude/hooks/hooklib.audit()
```
（以及 code-logger
```
events.jsonl
```
）写入的 audit jsonl 形态；或者完整的 Claude Code session transcript（
```
--transcript
```
）——信息最丰富的来源：可恢复检索参数（Grep
```
pattern
```
、Read
```
file_path
```
）、
```
observation
```
和每一步的
```
ok
```
，因此可以观察 grounding + outcome。Sidechain（sub-agent）行会被跳过；transcript 不携带 task-verdict，因此其
```
run.ok
```
被强制设为 True——应信任 FLAGS。
pass^k
：同一 task 的 k 次 run → 要求全部 k 次都成功交付；报告
```
pass^k
```
。
规则检查（配置驱动）：
```
forbidden_tool
```
（high）、
```
out_of_order
```
（medium，通过
```
must_precede
```
）、
```
retry_storm
```
（medium，同一工具连续失败超过 N 次）、
```
excessive_steps
```
（low，超过预算）、
```
edited_without_read
```
（medium——grounding 代理指标：对某文件执行 Edit/MultiEdit，但该文件此前未被 Read/Grep，也不是先由 Write 创建；由
```
grounding.enabled
```
控制）。
每次运行的 verdict：
```
clean-pass
```
/
```
pass-with-warnings
```
/
```
corrupt-success
```
（已交付且存在 high-severity flag）/
```
fail
```
（未交付）。

隔离在单一 adapter 之后——

harness/trace-grader.config.yaml

，

verified:false

：

规则参数（forbidden list、retry threshold、step budget、order constraints）——项目特定的最佳猜测，每项均标记为
```
# ASSUMPTION
```
；
推断型 agent-as-judge 轴（task-completion / tool-rationale / planning rubric）——
```
judge_trajectory()
```
是一个 STUB，在配置 judge model 并将文件切换至
```
verified:true
```
之前不会调用。

后续适配 = 只修改该文件，绝不修改 engine。

Reference — Files

参考 — 文件

File	Role
`harness/scripts/trace-grader.py`	deterministic engine: schema, parsers, pass^k, rule checks, report, judge STUB
`harness/trace-grader.config.yaml`	the ONE adapter ( `verified:false` ) — every value a `# ASSUMPTION`
`harness/tests/trace-grader.fixtures.json`	self-test fixtures: clean / corrupt-success / flaky

文件	作用
`harness/scripts/trace-grader.py`	确定性 engine：schema、解析器、 `pass^k` 、规则检查、报告、judge STUB
`harness/trace-grader.config.yaml`	唯一的 adapter（ `verified:false` ）——每个值均为 `# ASSUMPTION`
`harness/tests/trace-grader.fixtures.json`	self-test fixtures：clean / corrupt-success / flaky