write-code-eval
Compare original and translation side by side
🇺🇸
Original
English🇨🇳
Translation
ChineseWrite a code evaluator
编写代码评估器
Start with a failure mode found through error analysis. Write one check for that failure mode, much like a unit test that asserts what should hold for each trace.
- State the rule and identify the trace fields or reference data the check needs. If the rule requires interpretation, use .
write-judge-prompt - Implement the check in the project's language and eval framework. Return a result and a reason in the format that framework expects.
- Test known passes and failures, including borderline cases. Run the check on available traces and inspect mistakes. If the rule uses a proxy for human judgment, compare its results with human labels.
从通过错误分析发现的失败模式开始。为该失败模式编写一条检查规则，类似于对每个 trace 应该满足的条件进行断言的单元测试。
- 明确规则，并确定检查所需的 trace 字段或参考数据。如果规则需要解释，请使用 。
write-judge-prompt - 使用项目的语言和 eval 框架实现该检查。以框架期望的格式返回结果和原因。
- 测试已知的通过和失败用例，包括边界情况。在可用的 traces 上运行检查并审查错误。如果规则使用代理作为人工判断，将其结果与人工标签进行比较。
Examples
示例
| Failure mode | Possible check |
|---|---|
| Invalid output structure | Parse the output and check required fields |
| Missing or forbidden text | Match a string or pattern |
| Citation not in retrieved documents | Compare cited IDs with retrieved IDs |
| Bad tool call | Check arguments against the tool schema or run the call in a safe test environment |
| Wrong value | Compare the output with a reference value |
Choose the check from the failure rule. For a failure with both objective and interpretive parts, check the objective part with code and use a judge for the rest.
| 失败模式 | 可能的检查 |
|---|---|
| 无效输出结构 | 解析输出并检查必填字段 |
| 缺少或禁止出现的文本 | 匹配字符串或模式 |
| 引用不在检索文档中 | 比较引用的 ID 与检索到的 ID |
| 错误的工具调用 | 根据工具 schema 检查参数，或在安全测试环境中运行调用 |
| 错误的值 | 将输出与参考值比较 |
根据失败规则选择检查方式。对于同时包含客观和解释性部分的失败，用代码检查客观部分，其余部分使用 judge。