validate-evaluator

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Validate Evaluator

验证评估器

Calibrate an LLM judge against human judgment.

针对人工判断校准LLM judge。

Overview

概述

Split human-labeled data into train (10-20%), dev (40-45%), test (40-45%)
Run judge on dev set and measure TPR/TNR
Iterate on the judge until TPR and TNR > 90% on dev set
Run once on held-out test set for final TPR/TNR
Apply bias correction formula to production data

将人工标注数据拆分为训练集（10-20%）、开发集（40-45%）、测试集（40-45%）
在开发集上运行judge并测量TPR/TNR
迭代优化judge，直到开发集上的TPR和TNR均超过90%
在预留的测试集上运行一次，获取最终的TPR/TNR
对生产数据应用偏差校正公式

Prerequisites

前提条件

A built LLM judge prompt (from write-judge-prompt)
Human-labeled data: ~100 traces with binary Pass/Fail labels per failure mode
- Aim for ~50 Pass and ~50 Fail (balanced, even if real distribution is skewed)
- Labels must come from a domain expert, not outsourced annotators
Candidate few-shot examples from your labeled data

已编写好LLM judge提示词（来自write-judge-prompt）
人工标注数据：每个故障模式约100条带二元通过/不通过标签的轨迹
- 目标为约50条通过、50条不通过（保持平衡，即使实际分布存在偏差）
- 标签必须来自领域专家，而非外包标注人员
从标注数据中选取的候选少样本示例

Core Instructions

核心步骤

Step 1: Create Data Splits

步骤1：创建数据集拆分

Split human-labeled data into three disjoint sets:

Split	Size	Purpose	Rules
Training	10-20% (~10-20 examples)	Source of few-shot examples for the judge prompt	Only clear-cut Pass and Fail cases. Used directly in the prompt.
Dev	40-45% (~40-45 examples)	Iterative evaluator refinement	Never include in the prompt. Evaluate against repeatedly.
Test	40-45% (~40-45 examples)	Final unbiased accuracy measurement	Do NOT look at during development. Used once at the end.

Target: 30-50 examples of each class (Pass and Fail) across dev and test combined. Use balanced splits even if real-world prevalence is skewed — you need enough Fail examples to measure TNR reliably.

python

from sklearn.model_selection import train_test_split

将人工标注数据拆分为三个互不重叠的集合：

数据集拆分	占比	用途	规则
训练集	10-20%（约10-20个示例）	为judge提示词提供少样本示例来源	仅选用明确的通过/不通过案例，直接用于提示词中。
开发集	40-45%（约40-45个示例）	迭代优化评估器	绝不能用于提示词中，反复用其评估效果。
测试集	40-45%（约40-45个示例）	最终无偏准确率测量	开发过程中请勿查看，仅在最后使用一次。

目标：开发集和测试集合计包含30-50个通过类和不通过类示例。即使实际场景中分布存在偏差，也要使用平衡拆分——你需要足够的不通过示例来可靠测量TNR。

python

from sklearn.model_selection import train_test_split

First split: separate test set

第一次拆分：分离测试集

train_dev, test = train_test_split( labeled_data, test_size=0.4, stratify=labeled_data['label'], random_state=42 )

Second split: separate training examples from dev set

第二次拆分：从开发集中分离训练示例

train, dev = train_test_split( train_dev, test_size=0.75, stratify=train_dev['label'], random_state=42 )

Result: ~15% train, ~45% dev, ~40% test

结果：约15%训练集，~45%开发集，~40%测试集

undefined

undefined

Step 2: Run Evaluator on Dev Set

步骤2：在开发集上运行评估器

Run the judge on every example in the dev set. Compare predictions to human labels.

在开发集的每个示例上运行judge，将预测结果与人工标签对比。

Step 3: Measure TPR and TNR

步骤3：测量TPR和TNR

TPR (True Positive Rate): When a human says Pass, how often does the judge also say Pass?

TPR = (judge says Pass AND human says Pass) / (human says Pass)

TNR (True Negative Rate): When a human says Fail, how often does the judge also say Fail?

TNR = (judge says Fail AND human says Fail) / (human says Fail)

python

from sklearn.metrics import confusion_matrix

tn, fp, fn, tp = confusion_matrix(human_labels, evaluator_labels,
                                   labels=['Fail', 'Pass']).ravel()
tpr = tp / (tp + fn)
tnr = tn / (tn + fp)

Use TPR/TNR, not Precision/Recall or raw accuracy. These two metrics directly map to the bias correction formula. Use Cohen's Kappa only for measuring agreement between two human annotators, not for judge-vs-ground-truth.

**TPR（真阳性率）：**当人工标注为通过时，judge也判定为通过的频率？

TPR = (judge判定通过 AND 人工标注通过) / (人工标注通过的总数)

**TNR（真阴性率）：**当人工标注为不通过时，judge也判定为不通过的频率？

TNR = (judge判定不通过 AND 人工标注不通过) / (人工标注不通过的总数)

python

from sklearn.metrics import confusion_matrix

tn, fp, fn, tp = confusion_matrix(human_labels, evaluator_labels,
                                   labels=['Fail', 'Pass']).ravel()
tpr = tp / (tp + fn)
tnr = tn / (tn + fp)

使用TPR/TNR，而非精确率/召回率或原始准确率。这两个指标直接对应偏差校正公式。仅在衡量两名人工标注员之间的一致性时使用Cohen's Kappa，不要用于judge与真实标签的对比。

Step 4: Inspect Disagreements

步骤4：检查不一致案例

Examine every case where the judge disagrees with human labels:

Disagreement Type	Judge	Human	Fix
False Pass	Pass	Fail	Judge is too lenient. Strengthen Fail definitions or add edge-case examples.
False Fail	Fail	Pass	Judge is too strict. Clarify Pass definitions or adjust examples.

For each disagreement, determine whether to:

Clarify wording in the judge prompt
Swap or add few-shot examples from the training set
Add explicit rules for the edge case
Split the criterion into more specific sub-checks

检查所有judge与人工标签不一致的案例：

不一致类型	Judge判定	人工标注	修复方法
误判通过	通过	不通过	Judge过于宽松。强化不通过的定义或添加边缘案例示例。
误判不通过	不通过	通过	Judge过于严格。明确通过的定义或调整示例。

针对每个不一致案例，决定是否：

明确judge提示词中的措辞
替换或添加训练集中的少样本示例
为边缘案例添加明确规则
将判定标准拆分为更具体的子检查项

Step 5: Iterate

步骤5：迭代优化

Refine the judge prompt and re-run on the dev set. Repeat until TPR and TNR stabilize.

Stopping criteria:

Target: TPR > 90% AND TNR > 90%
Minimum acceptable: TPR > 80% AND TNR > 80%

If alignment stalls:

Problem	Solution
TPR and TNR both low	Use a more capable LLM for the judge
One metric low, one acceptable	Inspect disagreements for the low metric specifically
Both plateau below target	Decompose the criterion into smaller, more atomic checks
Consistently wrong on certain input types	Add targeted few-shot examples from training set
Labels themselves seem inconsistent	Re-examine human labels; the rubric may need refinement

优化judge提示词并重新在开发集上运行。重复此过程直到TPR和TNR稳定。

停止标准：

**目标：**TPR > 90% 且 TNR > 90%
**最低可接受标准：**TPR > 80% 且 TNR > 80%

若一致性停滞不前：

问题	解决方案
TPR和TNR均较低	使用能力更强的LLM作为judge
一个指标低，另一个可接受	专门检查低指标对应的不一致案例
两者均停滞在目标以下	将判定标准分解为更小、更原子化的检查项
对特定输入类型持续误判	从训练集中添加针对性的少样本示例
标签本身似乎不一致	重新检查人工标签；评分标准可能需要优化

Step 6: Final Measurement on Test Set

步骤6：在测试集上进行最终测量

Run the judge exactly once on the held-out test set. Record final TPR and TNR.

Do not iterate after seeing test set results. Go back to step 4 with new dev data if needed.

在预留的测试集上仅运行一次judge，记录最终的TPR和TNR。

看到测试集结果后请勿再迭代优化。若需要，使用新的开发数据回到步骤4。

Step 7 (Optional): Estimate True Success Rate (Rogan-Gladen Correction)

步骤7（可选）：估算真实通过率（Rogan-Gladen校正）

Raw judge scores on unlabeled production data are biased. If you need an accurate aggregate pass rate, correct for known judge errors:

theta_hat = (p_obs + TNR - 1) / (TPR + TNR - 1)

Where:

```
p_obs
```
= fraction of unlabeled traces the judge scored as Pass
```
TPR
```
,
```
TNR
```
= from test set measurement
```
theta_hat
```
= corrected estimate of true success rate

Clip to [0, 1]. Invalid when TPR + TNR - 1 is near 0 (judge is no better than random).

Example:

Judge TPR = 0.92, TNR = 0.88
500 production traces: 400 scored Pass -> p_obs = 0.80
theta_hat = (0.80 + 0.88 - 1) / (0.92 + 0.88 - 1) = 0.68 / 0.80 = 0.85
True success rate is ~85%, not the raw 80%

未标注生产数据上的原始judge评分存在偏差。若需要准确的汇总通过率，请针对已知的judge误差进行校正：

theta_hat = (p_obs + TNR - 1) / (TPR + TNR - 1)

其中：

```
p_obs
```
= judge判定为通过的未标注轨迹占比
```
TPR
```
,
```
TNR
```
= 测试集测量得到的值
```
theta_hat
```
= 校正后的真实通过率估算值

将结果限制在[0, 1]范围内。当TPR + TNR - 1接近0时（judge表现与随机无异），此公式无效。

示例：

Judge的TPR = 0.92，TNR = 0.88
500条生产轨迹：400条被判定为通过 -> p_obs = 0.80
theta_hat = (0.80 + 0.88 - 1) / (0.92 + 0.88 - 1) = 0.68 / 0.80 = 0.85
真实通过率约为85%，而非原始的80%

Step 8: Confidence Interval

步骤8：置信区间

Compute a bootstrap confidence interval. A point estimate alone is not enough.

python

import numpy as np

def bootstrap_ci(human_labels, eval_labels, p_obs, n_bootstrap=2000):
    """Bootstrap 95% CI for corrected success rate."""
    n = len(human_labels)
    estimates = []
    for _ in range(n_bootstrap):
        idx = np.random.choice(n, size=n, replace=True)
        h = np.array(human_labels)[idx]
        e = np.array(eval_labels)[idx]

        tp = ((h == 'Pass') & (e == 'Pass')).sum()
        fn = ((h == 'Pass') & (e == 'Fail')).sum()
        tn = ((h == 'Fail') & (e == 'Fail')).sum()
        fp = ((h == 'Fail') & (e == 'Pass')).sum()

        tpr_b = tp / (tp + fn) if (tp + fn) > 0 else 0
        tnr_b = tn / (tn + fp) if (tn + fp) > 0 else 0
        denom = tpr_b + tnr_b - 1

        if abs(denom) < 1e-6:
            continue
        theta = (p_obs + tnr_b - 1) / denom
        estimates.append(np.clip(theta, 0, 1))

    return np.percentile(estimates, 2.5), np.percentile(estimates, 97.5)

lower, upper = bootstrap_ci(test_human, test_eval, p_obs=0.80)
print(f"95% CI: [{lower:.2f}, {upper:.2f}]")

Or use

judgy

(

pip install judgy

python

from judgy import estimate_success_rate

计算bootstrap置信区间。仅有点估计值是不够的。

python

import numpy as np

def bootstrap_ci(human_labels, eval_labels, p_obs, n_bootstrap=2000):
    """为校正后的通过率计算95%自助法置信区间。"""
    n = len(human_labels)
    estimates = []
    for _ in range(n_bootstrap):
        idx = np.random.choice(n, size=n, replace=True)
        h = np.array(human_labels)[idx]
        e = np.array(eval_labels)[idx]

        tp = ((h == 'Pass') & (e == 'Pass')).sum()
        fn = ((h == 'Pass') & (e == 'Fail')).sum()
        tn = ((h == 'Fail') & (e == 'Fail')).sum()
        fp = ((h == 'Fail') & (e == 'Pass')).sum()

        tpr_b = tp / (tp + fn) if (tp + fn) > 0 else 0
        tnr_b = tn / (tn + fp) if (tn + fp) > 0 else 0
        denom = tpr_b + tnr_b - 1

        if abs(denom) < 1e-6:
            continue
        theta = (p_obs + tnr_b - 1) / denom
        estimates.append(np.clip(theta, 0, 1))

    return np.percentile(estimates, 2.5), np.percentile(estimates, 97.5)

lower, upper = bootstrap_ci(test_human, test_eval, p_obs=0.80)
print(f"95% CI: [{lower:.2f}, {upper:.2f}]")

或使用

judgy

库（

pip install judgy

）：

python

from judgy import estimate_success_rate

judgy expects 0/1 integer labels (1 = Pass, 0 = Fail)

judgy要求使用0/1整数标签（1 = 通过，0 = 不通过）

test_labels = [1 if l == 'Pass' else 0 for l in test_human_labels] test_preds = [1 if l == 'Pass' else 0 for l in test_eval_labels] unlabeled_preds = [1 if l == 'Pass' else 0 for l in prod_eval_labels]

theta_hat, lower, upper = estimate_success_rate( test_labels, test_preds, unlabeled_preds ) print(f"Corrected rate: {theta_hat:.2f}") print(f"95% CI: [{lower:.2f}, {upper:.2f}]")

undefined

theta_hat, lower, upper = estimate_success_rate( test_labels, test_preds, unlabeled_preds ) print(f"校正后通过率: {theta_hat:.2f}") print(f"95% CI: [{lower:.2f}, {upper:.2f}]")

undefined

Practical Guidance

实用指南

Pin exact model versions for LLM judges (a dated snapshot id like
```
<model>-<YYYY-MM-DD>
```
, not a floating alias). Providers update models without notice, causing silent drift.
Re-validate after changing the judge prompt, switching models, or when production confidence intervals widen unexpectedly.
Use ~100 labeled examples (50 Pass, 50 Fail). Below 60, confidence intervals become wide.
One trusted domain expert is the most efficient labeling path. If not feasible, have two annotators label 20-50 traces independently and resolve disagreements before proceeding.
Improving TPR narrows the confidence interval more than improving TNR. The correction divides by
```
(TPR + TNR - 1)
```
, so a low TPR shrinks the denominator and amplifies estimation errors into wide CIs.

固定LLM judge的精确模型版本（使用类似
```
<model>-<YYYY-MM-DD>
```
的日期快照ID，而非浮动别名）。服务商可能会在无通知的情况下更新模型，导致性能悄然漂移。
重新验证：在修改judge提示词、切换模型或生产环境置信区间意外扩大后，需重新验证。
使用约100条标注示例（50条通过，50条不通过）。少于60条时，置信区间会变得很宽。
一名可信的领域专家是最高效的标注途径。若不可行，让两名标注员独立标注20-50条轨迹，解决分歧后再继续。
提升TPR比提升TNR更能缩小置信区间。校正公式除以
```
(TPR + TNR - 1)
```
，因此低TPR会缩小分母，将估计误差放大为更宽的置信区间。

Anti-Patterns

反模式

Assuming judges "just work" without validation. A judge may consistently miss failures or flag passing traces.
Using raw accuracy or percent agreement. Use TPR and TNR. With class imbalance, raw accuracy is misleading.
Dev/test examples as few-shot examples. This is data leakage.
Reporting dev set performance as final accuracy. Dev numbers are optimistic. The test set gives the unbiased estimate.
Raw judge scores without bias correction. If you report an aggregate pass rate, apply the Rogan-Gladen formula (Step 7).
Point estimates without confidence intervals. A corrected rate of 85% could easily be 78-92% with small test sets. Report the range so stakeholders know how much to trust the number.

假设judge无需验证就能正常工作。judge可能会持续遗漏故障或误判通过轨迹。
使用原始准确率或一致率。请使用TPR和TNR。存在类别不平衡时，原始准确率具有误导性。
将开发/测试示例用作少样本示例。这属于数据泄露。
将开发集性能报告为最终准确率。开发集数据过于乐观，测试集才能提供无偏估计。
不进行偏差校正直接使用原始judge评分。若要报告汇总通过率，请应用Rogan-Gladen公式（步骤7）。
仅提供点估计值而无置信区间。校正后的85%通过率，在测试集较小时可能实际范围是78-92%。请报告范围，以便利益相关者了解该数值的可信度。