advanced-evaluation

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Advanced Evaluation

高级评估

This skill covers production-grade techniques for evaluating LLM outputs using LLMs as judges. It synthesizes research from academic papers, industry practices, and practical implementation experience into actionable patterns for building reliable evaluation systems.

Key insight: LLM-as-a-Judge is not a single technique but a family of approaches, each suited to different evaluation contexts. Choosing the right approach and mitigating known biases is the core competency this skill develops.

此 skill 涵盖使用 LLM 作为评判者来评估 LLM 输出的生产级技术。它综合了学术论文、行业实践和实际实施经验中的研究成果，并将其提炼为构建可靠评估系统的可执行模式。

核心洞察：LLM-as-a-Judge 并非一项单一技术，而是一组适用于不同评估场景的方法。选择合适的方法并缓解已知偏差，是本 skill 所培养的核心能力。

When to Activate

何时启用

Activate this skill when:

Building LLM-as-judge systems for LLM outputs
Comparing multiple model responses to select the best one
Establishing consistent quality standards across evaluation teams
Debugging evaluation systems that show inconsistent results
Designing A/B tests for prompt or model changes
Creating rubrics specifically for LLM or human/LLM hybrid judges
Analyzing correlation between automated and human judgments

Do not activate this skill for adjacent work owned by other skills:

General deterministic checks, regression suites, production quality gates, or outcome metrics:
```
evaluation
```
.
Autonomous loop governance, locked rubrics, rollback, or PR approval boundaries:
```
harness-engineering
```
.
Tool API contracts for evaluation tools:
```
tool-design
```
.

在以下情况下启用此 skill：

为 LLM 输出构建 LLM-as-judge 系统
比较多个模型回答并选出最佳回答
在评估团队之间建立一致的质量标准
调试结果不一致的评估系统
为 prompt 或 model 变更设计 A/B 测试
专门为 LLM 或 human/LLM hybrid judges 创建 rubric
分析自动判断与人工判断之间的相关性

请勿为以下由其他 skill 负责的相邻工作启用此 skill：

常规确定性检查、回归测试套件、生产质量门禁或结果指标：
```
evaluation
```
autonomous loop 治理、锁定的 rubric、回滚或 PR 审批边界：
```
harness-engineering
```
评估工具的 API 合约：
```
tool-design
```

Core Concepts

核心概念

The Evaluation Taxonomy

评估分类体系

Select between two primary approaches based on whether ground truth exists:

Direct Scoring — Use when objective criteria exist (factual accuracy, instruction following, toxicity). A single LLM rates one response on a defined scale. Achieves moderate-to-high reliability for well-defined criteria. Watch for score calibration drift and inconsistent scale interpretation.

Pairwise Comparison — Use for subjective preferences (tone, style, persuasiveness). An LLM compares two responses and selects the better one. Pairwise methods often correlate better with human preference than open-ended direct scoring for subjective tasks (claim-advanced-evaluation-position-swap). Watch for position bias and length bias.

根据是否存在 ground truth，在两种主要方法之间进行选择：

Direct Scoring — 当存在客观标准时使用（事实准确性、指令遵循度、毒性）。由单个 LLM 按既定量表为一个回答评分。对于定义明确的标准，可以获得中等到较高的可靠性。需要注意评分校准漂移和量表解读不一致。

Pairwise Comparison — 当需要判断主观偏好时使用（语气、风格、说服力）。由 LLM 比较两个回答并选出更优者。对于主观任务，Pairwise 方法与人类偏好的相关性通常高于开放式 Direct Scoring（

claim-advanced-evaluation-position-swap

）。需要警惕 position bias 和 length bias。

The Bias Landscape

偏差图谱

Mitigate these systematic biases in every evaluation system:

Position Bias: First-position responses get preferential treatment. Mitigate by evaluating twice with swapped positions, then apply majority vote or consistency check.

Length Bias: Longer responses score higher regardless of quality. Mitigate by explicitly prompting to ignore length and applying length-normalized scoring.

Self-Enhancement Bias: Models rate their own outputs higher. Mitigate by using different models for generation and evaluation.

Verbosity Bias: Excessive detail scores higher even when unnecessary. Mitigate with criteria-specific rubrics that penalize irrelevant detail.

Authority Bias: Confident tone scores higher regardless of accuracy. Mitigate by requiring evidence citation and adding a fact-checking layer.

在每个评估系统中都应缓解以下系统性偏差：

Position Bias：位于首位的回答会获得偏爱。缓解方式是交换位置进行两次评估，然后采用多数投票或一致性检查。

Length Bias：较长的回答无论质量如何都会获得更高分数。缓解方式是在 prompt 中明确要求忽略长度，并采用长度归一化评分。

Self-Enhancement Bias：模型会给自己的输出打更高分。缓解方式是在生成和评估环节使用不同的模型。

Verbosity Bias：过多的细节即使没有必要也会获得更高分。缓解方式是使用针对具体标准的 rubric，对无关细节进行惩罚。

Authority Bias：自信的语气无论准确性如何都会获得更高分。缓解方式是要求引用证据，并增加事实核查层。

Metric Selection Framework

指标选择框架

Match metrics to the evaluation task structure:

Task Type	Primary Metrics	Secondary Metrics
Binary classification (pass/fail)	Recall, Precision, F1	Cohen's kappa
Ordinal scale (1-5 rating)	Spearman's rho, Kendall's tau	Cohen's kappa (weighted)
Pairwise preference	Agreement rate, Position consistency	Confidence calibration
Multi-label	Macro-F1, Micro-F1	Per-label precision/recall

Prioritize systematic disagreement patterns over absolute agreement rates because a judge that consistently disagrees with humans on specific criteria is more problematic than one with random noise.

使指标与评估任务结构相匹配：

任务类型	主要指标	次要指标
二分类（pass/fail）	Recall, Precision, F1	Cohen's kappa
序数尺度（1-5 rating）	Spearman's rho, Kendall's tau	Cohen's kappa (weighted)
成对偏好	Agreement rate, Position consistency	Confidence calibration
多标签	Macro-F1, Micro-F1	Per-label precision/recall

应优先关注系统性的分歧模式，而非绝对一致率，因为一个始终在特定标准上与人类判断不一致的 judge，比存在随机噪声的 judge 更成问题。

Evaluation Approaches

评估方法

Direct Scoring Implementation

Direct Scoring 实现

Build direct scoring with three components: clear criteria, a calibrated scale, and structured output format.

Criteria Definition Pattern:

Criterion: [Name]
Description: [What this criterion measures]
Weight: [Relative importance, 0-1]

Scale Calibration — Choose scale granularity based on rubric detail:

1-3: Binary with neutral option, lowest cognitive load
1-5: Standard Likert, best balance of granularity and reliability
1-10: Use only with detailed per-level rubrics because calibration is harder

Prompt Structure for Direct Scoring:

You are an expert evaluator assessing response quality.

构建 Direct Scoring 时需要三个组成部分：清晰的标准、经过校准的量表和结构化输出格式。

标准定义模式：

Criterion: [Name]
Description: [What this criterion measures]
Weight: [Relative importance, 0-1]

量表校准 — 根据 rubric 的详细程度选择量表粒度：

1-3：带中性选项的二元量表，认知负担最低
1-5：标准 Likert，在粒度和可靠性之间达到最佳平衡
1-10：仅在每个等级都有详细 rubric 时使用，因为校准更加困难

Direct Scoring 的 Prompt 结构：

You are an expert evaluator assessing response quality.

Task

Evaluate the following response against each criterion.

Original Prompt

{prompt}

Response to Evaluate

{response}

Criteria

{for each criterion: name, description, weight}

Instructions

For each criterion:

Find specific evidence in the response
Score according to the rubric (1-{max} scale)
Justify your score with evidence
Suggest one specific improvement

For each criterion:

Find specific evidence in the response
Score according to the rubric (1-{max} scale)
Justify your score with evidence
Suggest one specific improvement

Output Format

Respond with structured JSON containing scores, justifications, and summary.


Require evidence before the score in scoring prompts so the judge must anchor its decision in observable output features before emitting a number.

Respond with structured JSON containing scores, justifications, and summary.


在 Direct Scoring prompt 中要求先提供证据、再给出分数，以确保 judge 必须先依据可观察的输出特征作出判断，然后才输出数字。

Pairwise Comparison Implementation

Pairwise Comparison 实现

Apply position bias mitigation in every pairwise evaluation:

Run deterministic pre-checks first: both candidates must satisfy the same schema, source-evidence requirements, and scope constraints.
First judge pass: Response A in first position, Response B in second.
Second judge pass: Response B in first position, Response A in second.
Consistency check: If passes disagree, return TIE with reduced confidence.
Final verdict: Consistent winner with averaged confidence and explicit tie-breaker rationale.

Prompt Structure for Pairwise Comparison:

You are an expert evaluator comparing two AI responses.

在每次 Pairwise Evaluation 中都应应用 position bias 缓解策略：

首先运行确定性预检：两个候选项必须满足相同的 schema、source-evidence 要求和 scope 约束。
第一轮评判：Response A 位于首位，Response B 位于次位。
第二轮评判：Response B 位于首位，Response A 位于次位。
一致性检查：如果两轮结果不一致，则返回 TIE，并降低 confidence。
最终判定：给出一致的获胜者，同时提供平均 confidence 和明确的 tie-breaker 理由。

Pairwise Comparison 的 Prompt 结构：

You are an expert evaluator comparing two AI responses.

Critical Instructions

Do NOT prefer responses because they are longer
Do NOT prefer responses based on position (first vs second)
Focus ONLY on quality according to the specified criteria
Ties are acceptable when responses are genuinely equivalent

Do NOT prefer responses because they are longer
Do NOT prefer responses based on position (first vs second)
Focus ONLY on quality according to the specified criteria
Ties are acceptable when responses are genuinely equivalent

Original Prompt

{prompt}

Response A

{response_a}

Response B

{response_b}

Comparison Criteria

{criteria list}

Instructions

Analyze each response independently first
Compare them on each criterion
Determine overall winner with confidence level

Analyze each response independently first
Compare them on each criterion
Determine overall winner with confidence level

Output Format

JSON with per-criterion comparison, overall winner, confidence (0-1), and reasoning.


**Confidence Calibration** — Map confidence to position consistency:
- Both passes agree: confidence = average of individual confidences
- Passes disagree: confidence = 0.5, verdict = TIE

JSON with per-criterion comparison, overall winner, confidence (0-1), and reasoning.


**Confidence 校准** — 根据位置一致性映射 confidence：
- 两轮评判一致：confidence = 两次单独 confidence 的平均值
- 两轮评判不一致：confidence = 0.5，verdict = TIE

Rubric Generation

Rubric 生成

Generate rubrics to reduce evaluation variance compared to open-ended scoring. Treat exact variance reduction as workload-specific unless measured on the target eval set.

Include these rubric components:

Level descriptions: Clear boundaries for each score level
Characteristics: Observable features that define each level
Examples: Representative text for each level (optional but valuable)
Edge cases: Guidance for ambiguous situations
Scoring guidelines: General principles for consistent application

Set strictness calibration for the use case:

Lenient: Lower passing bar, appropriate for encouraging iteration
Balanced: Typical production expectations
Strict: High standards for safety-critical or high-stakes evaluation

Adapt rubrics to the domain — use domain-specific terminology. A code readability rubric mentions variables, functions, and comments. A medical accuracy rubric references clinical terminology and evidence standards.

生成 rubric 可以降低评估方差，相比开放式评分更加稳定。除非在目标 eval set 上进行了测量，否则应将具体的方差降幅视为取决于工作负载。

加入以下 rubric 组成部分：

等级描述：为每个分数等级提供清晰的边界
特征：定义每个等级的可观察特征
示例：为每个等级提供代表性文本（可选，但非常有价值）
边界情况：为模糊情形提供处理指引
评分指南：提供确保一致应用的基本原则

根据使用场景设置严格程度（strictness）校准：

Lenient：降低通过门槛，适合鼓励迭代
Balanced：典型的生产环境期望
Strict：适用于安全关键或高风险评估的高标准

使 rubric 适配具体领域——使用领域专用术语。代码可读性 rubric 会提及 variables、functions 和 comments；医学准确性 rubric 会引用 clinical terminology 和 evidence standards。

Practical Guidance

实践指导

Evaluation Pipeline Design

评估流水线设计

Build production evaluation systems with these layers: Criteria Loader (rubrics + weights) -> Primary Scorer (direct or pairwise) -> Bias Mitigation (position swap, etc.) -> Confidence Scoring (calibration) -> Output (scores + justifications + confidence). See Evaluation Pipeline Diagram for the full visual layout.

构建生产级评估系统时，应采用以下分层：Criteria Loader（rubrics + weights）-> Primary Scorer（direct 或 pairwise）-> Bias Mitigation（position swap 等）-> Confidence Scoring（calibration）-> Output（scores + justifications + confidence）。有关完整的可视化布局，请参阅评估流水线图。

Decision Framework: Direct vs. Pairwise

决策框架：Direct 与 Pairwise

Apply this decision tree:

Is there an objective ground truth?
+-- Yes -> Direct Scoring
|   Examples: factual accuracy, instruction following, format compliance
|
+-- No -> Is it a preference or quality judgment?
    +-- Yes -> Pairwise Comparison
    |   Examples: tone, style, persuasiveness, creativity
    |
    +-- No -> Consider reference-based evaluation
        Examples: summarization (compare to source), translation (compare to reference)

应用以下决策树：

Is there an objective ground truth?
+-- Yes -> Direct Scoring
|   Examples: factual accuracy, instruction following, format compliance
|
+-- No -> Is it a preference or quality judgment?
    +-- Yes -> Pairwise Comparison
    |   Examples: tone, style, persuasiveness, creativity
    |
    +-- No -> Consider reference-based evaluation
        Examples: summarization (compare to source), translation (compare to reference)

Scaling Evaluation

扩展评估

For high-volume evaluation, apply one of these strategies:

Panel of LLMs (PoLL): Use multiple models as judges and aggregate votes to reduce individual model bias. More expensive but more reliable for high-stakes decisions.
Hierarchical evaluation: Use a fast cheap model for screening and an expensive model for edge cases. Requires calibration of the screening threshold.
Human-in-the-loop: Automate clear cases and route low-confidence decisions to human review. Design feedback loops to improve automated evaluation over time.

对于大批量评估，采用以下策略之一：

LLM 评审团（Panel of LLMs, PoLL）：使用多个模型作为 judges 并汇总投票，以减少单个模型的偏差。成本更高，但对于高风险决策更加可靠。
分层评估（Hierarchical evaluation）：使用快速且低成本的模型进行筛查，再使用高成本模型处理边界情况。需要对筛查阈值进行校准。
Human-in-the-loop：自动处理明确案例，并将低 confidence 决策转交人工审核。设计反馈循环，以便随着时间推移持续改进自动评估。

Examples

示例

Example 1: Direct Scoring for Accuracy

示例 1：用于准确性评估的 Direct Scoring

Input:

Prompt: "What causes seasons on Earth?"
Response: "Seasons are caused by Earth's tilted axis. As Earth orbits the Sun,
different hemispheres receive more direct sunlight at different times of year."
Criterion: Factual Accuracy (weight: 1.0)
Scale: 1-5

Output:

json

{
  "criterion": "Factual Accuracy",
  "score": 5,
  "evidence": [
    "Correctly identifies axial tilt as primary cause",
    "Correctly explains differential sunlight by hemisphere",
    "No factual errors present"
  ],
  "justification": "Response accurately explains the cause of seasons with correct
scientific reasoning. Both the axial tilt and its effect on sunlight distribution
are correctly described.",
  "improvement": "Could add the specific tilt angle (23.5 degrees) for completeness."
}

输入：

Prompt: "What causes seasons on Earth?"
Response: "Seasons are caused by Earth's tilted axis. As Earth orbits the Sun,
different hemispheres receive more direct sunlight at different times of year."
Criterion: Factual Accuracy (weight: 1.0)
Scale: 1-5

输出：

json

{
  "criterion": "Factual Accuracy",
  "score": 5,
  "evidence": [
    "Correctly identifies axial tilt as primary cause",
    "Correctly explains differential sunlight by hemisphere",
    "No factual errors present"
  ],
  "justification": "Response accurately explains the cause of seasons with correct
scientific reasoning. Both the axial tilt and its effect on sunlight distribution
are correctly described.",
  "improvement": "Could add the specific tilt angle (23.5 degrees) for completeness."
}

Example 2: Pairwise Comparison with Position Swap

示例 2：带位置交换的 Pairwise Comparison

Input:

Prompt: "Explain machine learning to a beginner"
Response A: [Technical explanation with jargon]
Response B: [Simple analogy-based explanation]
Criteria: ["clarity", "accessibility"]

First Pass (A first):

json

{ "winner": "B", "confidence": 0.8 }

Second Pass (B first):

json

{ "winner": "A", "confidence": 0.6 }

(Note: Winner is A because B was in first position)

Mapped Second Pass:

json

{ "winner": "B", "confidence": 0.6 }

Final Result:

json

{
  "winner": "B",
  "confidence": 0.7,
  "positionConsistency": {
    "consistent": true,
    "firstPassWinner": "B",
    "secondPassWinner": "B"
  }
}

输入：

Prompt: "Explain machine learning to a beginner"
Response A: [Technical explanation with jargon]
Response B: [Simple analogy-based explanation]
Criteria: ["clarity", "accessibility"]

第一轮（A 在前）：

json

{ "winner": "B", "confidence": 0.8 }

第二轮（B 在前）：

json

{ "winner": "A", "confidence": 0.6 }

注意：由于 B 位于首位，获胜者是 A。

映射后的第二轮结果：

json

{ "winner": "B", "confidence": 0.6 }

最终结果：

json

{
  "winner": "B",
  "confidence": 0.7,
  "positionConsistency": {
    "consistent": true,
    "firstPassWinner": "B",
    "secondPassWinner": "B"
  }
}

Example 3: Rubric Generation

示例 3：Rubric 生成

Input:

criterionName: "Code Readability"
criterionDescription: "How easy the code is to understand and maintain"
domain: "software engineering"
scale: "1-5"
strictness: "balanced"

Output (abbreviated):

json

{
  "levels": [
    {
      "score": 1,
      "label": "Poor",
      "description": "Code is difficult to understand without significant effort",
      "characteristics": [
        "No meaningful variable or function names",
        "No comments or documentation",
        "Deeply nested or convoluted logic"
      ]
    },
    {
      "score": 3,
      "label": "Adequate",
      "description": "Code is understandable with some effort",
      "characteristics": [
        "Most variables have meaningful names",
        "Basic comments present for complex sections",
        "Logic is followable but could be cleaner"
      ]
    },
    {
      "score": 5,
      "label": "Excellent",
      "description": "Code is immediately clear and maintainable",
      "characteristics": [
        "All names are descriptive and consistent",
        "Comprehensive documentation",
        "Clean, modular structure"
      ]
    }
  ],
  "edgeCases": [
    {
      "situation": "Code is well-structured but uses domain-specific abbreviations",
      "guidance": "Score based on readability for domain experts, not general audience"
    }
  ]
}

输入：

criterionName: "Code Readability"
criterionDescription: "How easy the code is to understand and maintain"
domain: "software engineering"
scale: "1-5"
strictness: "balanced"

输出（节选）：

json

{
  "levels": [
    {
      "score": 1,
      "label": "Poor",
      "description": "Code is difficult to understand without significant effort",
      "characteristics": [
        "No meaningful variable or function names",
        "No comments or documentation",
        "Deeply nested or convoluted logic"
      ]
    },
    {
      "score": 3,
      "label": "Adequate",
      "description": "Code is understandable with some effort",
      "characteristics": [
        "Most variables have meaningful names",
        "Basic comments present for complex sections",
        "Logic is followable but could be cleaner"
      ]
    },
    {
      "score": 5,
      "label": "Excellent",
      "description": "Code is immediately clear and maintainable",
      "characteristics": [
        "All names are descriptive and consistent",
        "Comprehensive documentation",
        "Clean, modular structure"
      ]
    }
  ],
  "edgeCases": [
    {
      "situation": "Code is well-structured but uses domain-specific abbreviations",
      "guidance": "Score based on readability for domain experts, not general audience"
    }
  ]
}

Guidelines

指南

Always require evidence before scores - Evidence-first prompts make judgments easier to audit and reduce ungrounded numeric scoring
Always swap positions in pairwise comparison - Single-pass comparison is corrupted by position bias
Match scale granularity to rubric specificity - Don't use 1-10 without detailed level descriptions
Separate objective and subjective criteria - Use direct scoring for objective, pairwise for subjective
Include confidence scores - Calibrate to position consistency and evidence strength
Define edge cases explicitly - Ambiguous situations cause the most evaluation variance
Use domain-specific rubrics - Generic rubrics produce generic (less useful) evaluations
Validate against human judgments - Automated evaluation is only valuable if it correlates with human assessment
Monitor for systematic bias - Track disagreement patterns by criterion, response type, model
Design for iteration - Evaluation systems improve with feedback loops

始终要求先提供证据，再给出分数 - evidence-first 的 prompt 更容易审计判断过程，并减少缺乏依据的数值评分
始终在 Pairwise Comparison 中交换位置 - 单轮比较会受到 position bias 污染
使 scale 粒度与 rubric 的具体程度匹配 - 如果没有详细的等级描述，不要使用 1-10 量表
区分客观标准与主观标准 - 客观标准使用 Direct Scoring，主观标准使用 Pairwise Comparison
包含 confidence score - 根据位置一致性和证据强度进行校准
明确定义边界情况 - 模糊情形是评估方差的主要来源
使用领域专用 rubric - 通用 rubric 只能产生通用且实用性较低的评估
根据人工判断进行验证 - 只有当自动评估与人工评估相关时，它才具有价值
监控系统性偏差 - 按标准、回答类型和模型跟踪分歧模式
面向迭代进行设计 - 评估系统应通过反馈循环持续改进

Gotchas

常见陷阱

Scoring without justification: Scores lack grounding and are difficult to debug. Always require evidence-based justification before the score.
Single-pass pairwise comparison: Position bias corrupts results when positions are not swapped. Always evaluate twice with swapped positions and check consistency.
Overloaded criteria: Criteria that measure multiple things at once produce unreliable scores. Enforce one criterion = one measurable aspect.
Missing edge case guidance: Evaluators handle ambiguous cases inconsistently without explicit instructions. Include edge cases in rubrics with clear resolution rules.
Ignoring confidence calibration: High-confidence wrong judgments are worse than low-confidence ones. Calibrate confidence to position consistency and evidence strength.
Rubric drift: Rubrics become miscalibrated as quality standards evolve or model capabilities improve. Schedule periodic rubric reviews and re-anchor score levels against fresh human-annotated examples.
Evaluation prompt sensitivity: Minor wording changes in evaluation prompts can cause material score swings. Version-control evaluation prompts and run regression tests before deploying prompt changes.
Uncontrolled length bias: Longer responses systematically score higher even when conciseness is preferred. Add explicit length-neutrality instructions to evaluation prompts and validate with length-controlled test pairs.

评分缺少依据：分数缺乏支撑，难以调试。始终要求在分数之前提供基于证据的理由。
单轮 Pairwise Comparison：不交换位置时，position bias 会污染结果。始终交换位置进行两轮评估并检查一致性。
标准过载：同时衡量多个维度的标准会产生不可靠的分数。必须坚持一个标准只衡量一个可观测方面。
缺少边界情况指引：如果没有明确说明，评估者会不一致地处理模糊情况。应在 rubric 中加入边界情况及明确的解决规则。
忽视 Confidence 校准：高 confidence 的错误判断比低 confidence 的判断更糟糕。根据位置一致性和证据强度校准 confidence。
Rubric 漂移：随着质量标准演变或模型能力提升，rubric 会逐渐失去校准。应定期审查 rubric，并使用新的人工标注示例重新校准分数等级。
评估 Prompt 敏感性：评估 prompt 中的细微措辞变化都可能导致分数大幅波动。对评估 prompt 进行版本控制，并在部署变更前运行回归测试。
未控制 Length Bias：即使偏好简洁的回答，较长的回答仍会系统性地获得更高分数。在评估 prompt 中加入明确的长度中立性指令，并使用长度受控的测试对进行验证。

Integration

集成

This skill owns judge design and bias mitigation. Adjacent skills own broader quality gates and infrastructure:

```
evaluation
```
: general deterministic checks, regression suites, quality gates, and production monitoring.
```
context-fundamentals
```
: context structure for judge prompts.
```
tool-design
```
: schemas and error handling for evaluation tools.
```
context-optimization
```
: token and latency efficiency for high-volume evals.
```
harness-engineering
```
: locked evaluator surfaces and governance for autonomous loops.

此 skill 负责 judge 设计和偏差缓解。更广泛的质量门禁和基础设施由相邻 skill 负责：

```
evaluation
```
：常规确定性检查、回归测试套件、质量门禁和生产监控
```
context-fundamentals
```
：judge prompt 的 context 结构
```
tool-design
```
：评估工具的 schema 和错误处理
```
context-optimization
```
：面向大批量 eval 的 token 和延迟效率
```
harness-engineering
```
：面向 autonomous loop 的锁定 evaluator surface 和治理

References

参考资料

Internal reference:

LLM-as-Judge Implementation Patterns - Read when: building an evaluation pipeline from scratch or integrating LLM judges into CI/CD
Bias Mitigation Techniques - Read when: evaluation results show inconsistent or suspicious scoring patterns
Metric Selection Guide - Read when: choosing statistical metrics to validate evaluation reliability
Evaluation Pipeline Diagram - Read when: designing the architecture of a multi-stage evaluation system

External research:

Eugene Yan: Evaluating the Effectiveness of LLM-Evaluators - Read when: surveying the state of the art in LLM evaluation
Judging LLM-as-a-Judge (Zheng et al., 2023) - Read when: understanding position bias and MT-Bench methodology
G-Eval: NLG Evaluation using GPT-4 (Liu et al., 2023) - Read when: implementing chain-of-thought evaluation scoring
Large Language Models are not Fair Evaluators (Wang et al., 2023) - Read when: diagnosing systematic bias in evaluation outputs

Related skills in this collection:

evaluation - Foundational evaluation concepts
context-fundamentals - Context structure for evaluation prompts
tool-design - Building evaluation tools

内部参考：

LLM-as-Judge 实现模式 - 阅读时机：从零构建评估流水线，或将 LLM judges 集成到 CI/CD
偏差缓解技术 - 阅读时机：评估结果出现不一致或可疑评分模式
指标选择指南 - 阅读时机：选择用于验证评估可靠性的统计指标
评估流水线图 - 阅读时机：设计多阶段评估系统的架构

外部研究：

Eugene Yan：评估 LLM 评估器的有效性 - 阅读时机：了解 LLM 评估技术的最新进展
评判 LLM-as-a-Judge（Zheng et al., 2023） - 阅读时机：理解 position bias 和 MT-Bench 方法
使用 GPT-4 进行 NLG 评估：G-Eval（Liu et al., 2023） - 阅读时机：实现 chain-of-thought 评估评分
大语言模型并非公平的评估者（Wang et al., 2023） - 阅读时机：诊断评估输出中的系统性偏差

本 collection 中的相关 skill：

evaluation - 基础评估概念
context-fundamentals - 评估 prompt 的 Context 结构
tool-design - 构建评估工具

Skill Metadata

Skill 元数据

Created: 2025-12-24 Last Updated: 2026-05-15 Author: Agent Skills for Context Engineering Contributors Version: 2.1.0

创建时间：2025-12-24 最后更新时间：2026-05-15 作者：Agent Skills for Context Engineering Contributors 版本：2.1.0