evals-validate

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

evals-validate

evals-validate

What this skill does

此 skill 的功能

Conducts comprehensive validation of the implemented evaluation system following EDD principles to ensure production readiness through statistical analysis, performance verification, and quality assurance.
Output:
  1. Statistical Validation - TPR/TNR analysis, accuracy metrics, confidence intervals
  2. Performance Validation - SLA compliance verification for evaluation pyramid tiers
  3. Quality Assurance - Goldset integrity, example balance, coverage analysis
  4. Holdout Dataset Validation - Unbiased accuracy assessment on reserved test set
  5. Auto-handoff to
    /evals-analyze
    for closed loop trajectory analysis
Key EDD Principles Applied:
  • Principle IV: Evaluation Pyramid - Tier performance SLA validation (Tier 1 <30s, Tier 2 <5min)
  • Principle II: Binary Pass/Fail - Statistical compliance verification
  • Principle IX: Test Data as Code - Holdout dataset validation integrity
  • Principle III: Error Analysis - Pattern stability validation
按照 EDD 原则,对已实现的评估系统进行全面验证,通过统计分析、性能验证和质量保障确保其达到生产就绪状态。
输出:
  1. 统计验证 - TPR/TNR 分析、准确率指标、置信区间
  2. 性能验证 - 验证 Evaluation Pyramid 各层级的 SLA 合规性
  3. 质量保障 - Goldset 完整性、样本均衡性、覆盖率分析
  4. Holdout 数据集验证 - 在保留的测试集上进行无偏准确性评估
  5. 自动交接至
    /evals-analyze
    ,形成闭环轨迹分析
已应用的关键 EDD 原则:
  • 原则 IV:Evaluation Pyramid - 验证层级性能 SLA(Tier 1 <30s,Tier 2 <5min)
  • 原则 II:Binary Pass/Fail - 统计合规性验证
  • 原则 IX:Test Data as Code - Holdout 数据集验证完整性
  • 原则 III:Error Analysis - 模式稳定性验证

When to use

何时使用

  • After
    /evals-implement
    : Execute the evaluation suite and measure quality
  • CI/CD Pipeline gate: Run evaluations before release to ensure no regressions
  • Periodic audit: Verify evaluator accuracy on holdout data to check for model drift
  • 执行
    /evals-implement
    之后
    :运行评估套件并衡量质量
  • CI/CD Pipeline gate:发布前运行评估,确保没有回归问题
  • 定期审计:在 holdout 数据上验证 evaluator 准确性,以检查模型漂移

When NOT to use

何时不应使用

  • Evaluator not generated: Run
    /evals-implement
    to build grader files first
  • Analysing failure traces: Use
    /evals-analyze
    to extract deep insights from run results
  • 尚未生成 evaluator:先运行
    /evals-implement
    构建 grader 文件
  • 正在分析失败轨迹:使用
    /evals-analyze
    从运行结果中提取深入洞察

Process

流程

User Input

用户输入

text
$ARGUMENTS
  • --holdout-only
    — Validate only on holdout dataset (unbiased validation)
  • --performance-only
    — Skip statistical analysis, focus on SLA compliance
  • --metrics METRICS
    — Specific metrics to validate (tpr, tnr, accuracy, performance)
text
$ARGUMENTS
  • --holdout-only
    — 仅在 holdout 数据集上验证(无偏验证)
  • --performance-only
    — 跳过统计分析,专注于 SLA 合规性
  • --metrics METRICS
    — 要验证的特定指标(tpr, tnr, accuracy, performance)

Execution Steps

执行步骤

Phase 1: Execute Evaluations

阶段 1:执行评估

Runs the underlying framework CLI directly:
  • PromptFoo:
    npx promptfoo eval --config evals/promptfoo/config.js
  • DeepEval:
    pytest evals/deepeval/ -v
    or
    python evals/deepeval/config.py
直接运行底层框架 CLI:
  • PromptFoo:
    npx promptfoo eval --config evals/promptfoo/config.js
  • DeepEval:
    pytest evals/deepeval/ -v
    或
    python evals/deepeval/config.py

Phase 2: Compute Statistical Validation

阶段 2:计算统计验证结果

  • Parse generated results JSON from
    evals/results/
    .
  • Calculate True Positive Rate (TPR) and True Negative Rate (TNR).
  • Calculate overall accuracy with 95% confidence intervals.
  • Ensure no Likert scales or numerical scores leak into results.
  • 解析
    evals/results/
    中生成的结果 JSON。
  • 计算 True Positive Rate (TPR) 和 True Negative Rate (TNR)。
  • 计算整体准确率及其 95% 置信区间。
  • 确保结果中不混入 Likert 量表或数值评分。

Phase 3: SLA Compliance Check

阶段 3:检查 SLA 合规性

  • Measure execution times for Tier 1 and Tier 2.
  • Verify Tier 1 completes under 30 seconds.
  • Verify Tier 2 completes under 5 minutes.
  • Check headroom analysis (SLA budget consumed).
  • 测量 Tier 1 和 Tier 2 的执行时间。
  • 验证 Tier 1 是否在 30 秒内完成。
  • 验证 Tier 2 是否在 5 分钟内完成。
  • 检查余量分析(SLA 预算消耗情况)。

Phase 4: Write Validation Report

阶段 4:编写验证报告

  • Write validation results to
    evals/results/validation_report.md
    .
  • Include pass/fail counts, TPR/TNR table, SLA timings, and holdout set results.
  • 将验证结果写入
    evals/results/validation_report.md
    。
  • 包含通过/失败计数、TPR/TNR 表格、SLA 耗时和 holdout 集结果。

Phase 5: Auto-Handoff

阶段 5:自动交接

Trigger
/evals-analyze
to close the loop.
触发
/evals-analyze
以形成闭环。

Verification

验证

  • Evaluation execution successfully completed with results JSON written to
    evals/results/
  • evals/results/validation_report.md
    created with TPR/TNR and SLA metrics
  • Statistical metrics calculated with confidence intervals
  • Headroom and SLA compliance verified
  • Handover summary lists results and validation report path
  • 评估执行已成功完成,结果 JSON 已写入
    evals/results/
  • 已创建包含 TPR/TNR 和 SLA 指标的
    evals/results/validation_report.md
  • 统计指标计算包含置信区间
  • 已验证余量与 SLA 合规性
  • 交接摘要列出了结果和验证报告路径