evals-validate
Compare original and translation side by side
🇺🇸
Original
English🇨🇳
Translation
Chineseevals-validate
evals-validate
What this skill does
此 skill 的功能
Conducts comprehensive validation of the implemented evaluation system following EDD principles to ensure production readiness through statistical analysis, performance verification, and quality assurance.
Output:
- Statistical Validation - TPR/TNR analysis, accuracy metrics, confidence intervals
- Performance Validation - SLA compliance verification for evaluation pyramid tiers
- Quality Assurance - Goldset integrity, example balance, coverage analysis
- Holdout Dataset Validation - Unbiased accuracy assessment on reserved test set
- Auto-handoff to for closed loop trajectory analysis
/evals-analyze
Key EDD Principles Applied:
- Principle IV: Evaluation Pyramid - Tier performance SLA validation (Tier 1 <30s, Tier 2 <5min)
- Principle II: Binary Pass/Fail - Statistical compliance verification
- Principle IX: Test Data as Code - Holdout dataset validation integrity
- Principle III: Error Analysis - Pattern stability validation
按照 EDD 原则,对已实现的评估系统进行全面验证,通过统计分析、性能验证和质量保障确保其达到生产就绪状态。
输出:
- 统计验证 - TPR/TNR 分析、准确率指标、置信区间
- 性能验证 - 验证 Evaluation Pyramid 各层级的 SLA 合规性
- 质量保障 - Goldset 完整性、样本均衡性、覆盖率分析
- Holdout 数据集验证 - 在保留的测试集上进行无偏准确性评估
- 自动交接至 ,形成闭环轨迹分析
/evals-analyze
已应用的关键 EDD 原则:
- 原则 IV:Evaluation Pyramid - 验证层级性能 SLA(Tier 1 <30s,Tier 2 <5min)
- 原则 II:Binary Pass/Fail - 统计合规性验证
- 原则 IX:Test Data as Code - Holdout 数据集验证完整性
- 原则 III:Error Analysis - 模式稳定性验证
When to use
何时使用
- After : Execute the evaluation suite and measure quality
/evals-implement - CI/CD Pipeline gate: Run evaluations before release to ensure no regressions
- Periodic audit: Verify evaluator accuracy on holdout data to check for model drift
- 执行 之后:运行评估套件并衡量质量
/evals-implement - CI/CD Pipeline gate:发布前运行评估,确保没有回归问题
- 定期审计:在 holdout 数据上验证 evaluator 准确性,以检查模型漂移
When NOT to use
何时不应使用
- Evaluator not generated: Run to build grader files first
/evals-implement - Analysing failure traces: Use to extract deep insights from run results
/evals-analyze
- 尚未生成 evaluator:先运行 构建 grader 文件
/evals-implement - 正在分析失败轨迹:使用 从运行结果中提取深入洞察
/evals-analyze
Process
流程
User Input
用户输入
text
$ARGUMENTS- — Validate only on holdout dataset (unbiased validation)
--holdout-only - — Skip statistical analysis, focus on SLA compliance
--performance-only - — Specific metrics to validate (tpr, tnr, accuracy, performance)
--metrics METRICS
text
$ARGUMENTS- — 仅在 holdout 数据集上验证(无偏验证)
--holdout-only - — 跳过统计分析,专注于 SLA 合规性
--performance-only - — 要验证的特定指标(tpr, tnr, accuracy, performance)
--metrics METRICS
Execution Steps
执行步骤
Phase 1: Execute Evaluations
阶段 1:执行评估
Runs the underlying framework CLI directly:
- PromptFoo:
npx promptfoo eval --config evals/promptfoo/config.js - DeepEval: or
pytest evals/deepeval/ -vpython evals/deepeval/config.py
直接运行底层框架 CLI:
- PromptFoo:
npx promptfoo eval --config evals/promptfoo/config.js - DeepEval:或
pytest evals/deepeval/ -vpython evals/deepeval/config.py
Phase 2: Compute Statistical Validation
阶段 2:计算统计验证结果
- Parse generated results JSON from .
evals/results/ - Calculate True Positive Rate (TPR) and True Negative Rate (TNR).
- Calculate overall accuracy with 95% confidence intervals.
- Ensure no Likert scales or numerical scores leak into results.
- 解析 中生成的结果 JSON。
evals/results/ - 计算 True Positive Rate (TPR) 和 True Negative Rate (TNR)。
- 计算整体准确率及其 95% 置信区间。
- 确保结果中不混入 Likert 量表或数值评分。
Phase 3: SLA Compliance Check
阶段 3:检查 SLA 合规性
- Measure execution times for Tier 1 and Tier 2.
- Verify Tier 1 completes under 30 seconds.
- Verify Tier 2 completes under 5 minutes.
- Check headroom analysis (SLA budget consumed).
- 测量 Tier 1 和 Tier 2 的执行时间。
- 验证 Tier 1 是否在 30 秒内完成。
- 验证 Tier 2 是否在 5 分钟内完成。
- 检查余量分析(SLA 预算消耗情况)。
Phase 4: Write Validation Report
阶段 4:编写验证报告
- Write validation results to .
evals/results/validation_report.md - Include pass/fail counts, TPR/TNR table, SLA timings, and holdout set results.
- 将验证结果写入 。
evals/results/validation_report.md - 包含通过/失败计数、TPR/TNR 表格、SLA 耗时和 holdout 集结果。
Phase 5: Auto-Handoff
阶段 5:自动交接
Trigger to close the loop.
/evals-analyze触发 以形成闭环。
/evals-analyzeVerification
验证
- Evaluation execution successfully completed with results JSON written to
evals/results/ - created with TPR/TNR and SLA metrics
evals/results/validation_report.md - Statistical metrics calculated with confidence intervals
- Headroom and SLA compliance verified
- Handover summary lists results and validation report path
- 评估执行已成功完成,结果 JSON 已写入
evals/results/ - 已创建包含 TPR/TNR 和 SLA 指标的
evals/results/validation_report.md - 统计指标计算包含置信区间
- 已验证余量与 SLA 合规性
- 交接摘要列出了结果和验证报告路径