evals-init

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

evals-init

evals-init

What this skill does

这个 skill 的作用

Initialize the project-level evaluation directory structure following EDD (Eval-Driven Development) principles to prepare for systematic evaluation development. This is completely standalone with zero spec-kit dependencies.
Output:
  1. Directory Structure -
    evals/{system}/
    with proper organization (promptfoo | deepeval)
  2. Security Baseline - Auto-created graders for PII leakage, prompt injection, hallucination detection, misinformation detection
  3. Configuration Files - Standalone config.yml and goldset templates under
    .adlc/evals/
  4. Auto-handoff to
    /evals-specify
    to begin error analysis
Key EDD Principles Applied:
  • Principle I: Spec-Driven Contracts - Evals validate spec compliance
  • Principle II: Binary Pass/Fail - No Likert scales in grader templates
  • Principle IV: Evaluation Pyramid - Tier 1 (fast) + Tier 2 (goldset) structure
  • Principle IX: Test Data as Code - Version control setup for datasets
按照 EDD(Eval-Driven Development)原则初始化项目级评估目录结构,为系统化评估开发做好准备。此过程完全独立,不依赖 spec-kit。
输出:
  1. 目录结构 -
    evals/{system}/
    ,采用适当的组织方式(promptfoo | deepeval)
  2. 安全基线 - 自动创建用于 PII leakage、prompt injection、hallucination detection 和 misinformation detection 的 graders
  3. 配置文件 - 在
    .adlc/evals/
    下提供独立的 config.yml 和 goldset 模板
  4. 自动移交至
    /evals-specify
    ,开始错误分析
所应用的 EDD 关键原则:
  • 原则 I:Spec-Driven Contracts - Evals 用于验证 spec 符合性
  • 原则 II:Binary Pass/Fail - grader 模板中不使用 Likert 量表
  • 原则 IV:Evaluation Pyramid - Tier 1(快速)+ Tier 2(goldset)结构
  • 原则 IX:Test Data as Code - 为数据集设置版本控制

When to use

何时使用

  • Starting systematic evaluation: Set up the initial evaluation harness for your application
  • EDD Adoption: Converting from traditional testing to evaluation-driven development
  • Security-first evaluation: Auto-generate baseline security checks from the start
  • 开始系统性评估:为应用搭建初始 evaluation harness
  • 采用 EDD:从传统 testing 转向 evaluation-driven development
  • 安全优先评估:从一开始自动生成 baseline security checks

When NOT to use

何时不应使用

  • Evals directory already exists: Use
    /evals-validate
    to run tests, or
    /evals-specify
    to add criteria
  • Evaluating team directives: This is for project-level application behavior testing, not directives compliance
  • Evals 目录已存在:使用
    /evals-validate
    运行测试,或使用
    /evals-specify
    添加评估标准
  • 评估团队 directives:此 skill 用于项目级应用行为测试,而非 directives 合规性检查

Process

流程

User Input

用户输入

text
$ARGUMENTS
Parse flags from the arguments first, then treat remaining text as focus areas:
  • --system SYSTEM
    — Choose
    promptfoo
    or
    deepeval
    . If omitted, choose interactively based on tech stack.
  • Remaining text — System description (focus setup)
text
$ARGUMENTS
先解析参数中的 flags,再将其余文本视为重点范围:
  • --system SYSTEM
    — 选择
    promptfoo
    或
    deepeval
    。如果未指定,则根据技术栈交互式选择。
  • 其余文本 — 系统描述(设置重点)

Execution Steps

执行步骤

Phase 1: Tech Stack Detection

阶段 1:技术栈检测

  • Scan project manifests (
    package.json
    ,
    requirements.txt
    ,
    Cargo.toml
    ,
    go.mod
    , etc.)
  • Recommends PromptFoo for mixed/JS stacks; DeepEval for Python-native stacks
  • 扫描项目清单文件(
    package.json
    、
    requirements.txt
    、
    Cargo.toml
    、
    go.mod
    等)
  • 对混合/JS 技术栈推荐 PromptFoo;对 Python 原生技术栈推荐 DeepEval

Phase 2: Create Directory Structure

阶段 2:创建目录结构

Creates:
evals/
├── {system}/                    # promptfoo | deepeval
│   ├── goldset.md              # Published goldset
│   ├── goldset.json            # Auto-generated for system consumption
│   ├── config.yml              # System-specific configuration
│   ├── config.{js,py}          # Generated system config (.js for promptfoo, .py for deepeval)
│   └── graders/                # Binary pass/fail graders
│       ├── check_pii_leakage.py           # Security baseline
│       ├── check_prompt_injection.py     # Security baseline
│       ├── check_hallucination.py        # Security baseline
│       └── check_misinformation.py       # Security baseline
├── results/                    # Git-ignored run outputs
└── .adlc/
    └── drafts/evals/           # Draft eval records (Markdown + YAML)
创建:
evals/
├── {system}/                    # promptfoo | deepeval
│   ├── goldset.md              # Published goldset
│   ├── goldset.json            # Auto-generated for system consumption
│   ├── config.yml              # System-specific configuration
│   ├── config.{js,py}          # Generated system config (.js for promptfoo, .py for deepeval)
│   └── graders/                # Binary pass/fail graders
│       ├── check_pii_leakage.py           # Security baseline
│       ├── check_prompt_injection.py     # Security baseline
│       ├── check_hallucination.py        # Security baseline
│       └── check_misinformation.py       # Security baseline
├── results/                    # Git-ignored run outputs
└── .adlc/
    └── drafts/evals/           # Draft eval records (Markdown + YAML)

Phase 3: Configuration Copy

阶段 3:复制配置

  • Create
    .adlc/evals/
    if missing.
  • Copy
    skills/evals/evals-templates/evals-config-template.yml
    to
    .adlc/evals/evals-config.yml
    .
  • 如果缺失,则创建
    .adlc/evals/
    。
  • 将
    skills/evals/evals-templates/evals-config-template.yml
    复制到
    .adlc/evals/evals-config.yml
    。

Phase 4: Auto-Handoff

阶段 4:自动移交

Trigger
/evals-specify
to begin error analysis.
触发
/evals-specify
,开始错误分析。

Verification

验证

  • evals/{system}/goldset.md
    exists (initially empty)
  • .adlc/evals/evals-config.yml
    exists
  • Graders directory populated with 4 security baseline python scripts
  • Results directory contains
    .gitignore
    to prevent versioning traces
  • Handover report generated with recommended framework and next steps
  • evals/{system}/goldset.md
    存在(初始为空)
  • .adlc/evals/evals-config.yml
    存在
  • graders 目录中有 4 个安全基线 Python 脚本
  • results 目录包含
    .gitignore
    ,防止运行轨迹被纳入版本控制
  • 生成移交报告,其中包含推荐框架和后续步骤