Back to Details

mutation-testing

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

AI-Powered Mutation Testing

AI驱动的Mutation Testing

Target: $ARGUMENTS
If no argument provided, default to files changed in the current branch (via 
git diff
).
This skill implements the mutation testing workflow: generate targeted mutants, filter unproductive ones, run tests to find survivors, and harden the test suite by strengthening or creating tests that kill surviving mutants.
目标：$ARGUMENTS
如果未提供参数，默认使用当前分支中变更的文件（通过
git diff
获取）。
此技能实现了Mutation Testing工作流：生成针对性的变异体，过滤无效变异体，运行测试找出存活的变异体，并通过强化或创建能消灭存活变异体的测试来加固测试套件。

Step 1: Gather Context

步骤1：收集上下文信息

1a. Identify Target Files

1a. 确定目标文件

Determine what to mutate based on 
$ARGUMENTS
:
  1. File path — Mutate the specified source file
  2. Directory — Mutate source files in the directory (exclude test files)
  3. No argument — Use git diff to find changed source files:
    bash
    git diff --name-only $(git merge-base HEAD main)...HEAD -- '*.ts' '*.tsx' ':!*.spec.*' ':!*.test.*'
If no target files found, notify the user and stop.
根据
$ARGUMENTS
确定要变异的对象：
  1. 文件路径 — 对指定的源文件进行变异
  2. 目录 — 对目录中的源文件进行变异（排除测试文件）
  3. 无参数 — 使用git diff查找变更的源文件：
    bash
    git diff --name-only $(git merge-base HEAD main)...HEAD -- '*.ts' '*.tsx' ':!*.spec.*' ':!*.test.*'
如果未找到目标文件，通知用户并终止流程。

1b. Gather Supporting Context

1b. 收集辅助上下文信息

For each target source file, collect:
  1. Source code — Read the full file contents
  2. Existing tests — Find corresponding test files (
    *.spec.ts
    , 
    *.test.ts
    ) via naming convention or import analysis
  3. Test coverage — Run tests with coverage for the target file to identify uncovered lines:
    bash
    bun run test -- --coverage --collectCoverageFrom='<target-file>' --silent 2>&1 | tail -20
  4. Recent git history — Check for recent defects or frequent changes:
    bash
    git log --oneline -10 -- <target-file>
  5. Risk factors — Assess which risk factors apply to each file:
Risk FactorIndicators
Data security / complianceHandles PII, auth, authorization, encryption, tokens
IntegrationsExternal API calls, HTTP clients, webhooks, message queues
Code vs data modelDatabase queries, ORM entities, schema validation
Historic defectsFrequent bug-fix commits, complex conditional logic
Change impactExported utility functions, shared modules, base classes
PerformanceLoops over large datasets, recursive calls, caching logic
针对每个目标源文件，收集以下信息：
  1. 源代码 — 读取文件的完整内容
  2. 现有测试 — 通过命名约定或导入分析找到对应的测试文件（
    *.spec.ts
    、
    *.test.ts
    ）
  3. 测试覆盖率 — 运行带覆盖率统计的测试，识别目标文件中未被覆盖的代码行：
    bash
    bun run test -- --coverage --collectCoverageFrom='<target-file>' --silent 2>&1 | tail -20
  4. 近期Git历史 — 检查近期的缺陷修复或频繁变更：
    bash
    git log --oneline -10 -- <target-file>
  5. 风险因素 — 评估每个文件对应的风险因素：
风险因素识别指标
数据安全/合规处理PII、auth、授权、加密、令牌
集成外部API调用、HTTP客户端、Webhook、消息队列
代码与数据模型数据库查询、ORM实体、schema验证
历史缺陷频繁的bug修复提交、复杂条件逻辑
变更影响导出的工具函数、共享模块、基类
性能遍历大型数据集、递归调用、缓存逻辑

1c. Determine Test Runner

1c. 确定测试运行器

Detect the project's test framework from 
package.json
scripts and devDependencies:
  • Jest: 
    bun run test
  • Vitest: 
    bun run test
  • Other: adapt accordingly
从
package.json
的脚本和devDependencies中检测项目的测试框架：
  • Jest: 
    bun run test
  • Vitest: 
    bun run test
  • 其他：相应调整

Step 2: Generate Mutants

步骤2：生成变异体

2a. Create Experimental Branch

2a. 创建实验分支

bash
git stash --include-untracked || true
git checkout -b mutation-testing/$(date +%Y-%m-%d-%H%M%S)
git stash pop || true
bash
git stash --include-untracked || true
git checkout -b mutation-testing/$(date +%Y-%m-%d-%H%M%S)
git stash pop || true

2b. Generate Risk-Factor-Guided Mutants

2b. 生成基于风险因素的变异体

For each target source file, generate 3-5 mutants that target the identified risk factors. Apply these mutation operators:
Operator TypeExamples
Decision mutationsChange 
>
to 
>=
, 
&&
to 
||
, 
===
to 
!==
Value mutationsChange constants, swap string literals, alter numeric values
Statement mutationsRemove guard clauses, delete error handling, skip validation
Integration mutationsRemove timeout handling, skip error code checks, bypass retry logic
Security mutationsRemove auth checks, bypass input validation, expose sensitive data in logs
For each mutant, produce:
  1. Mutant ID — Sequential identifier (M001, M002, etc.)
  2. File and line — Exact location of the change
  3. Mutation description — What was changed and why
  4. Risk factor — Which risk factor this targets
  5. The code change — A minimal, atomic edit to introduce the defect
Mutant quality criteria — Each mutant must be:
  • Non-trivial — Not just whitespace, comments, or formatting
  • Buildable — The project must still compile/typecheck
  • Realistic — Simulates a plausible programming error
  • Targeted — Addresses a specific risk factor
  • Atomic — Exactly one logical change per mutant
针对每个目标源文件，生成3-5个针对已识别风险因素的变异体。应用以下变异操作符：
操作符类型示例
决策变异将
>
改为
>=
，
&&
改为`
值变异修改常量、交换字符串字面量、更改数值
语句变异移除守卫子句、删除错误处理、跳过验证
集成变异移除超时处理、跳过错误码检查、绕过重试逻辑
安全变异移除认证检查、绕过输入验证、在日志中暴露敏感数据
针对每个变异体，生成以下信息：
  1. 变异体ID — 顺序标识符（如M001、M002等）
  2. 文件与行号 — 变更的精确位置
  3. 变异描述 — 变更内容及原因
  4. 风险因素 — 针对的风险因素
  5. 代码变更 — 引入缺陷的最小原子性修改
变异体质量标准 — 每个变异体必须满足：
  • 非琐碎 — 不能仅为空白、注释或格式变更
  • 可构建 — 项目仍能编译/类型检查通过
  • 真实可信 — 模拟合理的编程错误
  • 针对性 — 针对特定风险因素
  • 原子性 — 每个变异体仅包含一个逻辑变更

2c. Apply and Validate Each Mutant

2c. 应用并验证每个变异体

For each generated mutant, one at a time:
  1. Apply the mutation — Edit the source file to introduce the defect
  2. Build check — Run 
    bun run typecheck
    to verify the project still compiles
    • If build fails: discard the mutant, revert the change, log failure reason, continue to next
  3. Commit the mutant — 
    git commit -am "mutant: M00X - <description>"
针对每个生成的变异体，依次执行以下操作：
  1. 应用变异 — 编辑源文件以引入缺陷
  2. 构建检查 — 运行
    bun run typecheck
    验证项目仍可编译
    • 若构建失败：丢弃该变异体，回滚变更，记录失败原因，继续下一个
  3. 提交变异体 — 执行
    git commit -am "mutant: M00X - <description>"

Step 3: Filter Mutants

步骤3：过滤变异体

3a. Run Tests Against Each Mutant

3a. 针对每个变异体运行测试

Process mutants in reverse order (LIFO — latest commit first):
  1. Run relevant tests against the mutant:
    bash
    bun run test -- <test-file-path> --silent 2>&1
  2. Evaluate result:
    • Test fails (mutant killed) → Revert the mutant commit. Log as killed. Proceed to next mutant.
    • Test passes (mutant survived) → The mutant reveals a test gap. Keep it for Step 4.
按逆序处理变异体（后进先出LIFO — 最新提交优先）：
  1. 针对变异体运行相关测试：
    bash
    bun run test -- <test-file-path> --silent 2>&1
  2. 评估结果：
    • 测试失败（变异体被消灭）→ 回滚变异体提交，记录为已消灭，继续下一个变异体。
    • 测试通过（变异体存活）→ 该变异体暴露了测试缺口，保留至步骤4处理。

3b. Equivalence Detection for Surviving Mutants

3b. 存活变异体的等价性检测

For each surviving mutant, verify it is not semantically equivalent to the original:
  1. AST-level comparison — Compare the diff. If the change is purely syntactic (reordering independent statements, renaming to equivalent aliases), discard it.
  2. Behavioral analysis — Reason about whether any input could distinguish the mutant from the original:
    • If no distinguishing input exists → Discard as equivalent
    • If a distinguishing input exists → Keep as a unique actionable mutant
    • If uncertain → Keep and flag for human review
针对每个存活的变异体，验证其与原始代码在语义上不等价：
  1. AST层级比较 — 比较差异。若变更仅为语法层面（如重排独立语句、重命名为等价别名），则丢弃该变异体。
  2. 行为分析 — 判断是否存在输入能区分变异体与原始代码：
    • 若不存在区分输入 → 判定为等价并丢弃
    • 若存在区分输入 → 保留为可处理的唯一变异体
    • 若无法确定 → 保留并标记为需人工审核

3c. Record Mutant Metadata

3c. 记录变异体元数据

For each mutant, maintain a record:
json
{
  "mutant_id": "M001",
  "file": "<source-file>",
  "line": 42,
  "risk_factor": "Data security / compliance",
  "description": "Removed authentication check before data access",
  "status": "survived|killed|equivalent|discarded",
  "commit_hash": "<hash>",
  "tests_run": ["test-file.spec.ts"],
  "equivalence_result": "not_equivalent|equivalent|uncertain"
}
Print a summary table after filtering:
| Mutant | File | Risk Factor | Status |
|--------|------|-------------|--------|
| M001   | ... | ...          | survived |
| M002   | ... | ...          | killed   |
针对每个变异体，维护以下记录：
json
{
  "mutant_id": "M001",
  "file": "<source-file>",
  "line": 42,
  "risk_factor": "Data security / compliance",
  "description": "Removed authentication check before data access",
  "status": "survived|killed|equivalent|discarded",
  "commit_hash": "<hash>",
  "tests_run": ["test-file.spec.ts"],
  "equivalence_result": "not_equivalent|equivalent|uncertain"
}
过滤完成后打印汇总表格：
| Mutant | File | Risk Factor | Status |
|--------|------|-------------|--------|
| M001   | ... | ...          | survived |
| M002   | ... | ...          | killed   |

Step 4: Harden Test Suite

步骤4：加固测试套件

For each surviving, non-equivalent mutant:
针对每个存活且非等价的变异体：

4a. Determine Test Strategy

4a. 确定测试策略

  1. Existing test covers the mutated region → Strengthen the existing test
  2. No test covers the mutated region → Generate a new test
  1. 现有测试覆盖变异区域 → 强化现有测试
  2. 无测试覆盖变异区域 → 生成新测试

4b. Strengthen Existing Test or Generate New Test

4b. 强化现有测试或生成新测试

When strengthening an existing test, modify the test to:
  • Add assertions that detect the behavioral difference introduced by the mutant
  • Add or modify test inputs that expose the defect in the mutant code
  • Preserve the test method name, signature, and overall structure
  • Avoid adding trivial or unrelated checks
When generating a new test, create a test that:
  • Specifically exercises the behavioral difference between original and mutant code
  • Uses strong, precise assertions that detect the exact defect
  • Focuses on the identified risk factor
  • Includes realistic test data that exposes the mutant's flawed behavior
  • Follows existing test naming conventions and structure from the codebase
强化现有测试时，修改测试以实现：
  • 添加断言以检测变异体引入的行为差异
  • 添加或修改测试输入以暴露变异体代码中的缺陷
  • 保留测试方法名称、签名及整体结构
  • 避免添加琐碎或无关的检查
生成新测试时，创建的测试需满足：
  • 专门针对原始代码与变异体代码的行为差异
  • 使用强而精确的断言以检测确切缺陷
  • 聚焦于已识别的风险因素
  • 包含能暴露变异体缺陷行为的真实测试数据
  • 遵循代码库中现有的测试命名约定和结构

4c. Validate Each Test (3-attempt limit)

4c. 验证每个测试（最多3次尝试）

For each improved or new test, validate with up to 3 attempts:
  1. Revert the mutant — Switch to original code
  2. Build check — Ensure the test compiles
  3. Run on original code — Test must PASS on unmodified code
  4. Re-apply the mutant — Switch back to mutated code
  5. Run on mutant code — Test must FAIL on the mutant
If validation fails, refine the test (up to 3 total attempts). If still failing after 3 attempts, flag for manual review.
针对每个改进或新增的测试，最多尝试3次验证：
  1. 回滚变异体 — 切换回原始代码
  2. 构建检查 — 确保测试可编译
  3. 在原始代码上运行 — 测试必须在未修改的代码上通过
  4. 重新应用变异体 — 切换回变异后的代码
  5. 在变异体代码上运行 — 测试必须在变异体上失败
若验证失败，优化测试（最多3次尝试）。若3次尝试后仍失败，标记为需人工审核。

4d. Apply Decision Matrix

4d. 应用决策矩阵

Classify each test result:
Original CodeMutant CodeSignalAction
Builds + PassesBuilds + FailsStrong detectionKeep — alert developer
Builds + PassesBuilds + PassesWeak assertionRefine — strengthen assertion
Builds + FailsBuilds + PassesBad testDiscard — generate new test
Doesn't buildAnyBroken testDiscard — generate new test
Builds + PassesDoesn't buildUntestable mutantDiscard — notify developer
Builds + FailsBuilds + FailsAmbiguous signalDiscard — generate new test
对每个测试结果进行分类：
原始代码状态变异体代码状态信号操作
可构建 + 测试通过可构建 + 测试失败强检测保留 — 通知开发者
可构建 + 测试通过可构建 + 测试通过弱断言优化 — 强化断言
可构建 + 测试失败可构建 + 测试通过无效测试丢弃 — 生成新测试
无法构建任意状态损坏的测试丢弃 — 生成新测试
可构建 + 测试通过无法构建不可测试的变异体丢弃 — 通知开发者
可构建 + 测试失败可构建 + 测试失败模糊信号丢弃 — 生成新测试

4e. Finalize Tests

4e. 最终确定测试

For each validated test:
  1. Add an inline comment above the test referencing the mutant:
    typescript
    // Test hardened to kill mutant M001 (Risk Factor: Data security / compliance)
  2. Commit the test improvement to the experimental branch
针对每个验证通过的测试：
  1. 在测试上方添加内联注释，引用对应的变异体：
    typescript
    // Test hardened to kill mutant M001 (Risk Factor: Data security / compliance)
  2. 将测试改进提交到实验分支

Step 5: Cleanup and Report

步骤5：清理与报告

5a. Revert All Mutants

5a. 回滚所有变异体

Remove all mutant commits from the experimental branch, keeping only test improvements:
bash
undefined
从实验分支中移除所有变异体提交，仅保留测试改进：
bash
undefined

Revert mutant commits (identified by "mutant:" prefix in commit message)

Revert mutant commits (identified by "mutant:" prefix in commit message)

git log --oneline | grep "^.* mutant:" | awk '{print $1}' | while read hash; do git revert --no-commit "$hash" done git commit -m "chore: revert all mutants after test hardening"
undefined
git log --oneline | grep "^.* mutant:" | awk '{print $1}' | while read hash; do git revert --no-commit "$hash" done git commit -m "chore: revert all mutants after test hardening"
undefined

5b. Validate Clean State

5b. 验证干净状态

  1. Run full test suite on the clean code with test improvements:
    bash
    bun run test 2>&1
  2. All tests must pass. If any fail, investigate and fix.
  1. 在包含测试改进的干净代码上运行完整测试套件：
    bash
    bun run test 2>&1
  2. 所有测试必须通过。若有失败，调查并修复。

5c. Report Results

5c. 报告结果

Print a comprehensive mutation testing report:
undefined
打印一份全面的Mutation Testing报告：
undefined

Mutation Testing Report

Mutation Testing报告

Target: <files tested> Date: <timestamp> Branch: <experimental branch name>
目标: <files tested> 日期: <timestamp> 分支: <experimental branch name>

Summary

摘要

  • Total mutants generated: X
  • Killed by existing tests: X
  • Survived (test gaps found): X
  • Equivalent (discarded): X
  • Build failures (discarded): X
  • Mutation Score: X% (killed / (total - equivalent))
  • 生成的变异体总数: X
  • 被现有测试消灭的变异体: X
  • 存活的变异体（发现测试缺口）: X
  • 等价变异体（已丢弃）: X
  • 构建失败的变异体（已丢弃）: X
  • Mutation Score: X% (killed / (total - equivalent))

Test Improvements

测试改进

  • Tests strengthened: X
  • New tests generated: X
  • Tests validated successfully: X
  • Tests requiring manual review: X
  • 强化的测试数量: X
  • 生成的新测试数量: X
  • 验证成功的测试数量: X
  • 需人工审核的测试数量: X

Surviving Mutants (Unresolved)

存活变异体（未解决）

MutantFileLineRisk FactorDescription
M00X............
MutantFileLineRisk FactorDescription
M00X............

Oracle Gap

Oracle缺口

  • Code coverage: X%
  • Mutation score: X%
  • Oracle gap: X% (coverage - mutation score)
  • 代码覆盖率: X%
  • Mutation score: X%
  • Oracle gap: X% (coverage - mutation score)

Risk Factor Coverage

风险因素覆盖情况

Risk FactorMutantsKilledSurvivedScore
Data securityXXXX%
IntegrationsXXXX%
undefined
Risk FactorMutantsKilledSurvivedScore
Data securityXXXX%
IntegrationsXXXX%
undefined

5d. Present Options

5d. 提供操作选项

Ask the user:
  1. Merge test improvements — Cherry-pick test commits to the original branch, delete experimental branch
  2. Keep experimental branch — For further review before merging
  3. Discard everything — Delete the experimental branch, no changes kept
If the user chooses to merge:
bash
git checkout <original-branch>
git cherry-pick <test-improvement-commits>
git branch -D <experimental-branch>
询问用户：
  1. 合并测试改进 — 将测试提交拣选到原始分支，删除实验分支
  2. 保留实验分支 — 供合并前进一步审核
  3. 全部丢弃 — 删除实验分支，不保留任何变更
若用户选择合并：
bash
git checkout <original-branch>
git cherry-pick <test-improvement-commits>
git branch -D <experimental-branch>

Never

禁止操作

  • Modify test files to make them pass on mutants (defeats the purpose)
  • Generate mutants in test files (only mutate source code)
  • Skip the build check after generating a mutant
  • Commit mutants to protected branches (dev, staging, main)
  • Leave mutant code in the codebase after completion
  • Generate more than 5 mutants per file (diminishing returns)
  • Skip equivalence detection for surviving mutants
  • Mark a test as valid without running it on both original and mutant code
  • Use 
    --no-verify
    with any git command
  • 修改测试文件使其在变异体上通过（违背测试目的）
  • 在测试文件中生成变异体（仅对源代码进行变异）
  • 生成变异体后跳过构建检查
  • 将变异体提交到受保护分支（如dev、staging、main）
  • 完成后将变异体代码留在代码库中
  • 每个文件生成超过5个变异体（收益递减）
  • 跳过存活变异体的等价性检测
  • 未在原始代码和变异体代码上都运行就标记测试有效
  • 在任何git命令中使用
    --no-verify