mutation-testing

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Mutation Testing Skill

Mutation Testing 技能

This skill runs mutation testing on a target namespace, writes tests to kill surviving mutations, and creates draft PRs with Linear issues for each function.

本技能针对目标命名空间运行突变测试，编写测试用例以清除存活的突变，并为每个函数创建关联Linear问题的草稿PR。

Prerequisites

前置条件

A running nREPL connected to the Metabase dev environment
```
LINEAR_API_KEY
```
environment variable set with a valid Linear personal API key
```
gh
```
CLI authenticated with GitHub

已连接到Metabase开发环境的运行中的nREPL
已设置
```
LINEAR_API_KEY
```
环境变量，值为有效的Linear个人API密钥
已通过GitHub认证的
```
gh
```
CLI

Reference Files

参考文件

```
dev/src/dev/coverage.clj
```
— mutation testing tool (generates reports, runs mutations)
```
dev/src/dev/mutation_testing.clj
```
— Linear API client and PR template helpers
```
mutation-testing-report.lib.card.md
```
— example report from the pilot

```
dev/src/dev/coverage.clj
```
— 突变测试工具（生成报告、运行突变测试）
```
dev/src/dev/mutation_testing.clj
```
— Linear API客户端和PR模板工具
```
mutation-testing-report.lib.card.md
```
— 试点项目的示例报告

Invocation

调用方式

The argument is a Clojure namespace, e.g.:

/mutation-testing metabase.lib.order-by

参数为Clojure命名空间，例如：

/mutation-testing metabase.lib.order-by

Workflow

工作流程

Step 1: Parse the namespace

步骤1：解析命名空间

From the argument (e.g.,

metabase.lib.order-by

), derive:

Test namespace: append
```
-test
```
(e.g.,
```
metabase.lib.order-by-test
```
)
Short name: the last segment (e.g.,
```
order-by
```
)
Source path:
```
src/metabase/lib/order_by.cljc
```
(replace
```
-
```
with
```
_
```
in path segments)
Test path:
```
test/metabase/lib/order_by_test.cljc
```

从传入的参数（如

metabase.lib.order-by

）中推导：

测试命名空间：追加
```
-test
```
（如
```
metabase.lib.order-by-test
```
）
短名称：最后一段内容（如
```
order-by
```
）
源码路径：
```
src/metabase/lib/order_by.cljc
```
（路径段中的
```
-
```
替换为
```
_
```
）
测试路径：
```
test/metabase/lib/order_by_test.cljc
```

Step 2: Set up Linear and load tools

步骤2：配置Linear并加载工具

clojure

(require '[dev.coverage :as cov] :reload)
(require '[dev.mutation-testing :as mut-test] :reload)

If this is the first invocation (or the REPL was restarted), set up the Linear team:

clojure

;; Find the team
(mut-test/list-teams!)
;; Set team ID from the output
(mut-test/set-config! {:team-id "<team-id>"})

If config is already set from a previous invocation, skip this.

clojure

(require '[dev.coverage :as cov] :reload)
(require '[dev.mutation-testing :as mut-test] :reload)

如果是首次调用（或重启了REPL），需配置Linear团队：

clojure

;; 查找团队
(mut-test/list-teams!)
;; 根据输出设置团队ID
(mut-test/set-config! {:team-id "<team-id>"})

如果之前已配置过，可跳过此步骤。

Step 3: Generate baseline report and create Linear project

步骤3：生成基准报告并创建Linear项目

clojure

(cov/generate-report
  '<target-ns>
  ['<test-ns>]
  "mutation-testing-report.lib.<short-name>.before.md")

Read the generated report file. It has three sections:

Uncovered Functions — never called by any test
Partially Covered Functions — called but with surviving mutations
Fully Covered Functions — all mutations killed

Then create a Linear project for this namespace (uses report stats in the description):

clojure

(mut-test/create-project-for-namespace!
  "<target-ns>"
  "mutation-testing-report.lib.<short-name>.before.md")
;; => {:project-id "...", :name "Mutation Testing: metabase.lib.order-by"}
;; Automatically sets :project-id in config for subsequent create-issue! calls

clojure

(cov/generate-report
  '<target-ns>
  ['<test-ns>]
  "mutation-testing-report.lib.<short-name>.before.md")

阅读生成的报告文件，报告包含三个部分：

未覆盖函数：未被任何测试调用的函数
部分覆盖函数：被调用但仍有存活突变的函数
完全覆盖函数：所有突变均被清除的函数

随后为此命名空间创建Linear项目（报告统计信息会被用于项目描述）：

clojure

(mut-test/create-project-for-namespace!
  "<target-ns>"
  "mutation-testing-report.lib.<short-name>.before.md")
;; => {:project-id "...", :name "Mutation Testing: metabase.lib.order-by"}
;; 此操作会自动将:project-id设置到配置中，供后续create-issue!调用使用

Step 4: Plan the work

步骤4：规划工作

Before creating branches, analyze the report to plan how to group the work:

Identify killable functions. For each function with surviving mutations, assess whether any mutations are killable. Run
```
cov/test-mutations
```
to see the full mutation list. If ALL mutations for a function are unkillable (schema-only, semantically equivalent, etc.), skip that function — no branch or PR needed.
Group private functions by public entry point. Private functions cannot be tested directly. Group them with the public function that exercises them. Create one branch/PR per group, not per private function.
Track mutations per group. The
```
:mutations-before
```
and
```
:not-killed
```
fields in the PR should only reflect mutations for functions covered by that specific PR — never mutations from other functions.

在创建分支前，分析报告以规划工作分组：

识别可清除突变的函数：对于每个有存活突变的函数，评估是否存在可清除的突变。运行
```
cov/test-mutations
```
查看完整的突变列表。如果某个函数的所有突变均无法清除（如仅涉及Schema、语义等价等），则跳过该函数——无需创建分支或PR。
按公共入口点分组私有函数：私有函数无法直接测试，需将其与调用它们的公共函数归为一组。每组创建一个分支/PR，而非为每个私有函数单独创建。
跟踪每组的突变情况：PR中的
```
:mutations-before
```
和
```
:not-killed
```
字段应仅反映该PR覆盖的函数的突变情况——切勿包含其他函数的突变。

Step 5: Process each group

步骤5：处理每组函数

For each group of functions that has killable mutations:

对于每组存在可清除突变的函数：

5a. Create a branch

5a. 创建分支

clojure

(mut-test/create-branch! "<target-ns>" "<primary-fn-name>")
;; => "mutation-testing-lib-order-by-orderable-columns"

Use the primary public function name for the branch. If the group covers private functions too, name the branch after the public entry point.

clojure

(mut-test/create-branch! "<target-ns>" "<primary-fn-name>")
;; => "mutation-testing-lib-order-by-orderable-columns"

使用主公共函数名作为分支名。如果组中包含私有函数，则以公共入口点命名分支。

5b. Read context

5b. 了解上下文

Read the source function(s) from the source file
Read the full test file to understand existing test patterns, helpers, and metadata providers used
If the function is private, identify which public function(s) call it

从源码文件中读取目标函数
阅读完整的测试文件，了解现有测试模式、工具函数和使用的元数据提供者
如果是私有函数，确定调用它的公共函数

5c. Write tests

5c. 编写测试用例

Generate the simplest tests that kill the surviving mutations while still making semantic sense. Guidelines:

Never call private functions directly. Always test through public API functions. Use the coverage data to identify which public functions exercise the private function.
Keep tests simple. Each test should verify one meaningful behavior. Don't over-engineer tests just to chase mutation kills.
Follow existing patterns. Match the style of the existing tests: same metadata providers, same helper functions, same assertion patterns.
Insert tests near related existing tests for the same function, not at the end of the file. This minimizes merge conflicts between branches. If there are no existing tests for the function, find the most logical location based on the order of functions in the source file.

IMPORTANT: Use
file_edit
(not
clojure_edit
) when editing test files. The

clojure_edit

tool reformats the entire file, introducing unnecessary whitespace changes that pollute the diff. Use

file_edit

with exact string matching to make surgical insertions that only touch the lines you intend to change.

生成最简单的测试用例，在保证语义合理的前提下清除存活的突变。遵循以下准则：

切勿直接调用私有函数：始终通过公共API函数进行测试。使用覆盖率数据确定哪些公共函数会调用该私有函数。
保持测试简洁：每个测试应验证一个有意义的行为。不要为了追求清除突变而过度设计测试。
遵循现有模式：匹配现有测试的风格：使用相同的元数据提供者、工具函数和断言模式。
将测试插入到相关现有测试附近：为同一函数添加的测试应放在现有相关测试附近，而非文件末尾。这样可以最小化分支间的合并冲突。如果该函数没有现有测试，则根据源码文件中的函数顺序找到最合理的位置。

重要提示：编辑测试文件时请使用
file_edit
（而非
clojure_edit
）。

clojure_edit

工具会重新格式化整个文件，引入不必要的空白变更，污染差异对比。请使用

file_edit

配合精确的字符串匹配，进行精准插入，仅修改你目标的行。

5d. Verify mutations are killed

5d. 验证突变已被清除

Use the REPL to check that the new tests kill the targeted mutations:

clojure

;; Reload the test namespace to pick up new tests
(require '<test-ns> :reload)

;; Test mutations for the specific function
(cov/test-mutations
  '<target-ns>/<fn-name>
  ;; Set of test names that cover this function
  #{'<test-ns>/<test-name-1> '<test-ns>/<test-name-2>})

Check the

:survived

key in the result. If mutations still survive:

Try writing additional tests
If a mutation is truly unkillable (semantically equivalent), note it for the PR description
If a mutation reveals dead/unreachable code, consider suggesting removal as a code improvement

使用REPL检查新测试是否清除了目标突变：

clojure

;; 重新加载测试命名空间以获取新测试
(require '<test-ns> :reload)

;; 测试特定函数的突变
(cov/test-mutations
  '<target-ns>/<fn-name>
  ;; 覆盖该函数的测试名称集合
  #{'<test-ns>/<test-name-1> '<test-ns>/<test-name-2>})

查看结果中的

:survived

键。如果仍有突变存活：

尝试编写额外的测试用例
如果突变确实无法清除（语义等价），请在PR描述中注明
如果突变揭示了死代码/不可达代码，可以建议移除作为代码改进

5e. Handle unkillable mutations and improvements

5e. 处理无法清除的突变和代码改进

Do NOT edit the source namespace just for documentation. Instead:

Unkillable mutations: Note them in the PR description with rationale
Code improvements (dead code removal, expression simplification) that directly relate to surviving mutants:
1. Make the change in the source file
2. Verify all tests still pass with the change
3. Note the file path, line range, and the improved code
4. Reset the change:
```
git checkout -- <source-file>
```
  — do NOT commit it
5. After creating the PR, use
```
add-suggested-change!
```
  to post the improvement as a GitHub suggested change (Step 5h)
6. The reviewer decides whether to accept it

请勿仅为了文档修改源码命名空间。 应按以下方式处理：

无法清除的突变：在PR描述中注明并说明理由
与存活突变直接相关的代码改进（如死代码移除、表达式简化）：
1. 在源码文件中进行修改
2. 验证所有测试在修改后仍能通过
3. 记录文件路径、行范围和改进后的代码
4. 重置修改：执行
```
git checkout -- <source-file>
```
  —— 不要提交该修改
5. 创建PR后，使用
```
add-suggested-change!
```
  将改进作为GitHub建议变更发布（步骤5h）
6. 由评审者决定是否接受该建议

5f. Commit and push

5f. 提交并推送

clojure

;; Only commit the test file — never commit source changes for improvements
(mut-test/commit-and-push! "<target-ns>" "<primary-fn-name>"
  ["test/metabase/lib/<short_name>_test.cljc"])

clojure

;; 仅提交测试文件——切勿提交代码改进相关的源码变更
(mut-test/commit-and-push! "<target-ns>" "<primary-fn-name>"
  ["test/metabase/lib/<short_name>_test.cljc"])

5g. Create Linear issue and draft PR

5g. 创建Linear问题和草稿PR

clojure

;; Create a Linear issue (use the primary function name)
(def issue (mut-test/create-issue-for-function! "<target-ns>" "<primary-fn-name>"))
;; => {:identifier "QUE-1234", :url "https://linear.app/...", ...}

;; Create draft PR linked to the Linear issue
;; :fn-names lists ALL functions covered by this PR
;; :mutations-before counts only mutations for functions in this PR
;; :not-killed lists only unkillable mutations for functions in this PR
(def pr-url
  (mut-test/create-draft-pr!
    {:target-ns         "<target-ns>"
     :fn-name           "<primary-fn-name>"
     :linear-identifier (:identifier issue)
     :mutations-before  9       ;; surviving mutations for these functions only
     :tests-added       2       ;; number of new test functions
     :killed            ["Replace :segment-id with :segment-id__"
                         "Replace cycle-path with nil"]
     :not-killed        [{:description "Replace acc with nil"
                          :rationale   "reduce always has single iteration"}]
     :suggested-changes []}))
;; => "https://github.com/metabase/metabase/pull/12345"

clojure

;; 创建Linear问题（使用主函数名）
(def issue (mut-test/create-issue-for-function! "<target-ns>" "<primary-fn-name>"))
;; => {:identifier "QUE-1234", :url "https://linear.app/...", ...}

;; 创建关联Linear问题的草稿PR
;; :fn-names列出此PR覆盖的所有函数
;; :mutations-before仅统计此PR覆盖的函数的突变数量
;; :not-killed仅列出此PR覆盖的函数中无法清除的突变
(def pr-url
  (mut-test/create-draft-pr!
    {:target-ns         "<target-ns>"
     :fn-name           "<primary-fn-name>"
     :linear-identifier (:identifier issue)
     :mutations-before  9       ;; 仅针对这些函数的存活突变数量
     :tests-added       2       ;; 新增测试用例数量
     :killed            ["Replace :segment-id with :segment-id__"
                         "Replace cycle-path with nil"]
     :not-killed        [{:description "Replace acc with nil"
                          :rationale   "reduce always has single iteration"}]
     :suggested-changes []}))
;; => "https://github.com/metabase/metabase/pull/12345"

5h. Add code improvement suggestions (if any)

5h. 添加代码改进建议（如有）

If you identified code improvements in step 5e (and reset them), post each as a GitHub suggested change:

clojure

(mut-test/add-suggested-change!
  {:pr-url     pr-url
   :path       "src/metabase/lib/<short_name>.cljc"
   :start-line 42    ;; first line of the code to replace
   :end-line   45    ;; last line of the code to replace
   :suggestion "(improved-code-here)"  ;; the replacement code
   :comment    "**Suggested improvement:** <description of the change and why it's related to the surviving mutant>"})

如果在步骤5e中识别出代码改进（并已重置修改），请将每个改进作为GitHub建议变更发布：

clojure

(mut-test/add-suggested-change!
  {:pr-url     pr-url
   :path       "src/metabase/lib/<short_name>.cljc"
   :start-line 42    ;; 待替换代码的起始行
   :end-line   45    ;; 待替换代码的结束行
   :suggestion "(improved-code-here)"  ;; 替换后的代码
   :comment    "**Suggested improvement:** <description of the change and why it's related to the surviving mutant>"})

Step 6: Return to master and repeat

步骤6：返回主分支并重复

clojure

(mut-test/return-to-master!)

Repeat Step 5 for the next group.

clojure

(mut-test/return-to-master!)

重复步骤5处理下一组函数。

Step 7: Summary

步骤7：总结

Print a summary of what was done:

Link to the Linear project for this namespace
Number of functions processed
Number of functions skipped (all mutations unkillable)
Number of draft PRs created
Number of mutations killed vs. unkillable
List of test PRs with links
List of mutation rule PRs with links (if any were created)

打印工作完成情况总结：

此命名空间对应的Linear项目链接
处理的函数数量
跳过的函数数量（所有突变均无法清除）
创建的草稿PR数量
已清除的突变数量 vs 无法清除的突变数量
测试PR列表及链接
突变规则PR列表及链接（如有创建）

Suggesting New Mutation Rules

建议新增突变规则

If you notice patterns in surviving mutations that suggest the mutation testing tool should have additional rules (e.g., a common Clojure form that isn't being mutated), open a separate PR proposing additions to

dev.coverage/mutation-rules

with an explanation.

如果发现存活突变存在规律，表明突变测试工具需要添加新规则（例如，未被突变的常见Clojure语法结构），请单独提交PR，建议向

dev.coverage/mutation-rules

添加新规则并附上说明。

Configuration

配置

The

dev.mutation-testing

namespace reads

LINEAR_API_KEY

from the environment and stores team/project IDs in an atom.

Team ID: Set once per REPL session via

(mut-test/list-teams!)

and

(mut-test/set-config! {:team-id "..."})

(Step 2)

Project ID: Set automatically when
```
create-project-for-namespace!
```
is called (Step 3) — one project per namespace

dev.mutation-testing

命名空间从环境变量中读取

LINEAR_API_KEY

，并将团队/项目ID存储在atom中。

团队ID：每个REPL会话通过

(mut-test/list-teams!)

和

(mut-test/set-config! {:team-id "..."})

设置一次（步骤2）

项目ID：调用
```
create-project-for-namespace!
```
时自动设置（步骤3）——每个命名空间对应一个项目