review-jit-test-gen

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

JIT Test Generation

JIT 测试生成

Generate focused tests for the code being changed right now, and prove each one would catch a regression in that change.

The JIT approach: write tests at the moment of change, when context is freshest and the risk of regression is highest. A test that passes on the change and also passes on the broken version catches nothing, so the core of this skill is Step 6: run the new tests against a broken version and show that at least one fails.

Background:

references/wisdom.md

(from What Is Just-in-Time Catching Test Generation?) explains catching versus hardening tests, mutants, and the "dodgy diff". Read its first two sections if the terms are new.

为当前正在变更的代码生成聚焦测试，并证明每个测试都能捕获该变更中的回归。

JIT 方法：在变更发生时编写测试，此时上下文最新鲜，回归风险最高。一个在变更版本上通过、并且在损坏版本上也通过的测试毫无作用，因此这个 skill 的核心是步骤 6：针对损坏版本运行新测试，并证明至少有一个测试失败。

背景：

references/wisdom.md

（来自什么是即时捕获测试生成？）介绍了捕获型（catching）测试与强化型（hardening）测试、变异体（mutant）以及“可疑 diff（dodgy diff）”。如果这些术语不熟悉，请阅读前两节。

Keep or Throw Away

保留还是丢弃

wisdom.md

describes catching tests as disposable: they are generated to fail on a buggy diff, and they never join the codebase. A user who asks for "tests for my changes" usually wants tests they can keep. This skill produces both kinds and treats them differently:

Result of Step 6	What it is	What to do
Passes on the change, fails on the parent or a mutant	Proven hardening test: it guards this change against future regressions	Keep it in the project's test tree (default)
Fails on the change itself	Weak catch: the change may have a bug, or the test misread the intent	Never commit it. Ask the user, in one line, "this used to do X, now it does Y; is that intended?" A "no" means the code has a bug: fix it, and the test becomes a keepable regression test. A "yes" means the test was wrong: delete it
Passes on both	Catches nothing for this change	Rewrite it to target a risk from Step 3, or delete it

Why keep the proven ones: wisdom.md's "disposable" rule exists because large pipelines generate thousands of unreviewed tests per diff, and nobody can maintain them. Here a person asked for a few tests, reviews them, and each one is proven to fail on a real breakage, so it earns its maintenance cost.

Ask the user before writing when intent is unclear, for example when they say "jit tests" or "catching tests" (which suggests disposable), or when the repo has no test tree. Ask: "Keep the proven tests in the repo, or run them as throwaway catching tests and report only what they find?" In throwaway mode, write the tests to the scratch directory (see Output), or into the test tree only for the run, and remove them from the project afterwards.

wisdom.md

将捕获型测试描述为一次性的测试：它们被生成用于在有 bug 的 diff 上失败，并且永远不会加入代码库。通常，用户要求“为我的变更编写测试”时，希望得到可以保留的测试。这个 skill 会生成两种测试，并以不同方式处理它们：

步骤 6 的结果	它是什么	应该怎么做
在变更版本上通过，在父版本或变异体上失败	已证明的强化测试：可防止此变更在未来发生回归	默认保留在项目的测试目录中
在变更版本本身失败	弱捕获：变更可能有 bug，或者测试误解了意图	绝不提交。用一行询问用户：“它过去会执行 X，现在会执行 Y；这是有意为之吗？”如果回答“否”，说明代码有 bug：修复代码，测试随即成为可保留的回归测试。如果回答“是”，说明测试写错了：删除测试
在两者上都通过	对该变更没有捕获能力	重写测试以针对步骤 3 中的某项风险，或者删除测试

之所以保留已证明的测试：

wisdom.md

中的“一次性”规则之所以存在，是因为大型流水线会为每次 diff 生成数千个未经审查的测试，根本无人维护。而这里，用户只要求了少量测试，会亲自审查，而且每个测试都已被证明会在真实损坏上失败，因此值得付出维护成本。

在写入前询问用户，当意图不清楚时，例如用户说“jit tests”或“catching tests”（这表明测试应是一次性的），或者仓库没有测试目录时。询问：“将已证明的测试保留在仓库中，还是将其作为一次性的捕获测试运行，并且只报告发现的问题？”在一次性模式下，将测试写入临时目录（参见“输出”），或者仅为本次运行写入测试目录，之后再从项目中移除。

Workflow

工作流程

Step 1: Identify Changed Code

步骤 1：识别变更代码

bash

git diff --name-only HEAD          # staged and unstaged
git status --porcelain             # new untracked files too

If no changes exist, ask the user which files or commit range to target (for a range, the parent is the range's base instead of

HEAD

bash

git diff --name-only HEAD          # staged and unstaged
git status --porcelain             # new untracked files too

如果不存在变更，请询问用户要针对哪些文件或提交范围（对于范围，使用该范围的基线作为父版本，而不是

HEAD

）。

Step 2: Read the Change and Its Existing Tests

步骤 2：读取变更及其现有测试

For each changed source file:

Read the full file and the diff (
```
git diff HEAD -- <file>
```
).
Find existing tests. Grep for the module name, the changed function names, and the file path. Grep alone misses indirection, so also read the imports and fixtures of the test files nearest the code (loader helpers,
```
conftest.py
```
, test utilities that import by path). A test that runs a script by path (for example
```
bash scripts/foo.sh
```
) covers it too, as does a CI step that runs it.
Run the existing tests that touch the file against the change, and note whether they already fail on the parent (Step 6 does this). If they already catch the change, say so and generate only the gaps.

对于每个变更的源文件：

读取完整文件及其 diff（
```
git diff HEAD -- <file>
```
）。
查找现有测试。使用 grep 搜索模块名、变更的函数名和文件路径。仅靠 grep 会遗漏间接引用，因此还要读取离代码最近的测试文件的导入（imports）和 fixtures（loader 辅助函数、
```
conftest.py
```
、通过路径导入的测试工具）。通过路径运行脚本的测试（例如
```
bash scripts/foo.sh
```
）也覆盖该代码，CI 中运行它的步骤同样如此。
针对该变更运行涉及该文件的现有测试，并记录它们是否已经在父版本上失败（步骤 6 会执行此操作）。如果现有测试已经捕获了该变更，应明确说明，并且只针对覆盖缺口生成测试。

Step 3: List Plausible Regressions First

步骤 3：首先列出可能的回归

Before writing any test, write down 3-5 concrete ways this change could be wrong or could later be broken. These are the mutants your tests must kill. Draw them from the diff itself:

Reverting the change (the "dodgy diff": the parent is the first mutant)
An off-by-one or flipped comparison in a changed condition
A changed branch taken for the wrong inputs: boundary values, empty, null, zero, negative, fractional, unicode, paths with spaces
An error path that now swallows or misreports a failure
A side effect that runs twice, runs in the wrong place, or is not idempotent on a rerun
A second caller or entry point that bypasses the changed code

Each test you write in Step 5 should name the regression it targets. Include this list in the report: it is the reader's quickest way to judge whether the tests cover the right risks.

在编写任何测试前，先列出该变更可能出错或在未来被破坏的 3 至 5 种具体方式。这些就是测试必须能够杀死的变异体。应从 diff 本身提取它们：

撤销变更（“可疑 diff”：父版本是第一个变异体）
变更条件中出现差一错误或比较结果反转
变更后的分支针对错误输入被采用：边界值、空值、null、零、负数、小数、unicode、带空格的路径
错误路径现在会吞掉失败或错误报告失败
副作用运行两次、运行位置错误，或者重新运行时不具备幂等性
第二个调用方或入口点绕过变更后的代码

步骤 5 中编写的每个测试都应命名其针对的回归。报告中应包含此列表：这是读者判断测试是否覆盖了正确风险的最快捷方式。

Step 4: Detect the Runner and Harness

步骤 4：检测测试运行器和测试框架

Find out how this project really runs tests before writing any. Check, in this order, and stop when you have a command that runs the existing tests green:

CI config:

.github/workflows/*.yml

.gitlab-ci.yml

.circleci/config.yml

Jenkinsfile

. The command CI runs is the ground truth, including wrappers like

uv run

poetry run

npx

Project manifests:

pyproject.toml

(

[tool.pytest.ini_options]

[tool.uv]

uv.lock

package.json

scripts.test

Makefile

justfile

test targets,

go.mod

Cargo.toml

tox.ini

noxfile.py

Existing test files: their framework and helpers. For shell code, look for
```
tests/*.sh
```
sourcing a shared harness (such as a
```
lib.sh
```
with
```
pass
```
/
```
fail
```
/
```
summarize
```
),
```
*.bats
```
files (bats-core), or
```
shunit2
```
.

Run the existing suite once with the detected command to confirm it works before adding anything. If the system interpreter lacks the framework (for example no

pytest

on the system

python3

), use the project's wrapper rather than installing packages globally.

在编写任何测试前，先弄清楚项目实际如何运行测试。按以下顺序检查，找到一个能让现有测试全部通过的命令后即停止：

CI 配置：
```
.github/workflows/*.yml
```
、
```
.gitlab-ci.yml
```
、
```
.circleci/config.yml
```
、
```
Jenkinsfile
```
。CI 运行的命令是事实依据，包括
```
uv run
```
、
```
poetry run
```
、
```
npx
```
等封装命令。

项目清单：

pyproject.toml

（

[tool.pytest.ini_options]

、

[tool.uv]

、

uv.lock

）、

package.json

的

scripts.test

、

Makefile

justfile

中的 test 目标、

go.mod

、

Cargo.toml

、

tox.ini

、

noxfile.py

。

现有测试文件：它们的框架和辅助工具。对于 shell 代码，查找引用共享 harness 的
```
tests/*.sh
```
（例如包含
```
pass
```
/
```
fail
```
/
```
summarize
```
的
```
lib.sh
```
）、
```
*.bats
```
文件（bats-core）或
```
shunit2
```
。

在添加任何内容前，使用检测到的命令运行一次现有测试套件，以确认其正常工作。如果系统解释器缺少框架（例如系统

python3

中没有

pytest

），使用项目封装的命令，而不是全局安装软件包。

Step 5: Write the Tests

步骤 5：编写测试

Match conventions: framework, file location, naming, import style (including loader indirection found in Step 2), and harness.
Test behavior, not implementation: assert on outputs, exit codes, and side effects.
One behavior per test: parametrize over cases (
```
pytest.mark.parametrize
```
, table-driven tests) instead of copying a test per input.
Name the regression: the test name or a one-line comment says which Step 3 risk it targets.
Realistic data: domain values, not "foo"/"bar".
Isolate side effects: scripts and functions that write files, touch
```
$HOME
```
, edit dotfiles, install, symlink, or call the network must run against fixtures, never against the user's real environment:
- Create a temp dir (
```
mktemp -d
```
  ,
```
tmp_path
```
  ,
```
t.TempDir()
```
  ) and remove it in a
```
trap ... EXIT
```
  or fixture teardown.
- Point
```
HOME
```
  (and
```
XDG_CONFIG_HOME
```
  ,
```
XDG_DATA_HOME
```
  if used) at the temp dir for the run, and pass explicit target paths when the script accepts them.
- Never run a no-argument "install to default location" mode outside a temp
```
HOME
```
  .
- Replace network calls and external services with stubs (a fake binary first on
```
PATH
```
  , a mock, a local fixture).
Rerun safety: if the code mutates state, run it twice in the test and assert the second run leaves the same result. Rerun bugs are common and cheap to catch.

匹配约定：框架、文件位置、命名、导入风格（包括步骤 2 中发现的 loader 间接引用）以及 harness。
测试行为，而非实现：断言输出、退出码和副作用。
每个测试只验证一种行为：使用参数化（
```
pytest.mark.parametrize
```
、表驱动测试），而不是为每个输入复制一个测试。
命名回归：通过测试名称或一行注释说明其针对步骤 3 中的哪项风险。
使用真实数据：采用领域值，而不是“foo”/“bar”。
隔离副作用：会写文件、触碰
```
$HOME
```
、编辑 dotfiles、执行安装、创建符号链接或调用网络的脚本和函数，必须针对 fixtures 运行，绝不能针对用户的真实环境运行：
- 创建临时目录（
```
mktemp -d
```
  、
```
tmp_path
```
  、
```
t.TempDir()
```
  ），并通过
```
trap ... EXIT
```
  或 fixture teardown 移除。
- 在运行期间将
```
HOME
```
  （以及使用到的
```
XDG_CONFIG_HOME
```
  、
```
XDG_DATA_HOME
```
  ）指向临时目录，并在脚本接受目标路径时显式传入这些路径。
- 绝不在临时
```
HOME
```
  之外运行无参数的“安装到默认位置”模式。
- 用 stub 替换网络调用和外部服务（优先在
```
PATH
```
  中放置伪二进制文件，也可使用 mock 或本地 fixture）。
重新运行安全性：如果代码会改变状态，在测试中运行两次，并断言第二次运行得到相同结果。重新运行时出现的 bug 很常见，而且很容易捕获。

Step 6: Prove the Tests Catch (Required)

步骤 6：证明测试能够捕获问题（必需）

A test only counts if it fails on a broken version. Do this for every new test file before reporting:

Run the new tests on the change. They must pass (or see the "fails on the change" row in Keep or Throw Away).
Run them on the parent: the changed source files as they are at
```
HEAD
```
(or the range base). Leave the new test files in place. At least one new test must fail.
If the change is new code with no parent (a new file or function), or the parent run cannot work, apply a hand-made mutant instead: one of the Step 3 regressions as a small patch to the changed lines. At least one new test must fail on it. Prefer two or more mutants when the change has several branches.
Restore the working tree and confirm
```
git diff
```
shows only the user's change plus your new tests.

The bundled script does steps 1-4 safely, restoring the files even on Ctrl-C:

bash

bash "${CLAUDE_SKILL_DIR}/scripts/prove-catch.sh" --file path/to/changed.py -- uv run pytest tests/unit/test_new.py -q
bash "${CLAUDE_SKILL_DIR}/scripts/prove-catch.sh" --mutant /tmp/off-by-one.patch -- bash tests/new-tests.sh

${CLAUDE_SKILL_DIR}

is this skill's directory; if your agent does not expand it, use the directory containing this

SKILL.md

. Pass only source files to

--file

, never the new tests. Do not use

git stash

for this: it also stashes unrelated work and can conflict on pop.

If no test fails on any broken version, the tests catch nothing: go back to Step 3 and target a different regression. If you cannot make one fail, say so plainly in the report instead of claiming coverage.

Then run the full existing suite with the new tests added, to confirm no collateral breakage.

一个测试只有在损坏版本上失败才算有效。报告前，对每个新测试文件执行以下操作：

在变更版本上运行新测试。测试必须通过（或参见“保留还是丢弃”中“在变更版本本身失败”这一行）。
在父版本上运行测试：使用
```
HEAD
```
（或范围的基线）中的变更源文件。保留新测试文件。至少一个新测试必须失败。
如果变更是没有父版本的新代码（新文件或新函数），或者父版本无法运行，则改为手工制作一个变异体：将步骤 3 中的某项回归作为对变更行的小型补丁。至少一个新测试必须在该变异体上失败。当变更包含多个分支时，优先使用两个或更多变异体。
恢复工作树，并确认
```
git diff
```
只显示用户的变更和新测试。

内置脚本会安全地执行步骤 1 至 4，即使按下 Ctrl-C 也会恢复文件：

bash

bash "${CLAUDE_SKILL_DIR}/scripts/prove-catch.sh" --file path/to/changed.py -- uv run pytest tests/unit/test_new.py -q
bash "${CLAUDE_SKILL_DIR}/scripts/prove-catch.sh" --mutant /tmp/off-by-one.patch -- bash tests/new-tests.sh

${CLAUDE_SKILL_DIR}

是此 skill 所在目录；如果你的 Agent 不会展开它，请使用包含此

SKILL.md

的目录。只将源文件传给

--file

，绝不传入新测试。不要使用

git stash

：它还会暂存无关工作，并且可能在 pop 时发生冲突。

如果任何损坏版本上都没有测试失败，这些测试就没有捕获任何问题：返回步骤 3，针对不同的回归重新设计。如果无法让测试失败，应在报告中直截了当地说明，而不是声称已实现覆盖。

然后加入新测试，运行完整的现有测试套件，以确认没有连带破坏。

Step 7: Wire Into CI

步骤 7：接入 CI

Check whether CI picks up the new test file automatically (pytest discovery,

go test ./...

) or lists test files by name (for example, one workflow step per

tests/*.sh

). If CI lists them by name, the new file never runs in CI until it is added: add the step, or, if CI config is out of scope for the user, name the exact line to add in the report.

检查 CI 是自动发现新测试文件（例如 pytest discovery、

go test ./...

），还是按名称列出测试文件（例如每个

tests/*.sh

对应一个 workflow step）。如果 CI 按名称列出，新文件在添加前不会在 CI 中运行：添加相应步骤；如果 CI 配置不在用户关注范围内，则在报告中指出需要添加的确切行。

Output

输出

Print the report in the conversation. If the user wants a file, or the report is long, also save it outside the repo so it never ends up in a commit:

${TMPDIR:-/tmp}/jit-tests/<repo-name>-<YYYYMMDD-HHMM>/report.md

, and tell the user the path. Throwaway catching tests go in the same directory. Kept tests go in the project's test tree.

For each changed file:

markdown

undefined

在对话中打印报告。如果用户需要文件，或者报告较长，还应将报告保存到仓库之外，使其永远不会进入提交：

${TMPDIR:-/tmp}/jit-tests/<repo-name>-<YYYYMMDD-HHMM>/report.md

，并告知用户路径。一次性捕获测试也放在同一目录。可保留的测试放入项目的测试目录。

对于每个变更文件：

markdown

undefined

Tests for

path/to/changed-file.ext

Tests for

path/to/changed-file.ext

Change: [one line on what changed and why] Existing coverage: [tests that touch this code, or "none"; whether they already fail on the parent] Plausible regressions: [the Step 3 list, numbered] Runner:

[exact command]

(from [CI file / pyproject / Makefile / ...]) New tests: [count] in

path/to/test-file.ext

(kept | throwaway)

Catch proof:

Version	Passed	Failed
Change	N	0
Parent ( `HEAD` )	N	M
Mutant: [regression #k, one line]	N	M

Full suite: [N passed, 0 failed] CI: [picked up automatically | added to <file> | needs this line in <file>: ...]

[exact command]

(from [CI file / pyproject / Makefile / ...]) New tests: [count] in

path/to/test-file.ext

(kept | throwaway)

Catch proof:

Version	Passed	Failed
Change	N	0
Parent ( `HEAD` )	N	M
Mutant: [regression #k, one line]	N	M

Full suite: [N passed, 0 failed] CI: [picked up automatically | added to <file> | needs this line in <file>: ...]

Test: [descriptive name]

Catches: regression #k ([one line])


End with any weak catches (tests that failed on the change) as "used to X, now Y; intended?" questions, and any real bugs found while probing, even outside the diff.

Catches: regression #k ([one line])


最后，将任何弱捕获（在变更版本上失败的测试）以“过去会执行 X，现在会执行 Y；这是有意为之吗？”的问题形式列出，并报告探测过程中发现的任何真实 bug，包括 diff 之外的 bug。

What Not to Generate

不要生成的测试

Tests for trivial getters and setters
Tests that duplicate existing coverage (check Step 2 first)
Tests tied to implementation details (private names, call order, exact log wording) unless that is the behavior that changed
Tests for generated or vendored code
Tests for pure configuration data. A script or CI step that consumes config is code and can be tested

针对简单 getter 和 setter 的测试
与现有覆盖重复的测试（先执行步骤 2）
与实现细节绑定的测试（私有名称、调用顺序、准确的日志措辞），除非这些正是发生变更的行为
针对生成代码或 vendored 代码的测试
针对纯配置数据的测试。消费配置的脚本或 CI 步骤属于代码，也可以进行测试

review-jit-test-gen

Original

Translation

JIT Test Generation

JIT 测试生成

Keep or Throw Away

保留还是丢弃

Workflow

工作流程

Step 1: Identify Changed Code

步骤 1：识别变更代码

Step 2: Read the Change and Its Existing Tests

步骤 2：读取变更及其现有测试

Step 3: List Plausible Regressions First

步骤 3：首先列出可能的回归

Step 4: Detect the Runner and Harness

步骤 4：检测测试运行器和测试框架

Step 5: Write the Tests

步骤 5：编写测试

Step 6: Prove the Tests Catch (Required)

步骤 6：证明测试能够捕获问题（必需）

Step 7: Wire Into CI

步骤 7：接入 CI

Output

输出

Tests for
`path/to/changed-file.ext`

Tests for
`path/to/changed-file.ext`

Test: [descriptive name]

Test: [descriptive name]

What Not to Generate

不要生成的测试