Mutation Adequacy Skill
Overview
The
review dimension is the
adequacy backstop for the relaxed verification
mix (verification ladder slice 3, R5). When the planner ships the cheaper testing strategy — strict
types + inline invariants + one PBT + one acceptance test instead of granular per-behavior red-green
— mutation adequacy is the machine guard against the vacuous tests (and vacuous PBT properties) that
omission could otherwise admit. It scores whether the test suite actually
kills injected mutants:
the strongest signal that tests fail for the right reason.
This dimension gates the
HIGH risk tier only, at the
boundary only — it is the
review-phase counterpart to the delegation-time
(R3) gate. The coupling to the
high tier is policy data in
(the dimension name = this skill's folder name); you
never decide tier membership in this skill. The gate is registered as a required
D1 review
dimension (
adaptLadderGate(MUTATION_GATE_NAME, 'D1', handleMutationAdequacy)
) — a structural
participation fact, distinct from severity (below).
Two orthogonal axes, two default severities. The mutation
score stays advisory by default: a
sub-threshold
surfaces survivor follow-ups and warns, but does not block, unless an
explicit
review.gates['mutation-adequacy']
config override raises its severity. A 100% score is
neither expected nor required (equivalent mutants exist).
The
NoCoverage axis is a second, independent check DR-6 added, with its own real, functioning
enforcement path: under
review.mutationEnforcement: block
, the block-mode guard
(
Check 4b) fails the
transition whenever the diff-scoped
count exceeds the configured
budget (default
0 — zero uncovered
changed lines tolerated), independent of the score. This is live code, not a deferred plan — but it is
opt-in, the same way the score's severity is:
review.mutationEnforcement
defaults to
, and
's per-gate severity default is deliberately advisory too (so a project's
dimension-level defaults never silently re-block a demoted gate). A project flips NoCoverage
enforcement on with one config line (
review.mutation-enforcement: block
); honor whichever way it's
resolved, never hardcode "advisory" when recording this dimension's result.
The CI side is separate and narrower: a diff-scoped mutation gate runs on pull-request events, but it
runs in
observe mode unconditionally — it logs its verdict and always exits 0, regardless of any
setting. It is not yet a blocking CI backstop; the flip is deferred, pending a clean
runner verdict on the full server suite. Never represent the CI gate as a merge blocker — where
NoCoverage enforcement is live, it is on the review path above, not in CI.
Note:
advisory (where it applies) refers to whether a sub-threshold score or an over-budget
NoCoverage count actually stops the transition. The dimension
entry is required on the high tier
regardless of either axis's resolved severity (the
guard fails if the
review is absent), so run and record it every time.
Triggers
Activate this skill when:
- reaches the quality stage on a HIGH-tier feature
- The review contract lists in the required reviews for this workflow
- You need to assess whether the (possibly relaxed) test mix actually kills mutants
Do
not activate for medium/low-tier work, or outside the
boundary — those paths do not
require this dimension.
Execution
Step 1: Run the diff-scoped mutation gate
Invoke the action against the review/PR base ref. It runs the resolved mutation command
diff-scoped (Stryker
, cargo-mutants
, mutmut path restriction — resolved from
the toolchains SoT, never composed by hand), so the run completes in
, not the full-tree
time budget.
typescript
exarchos:exarchos_orchestrate({
action: "mutation-adequacy",
featureId: "<featureId>",
base: "<review/PR base ref>", // e.g. "main" — reuse the same base the review diff uses
worktreePath: "<optional worktree>", // 'auto' resolves the calling delegation's worktree
operationId: "<optional idempotency key>"
// scope defaults to "diff"; do NOT pass scope:"full" here (see Anti-Patterns)
})
The action emits
mutation.executing_started
/
(liveness, INV-10) and a foldable
carrying
(INV-1) automatically. Do
not hand-emit these events.
Step 2: Read the carrier
The action returns the fixed carrier (
). The shape is stable regardless of pass/fail/degrade:
| Field | Meaning |
|---|
| mutationScore >= threshold && noCoverage <= maxNoCoverage
|
| killed / (total − noCoverage)
|
| mutants a test caught (good) |
| mutants that escaped — tests ran but did not catch them |
| mutants in code no test exercises at all |
| total mutants generated within the diff scope |
| the effective advisory threshold (override > config > soft default) |
| the parsed Stryker mutation-testing-report-schema
|
| "write a test that kills " follow-ups (Step 3) |
Degrade signals to recognize (each returns
so the gate never blocks closed-with-error):
- + — no mutation runner resolved. Report the remediation; do not treat as a failure.
- + a entry — the runner produced no parseable report. Note it; do not throw.
- + — you (incorrectly) requested full scope; re-run with the default diff scope.
See
references/reading-the-carrier.md
for the full carrier and report shape.
Step 3: Turn survivors into kill-this-mutant follow-ups (INV-12)
The action already maps each surviving and
mutant to a
entry of the form
"write a test that kills ". Surface these as concrete, actionable review findings —
each one names a specific assertion the suite is missing:
- A survived mutant means a test exercises that line but asserts nothing strong enough to detect
the mutation. The follow-up: add an assertion that distinguishes the mutated behavior.
- A NoCoverage mutant means no test touches that line at all. The follow-up: add a test that
exercises it, then assert on the observable behavior.
Record these as
with
so they ride the same review-report contract
as the other dimensions. Do not invent generic "improve coverage" advice — quote the file:line the
action surfaced.
Step 4: Apply the verdict — two axes, each with its own resolved severity
The
score axis stays advisory (warning) by default. A sub-threshold
:
- surfaces the survivor follow-ups (Step 3),
- warns in the review report,
- does not block the transition — unless an explicit
review.gates['mutation-adequacy']
references/advisory-threshold.md
The
NoCoverage axis is independent, with its own real block-mode enforcement path: under
review.mutationEnforcement: block
, if the folded
count exceeds the configured
budget (default 0), the block-mode guard fails the transition regardless of the score.
Like the score, it is opt-in (
review.mutationEnforcement
defaults to
) — resolve and honor
the actual configured mode, don't assume either axis is on or off. Record both axes honestly in the
review report — do not fold a blocking NoCoverage failure into advisory-score language, and do not
describe either axis as enforced by CI (the diff-scoped CI wiring runs in observe mode; see
Anti-Patterns).
Required Output Format
Record the dimension result on the review state. The review key MUST be the kebab-case dimension name
(it equals this skill's folder name):
typescript
exarchos:exarchos_workflow({ action: "update", featureId: "<id>", updates: {
reviews: { "mutation-adequacy": {
status: "pass", // "pass" is honest when both axes are advisory (the default); if either
// axis's severity is configured to block, report the real result instead
summary: "mutationScore 0.62 (threshold 0.40); 3 survivors surfaced as kill-test follow-ups",
issues: [
{ severity: "MEDIUM", category: "test-quality", file: "src/foo.ts", line: 42,
description: "surviving mutant — no assertion distinguishes the mutated branch",
required_fix: "write a test that kills src/foo.ts:42" }
]
} }
}})
A passing-value
(
pass | passed | approved | fixes-applied
, case-insensitive) is required
for the
guard — a flat string is silently ignored and blocks the transition.
Anti-Patterns
| Don't | Do Instead |
|---|
| Run inline | Full-tree mutation is the long-running op deferred to R10/v2.12 — it returns a deferred advisory, never an inline run |
| Compose / by hand | Let the action resolve the diff scope from the toolchains SoT |
| Block the merge on a sub-threshold score with no config override | The score axis is advisory by default — surface follow-ups, honor the resolved severity |
| Assume NoCoverage is always advisory, or always blocking | It has its own real block-mode path (review.mutationEnforcement: block
|
| Claim the CI mutation gate blocks merges | It runs diff-scoped in observe mode unconditionally (logs its verdict, always exits 0, regardless of config) — never represent it as a merge blocker |
| Treat a Skipped/Warning carrier as a hard failure | Both return — report the reason, never throw |
| Run this dimension for medium/low tier | It gates the HIGH tier at the boundary only |
| Emit manually | The action auto-emits liveness + the foldable gate event |
| Give generic "add more tests" advice | Quote the the action surfaced in |
References
references/reading-the-carrier.md
mutation-testing-report-schema
references/advisory-threshold.md