Testing Boss

test-antipatterns

Iron Laws

1. Test the behavior, never the mock.
2. Push every test to the lowest layer that can detect the failure.
3. When a test fails, fix production first — change the test only after writing why.
4. Real systems gate the merge. Mocks isolate; they do not validate.
5. Coverage is a flashlight. Mutation score is a quality probe. Neither is a target.
6. No test-only methods, branches, or flags leak into production code.

Required Reading Router

references/foundations.md

references/patterns.md

references/antipatterns.md

references/ai-writes-tests.md

references/antipatterns.md

references/ci-automation.md

references/llm-eval.md

references/sources.md

Reference Index

• references/foundations.md

  — placement doctrine (invariant + owning layer), pyramid vs trophy debate resolution, risk-based prioritization, coverage philosophy, test-boundary contracts.

• references/patterns.md

  — 12 cross-framework positive patterns with agnostic pseudo-code: selector hierarchy, condition-based waits, per-test isolation, table-driven, builders/factories, behavior-first assertions, boundary-only mocking.

• references/antipatterns.md

  — 25 anti-patterns across five families (Brittleness, Flakiness, Mock misuse, Process, AI-specific). Each entry: violation, why wrong, fix, gate question, evidence URL.

• references/ai-writes-tests.md

  — seven mandatory gates with verbatim prompt blocks for any agent that generates tests: invariant first, owning layer, real execution, failure→fix production, no snapshot without contract, no assertion on self-set mock, negative companion.

• references/ci-automation.md

  — flaky-test taxonomy, quarantine-plus-owner workflow, CI stage pyramid, contract / property / mutation / accessibility testing patterns, deterministic test architecture.

• references/llm-eval.md

  — eval-driven development primer, oracle ladder, LLM-as-judge biases and calibration, RAG metrics, agent trajectory vs outcome eval, benchmark pitfalls.

• references/sources.md

  — consolidated bibliography (all URLs grouped by axis) for citation and audit.

Decide before the first line of test code

• Name the invariant in one sentence before opening any test file. If the sentence is fuzzy, the invariant is not clear enough to test.
• Place the test at the lowest layer that can fail when the invariant breaks. A higher-layer test is justified only when the invariant requires real integration the lower layer cannot prove.
• Reject the test entirely when (likelihood-of-bug × blast-radius) is below the threshold worth ten minutes of maintenance. Not every line deserves a test.

references/foundations.md

Pattern catalog (write tests that survive refactor)

1. Query by behavior and accessible role, never by CSS selector or DOM index.
2. Selector hierarchy: role → label → text → test-id → structural. Stop at the highest rung that disambiguates.
3. Wait on observable conditions, never on wall-clock sleeps.
4. Each test is independent and order-free; setup beats teardown.
5. One behavior per test, but as many assertions as that behavior needs.
6. Test names read as specifications:

   should <outcome> when <condition> given <state>

   .
7. Table-driven / parameterized when only the inputs vary.
8. Build test data via factories or builders; literal blobs only for the field under test.
9. Mock at boundaries you do not control; real wiring for what you own.
10. Real systems gate the final merge; contract tests bridge unit and E2E.
11. Mutation score, not coverage percentage, measures suite strength.
12. Page Object Model is a tool, not a religion — collapse it for small suites.

references/patterns.md

Anti-pattern families (do not do these)

1. Brittle/implementation-detail selectors.
2. Testing internal structure instead of observable behavior.
3. Testing private methods directly.
4. Snapshot-as-test (a snapshot replacing real assertions).
5. Vague existence assertions (

   .should('exist')

   ,

   toBeTruthy

   ).
6. Action without assertion.

sleep

test-antipatterns

beforeAll

afterEach

beforeEach

test-antipatterns

references/antipatterns.md

AI agents writing tests

1. Invariant first — agent prints

   INVARIANT: …

   ,

   OWNING_LAYER: …

   ,

   EXISTING_SUITE: …

   before any test code.
2. Owning layer — extend an existing suite; reject new files without a named invariant.
3. Real execution — every new test must run against a real DB / route / external integration at least once.
4. Failure → fix production — on red, the next tool call reads the production code, not the test.
5. No snapshot without contract — classify the artifact as

   PRODUCT_CONTRACT

   or

   IMPLEMENTATION_DETAIL

   ; the latter forbids snapshots.
6. No assertion on self-set mock — cannot assert on a value the same test body wrote into a mock.
7. Negative companion — every positive assertion ships with a negative test for invalid input or failure mode.

references/ai-writes-tests.md

CI & flaky discipline

• Quarantine a flaky test the same hour it is detected. Assign a named human owner within 24 hours with a fix-by date. No anonymous quarantines.
• Track

  flaky_rate

  as a first-class operational metric. SLO < 1–2%; alert at > 5%. Retry without telemetry is debt accrual, not stability.
• Real systems at the final gate. Mock at unit; contract-test the boundary; real DB / queue / route at integration; near-zero mocks at E2E.

references/ci-automation.md

LLM and agent evaluation (Part 6, enxuta)

• Start small. Twenty unambiguous tasks drawn from real failures beat two hundred synthetic ones.
• Climb the oracle ladder: exact / schema / outcome-state checks before LLM-as-judge before human review. Use the cheapest oracle that catches the failure.
• LLM-as-judge needs calibration. Validate against humans (target ≥ 0.80 Spearman) before trusting any judge in CI. Always use a different model than the system under test.
• Agents need outcome checks. Trajectory grading punishes valid creativity; outcome-only grading misses ghost actions where the transcript claims success and nothing changed.

references/llm-eval.md

Red flags (cross-cutting)

• Mock setup is larger than the test logic.
• Test breaks when an internal method is renamed (not a public contract).
• Removing the assertion body leaves the test still green.
• Test fails when run with

  .only

  in isolation.

• sleep

  ,

  Thread.sleep

  , or

  cy.wait(<number>)

  appears anywhere.
• Selector contains a CSS class, an index, or

  xpath

  .
• Test asserts a third-party site is reachable.
• Snapshot diffs are accepted in code review without reading them.
• Coverage percentage is the only quoted quality metric.
• Failing tests are auto-retried until green; nobody investigates.
• Skipped or quarantined tests have no named owner and no fix-by date.
• Test depends on

  new Date()

  ,

  Math.random()

  , or system locale.

• afterEach

  resets the database (move it to

  beforeEach

  ).
• An AI-written test has six+ assertions and zero edge cases.
• The phrase "I'll mock this to be safe" appears anywhere in the diff.

When NOT to use this skill

• General code review unrelated to tests — use a code-review skill.
• Library-specific debugging where the test is just the reproduction — use the library's own debugging skill.
• Non-testing CI pipeline design (deploys, artifact promotion, secrets management).
• Production observability and alerting design — those are runtime concerns, not test concerns.
• Single-line typo fixes in existing tests — the doctrine is for non-trivial work.

Bottom line

A test that cannot fail is decorative. A test that fails for the wrong
reason is misleading. Build tests that fail for exactly one reason —
the reason the invariant was violated — and trust them when they do.

Mocks isolate. Real systems validate. Coverage shines a light. Mutation
score grades the suite. Agents will reach for the mock and the snapshot;
the gates here make them put both down.

Tests reveal bugs, not just pass.