ai-qa-review

.agents/qa-project-context.md

1. Review scope: Reviewing test code for quality, application code for testability, or both? Each triggers a different Quick Route path.
2. Framework conventions: What test framework (Jest, Vitest, Playwright, pytest)? Conventions differ —

   describe/it

   nesting, fixture usage, assertion style — and the Verification commands are per-framework.
3. PR review or batch audit? A PR review runs and scores only the changed files. A batch audit scans the entire suite for systemic patterns.
4. Existing quality standards: Does the team have documented test conventions? Check for

   .eslintrc

   test rules,

   CONTRIBUTING.md

   test guidelines, or a test style guide.
5. Known pain points: Recurring flaky tests, slow suites, unclear failures? These prioritize which smells to focus on first.

1. Test code is production code. Apply the same quality standards: readability, maintainability, single responsibility. Test code that is hard to read is hard to trust.
2. Review what is asserted, not just what is executed. Coverage proves a line ran; it says nothing about whether a wrong value would be caught. A 95%-coverage suite of

   toBeTruthy

   assertions catches almost nothing. Mutation score (see Verification) measures the thing coverage can't.
3. Testability review prevents test debt. Reviewing application code for testability catches design problems before they force awkward test workarounds. If code is hard to test, it is usually hard to maintain.
4. Codify patterns, not just knowledge. Turn recurring review feedback into lint rules, custom ESLint plugins, or shared fixtures. Reviews that repeat the same feedback indicate missing automation.
5. Smells are symptoms, not verdicts. A test smell indicates a potential problem; context decides whether it is actually harmful. A long test for a complex workflow may be appropriate. A mock-heavy test for a boundary may be correct.
6. Actionable feedback only. Every review comment must include what is wrong, why it matters, and how to fix it. "This test is bad" is not actionable. "This test uses sleep-based waiting which causes flakiness — replace with an explicit wait condition" is.

data-testid

@testing-library/something-that-doesnt-exist

example.com

test@test.com

Lorem ipsum

John Doe

shift-left-testing

ai-test-generation

promptfoo eval

deepeval test run

ragas evaluate

Promptfoo ownership note: Promptfoo was acquired by OpenAI (announced 9 Mar 2026). The core stays MIT-licensed, open source, and model-agnostic; red-team capabilities are being folded into OpenAI Frontier.

promptfoo eval

is still the correct quality-gate command — just expect the vendor to be OpenAI going forward.

• Can I understand what each test verifies in under 10 seconds?
• Is setup minimal and test-relevant?
• Are test names descriptive: "should [behavior] when [condition]"?

• No sleep/waitForTimeout/setTimeout for synchronization?
• No shared mutable state between tests?
• No dependency on test execution order?
• No calls to real external services?

• Will failures produce messages that identify the problem?
• Does each test have one reason to fail?
• Are assertions specific (not toBeTruthy/toBeDefined)?

• No conditional logic (if/else/switch/for) in test bodies?
• Fixtures/setup proportional to what each test needs?
• Mocking limited to external boundaries?
• Parameterized tests (it.each) used for data-driven scenarios?

• Locators verified against a real page (no hallucinated data-testid)?
• Imports resolve (no fabricated APIs)?
• Project data factory used, not test@test.com / John Doe?

• Happy path AND error/negative paths tested?
• Boundary values tested (0, 1, max, max+1)?
• Edge cases: empty, null, duplicate, concurrent?

undefined

1. Quantify: Count tests by type (unit/integration/E2E), framework, and directory.
2. Sample: Review 10-20% of test files, prioritizing the largest and most recently changed.
3. Pattern: Identify the 3-5 most common smells across the sample.
4. Prioritize: Rank by impact: reliability smells > diagnostic smells > design smells > readability smells.
5. Automate: For each common smell, determine if an ESLint rule or a mutation-score gate can catch it automatically.
6. Report: Document findings (one row per file reviewed) with specific examples, suggested fixes, severity (high/medium/low), and estimated effort.

1. Review test quality: "Check this test file for readability, reliability, diagnostic, design, AI-generated, and coverage smells. For each issue: name the smell, cite the line, explain why it matters, and provide a fix."
2. Identify coverage gaps: "Given this application code and its existing tests, identify missing scenarios across happy path, error handling, boundaries, edge cases, concurrency, and security. Prioritize as P0/P1/P2."
3. Testability improvements: "Review this application code for hard-coded dependencies, mixed side effects, extractable pure functions, and overly broad interfaces. Show current vs. refactored code."

1. Reviewing test code with production-code standards only. Test code has additional quality dimensions (reliability, diagnostics, coverage) that production-code linters do not check. Apply the six smell buckets, not just "clean code" principles.
2. Flagging every smell without context. A 50-line test for a complex state machine is not obscure setup — it is necessary complexity. Evaluate smells against the behavior being tested.
3. Suggesting mocks for everything. Over-mocking is itself a smell. Do not recommend mocking pure functions, value objects, or fast in-process collaborators. Mock boundaries: network, database, filesystem, clock.
4. Focusing on coverage percentage over coverage quality. 95% line coverage with only happy-path tests is worse than 75% coverage that includes error paths and boundaries. Review what is asserted, not just what is executed — and back it with mutation score.
5. Reviewing without running. Static analysis misses runtime issues. Run the suite (see Verification), check for flakiness with repeated runs, and check execution time. A passing suite that takes 20 minutes has a performance smell.
6. One-time review without follow-up. Test quality degrades over time. Establish a recurring review cadence or automated quality gates (lint rule, mutation threshold) that catch regression.

1. Run the suite once. Confirm it is green before reviewing — never review red or skipped tests as if they pass.
   • Vitest:

     npx vitest run <files>

     · Jest:

     npx jest <files>

     · Playwright:

     npx playwright test <files>

     · pytest:

     pytest <files>

2. Repeat to surface flakiness. A test that passes once and fails on the next run is the reliability smell, not a fluke.
   • Vitest:

     npx vitest run --retry=0 <files>

     looped, or

     for i in 1 2 3; do npx vitest run <files> || break; done

   • Jest:

     npx jest <files> && npx jest <files> && npx jest <files>

     (or

     jest-circus

     repeat)
   • Playwright:

     npx playwright test --repeat-each=3 <files>

   • pytest:

     pytest --count=3 <files>

     (pytest-repeat) or

     pytest -p no:randomly

     vs random order to catch order dependency
3. Capture per-test timing. A 20-minute suite has a performance smell.
   • Vitest:

     --reporter=verbose

     (prints per-test duration) · Jest:

     --verbose

     · Playwright:

     --reporter=list

     · pytest:

     --durations=10

4. Mutation score — the objective backstop. For the qualitative smells you can't eyeball at scale (Closed AI loop, weak assertions), run a mutation runner and read the score. AI-generated tests under ~60% mutation score are not constraining the implementation. Full tooling, thresholds, and config in

   references/mutation-testing.md

   .
   • JS/TS:

     npx stryker run

     · Java: PIT

     mvn ... mutationCoverage

     · Rust:

     cargo mutants

     · Python:

     mutmut run

• The findings artifact exists with one row per reviewed file, each row carrying a severity rating (high/medium/low). (file exists, row count == files reviewed)
• Every smell dimension that applies to the reviewed code is covered: readability, reliability, diagnostic, design, AI-generated, coverage. (six headings present in the report; "N/A" allowed, blank not)
• Each high-severity finding has an actionable remediation step (what + why + fix), not just a description.
• Verification ran: the suite executed at least 3x green (no flaky failures) and per-test timing captured; for AI-generated suites, a mutation score is recorded.
• At least one recurring smell was converted to an automated gate — an ESLint rule, a CI check, or a mutation-score threshold committed — or the report explicitly states none was warranted.

• smell-examples.md — Full SMELL/FIX code for every catalogued smell across reliability, diagnostic, design, and coverage dimensions.
• testability-refactors.md — Hard-to-test vs. testable before/after for all four testability subsections: dependency injection, side effect isolation, pure function extraction, interface segregation.
• mutation-testing.md — Tooling (Stryker/PIT/cargo-mutants/mutmut), thresholds, Stryker config, and the AI-loop / weak-assertion gate.

• unit-testing — Framework-specific patterns (Jest, Vitest, pytest) that inform what "good" looks like for each framework.
• shift-left-testing — Pre-commit hooks and IDE integration that catch test smells before review.
• coverage-analysis — Interpreting coverage reports to find meaningful gaps, not just percentage targets.
• ai-test-generation — Generated tests need review too. Apply these smell checks to AI-generated test code.
• test-reliability — Reliability smells (sleep-based waits, order dependency) overlap with flaky test patterns.