Back to Details

ai-test-generation

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese
<objective> LLMs will happily emit fifty plausible-looking tests that assert nothing, target endpoints that do not exist, and duplicate each other. This skill is a staged pipeline that forces structured intermediates — assumptions, coverage matrix, oracle definitions — out of the model BEFORE any test code, so what you get is traceable, reviewable, and grounded in the real codebase instead of ad-hoc generated noise.
Before starting: Check for 
.agents/qa-project-context.md
in the project root. It carries tech stack, test frameworks, naming conventions, selector strategy, and known risk areas that dramatically improve generated test quality. </objective>
<objective> 大语言模型（LLM）会随意生成50个看似合理但实则毫无断言、指向不存在的端点且彼此重复的测试。本技能是一个分阶段流程，要求模型在生成任何测试代码之前先输出结构化中间产物——假设、覆盖矩阵、验证机制定义，这样你得到的测试是可追溯、可审核且基于真实代码库的，而非随意生成的冗余内容。
开始前准备： 检查项目根目录下的
.agents/qa-project-context.md
文件。它包含技术栈、测试框架、命名规范、选择器策略以及已知风险区域，能显著提升生成测试的质量。 </objective>

Quick Route

快速指南

The pipeline is the same for every input; only the Step 1 extraction emphasis changes. Jump to the matching row, then run Steps 2-7 unchanged.
Input typeStep 1 extractsWatch for
PRD / feature specEntities, business rules, acceptance criteria, NFRs, stated assumptionsImplicit requirements inferred from "seamless"/"fast" language
User story + ACEach AC → ≥1 happy + ≥1 negative scenarioACs that hide multiple behaviors in one line
Code diff (
git diff main...HEAD
)		New/changed code paths, modified conditionals, removed behaviorRegression scope: test the changed paths, not the whole module
Bug reportRepro steps, expected vs actual, environmentWrite a test asserting expected — fails now, passes after fix
OpenAPI / GraphQL SDLEndpoints, schemas, required fields, enums, authValidation, auth-failure, and edge cases per endpoint, not just 200s
Playwright projects also pick an agent integration mode — see Discovery Q2.
无论输入类型如何，流程都是相同的；仅第1步的提取重点会变化。跳转到匹配的行，然后按原样执行第2-7步。
输入类型第1步提取内容注意事项
PRD / 功能规格实体、业务规则、验收标准、非功能性需求（NFR）、明确说明的假设从“无缝”“快速”等表述中推断出的隐含需求
用户故事 + 验收标准（AC）每个AC对应≥1个正向场景 + ≥1个负向场景单个AC中隐藏多种行为的情况
代码差异（
git diff main...HEAD
）		新增/修改的代码路径、修改的条件判断、移除的功能回归测试范围：仅测试变更的路径，而非整个模块
Bug报告复现步骤、预期与实际结果、环境信息编写断言预期结果的测试——当前会失败，修复后会通过
OpenAPI / GraphQL SDL端点、Schema、必填字段、枚举、认证每个端点的验证、认证失败及边缘场景，而非仅200成功响应
Playwright项目还需选择代理集成模式——请查看发现环节问题2。

Discovery Questions

发现环节问题

Check 
.agents/qa-project-context.md
first — if it exists, use it and skip anything already answered there. Then clarify:
  1. What is the input source? PRD / spec, user story + AC, code diff, bug report, or API schema. Determines Step 1 extraction emphasis (see Quick Route). For an LLM/AI feature spec, stop — generate eval datasets in 
    ai-system-testing
    , not Playwright specs.
  2. What is the target test framework, and (for Playwright) which agent integration mode?
    • E2E: Playwright (preferred), Cypress. Unit: Jest, Vitest, pytest. API: Playwright 
      APIRequestContext
      , Supertest, requests.
    • Playwright CLI + agents (recommended for Claude Code / Codex / Cursor): 
      npx playwright init-agents --loop=claude
      scaffolds planner/generator/healer agents into 
      .claude/agents/
      as markdown. They are interactive dev tools that produce standard Playwright tests which run unchanged in CI. Token-efficient; runs inside the agent's loop.
    • Playwright MCP (
      npx @playwright/mcp@latest
      ): higher overhead, right when the agent must drive a live browser interactively over a long session.
    • Neither — hand-write tests using AI as a scratch-pad helper.
  3. What project context is available? Existing test patterns, Page Objects / helpers, data factories / fixtures, CI constraints (timeout, parallelism). More context = less cleanup.
  4. What is the review workflow? Full pipeline → human review → merge (default); scenarios only → human writes code; or code → human refines iteratively.
  5. What domain knowledge is needed? Regulated industry (healthcare, finance) compliance, domain invariants (money never negative, appointments cannot overlap), known risk areas from past incidents.
先检查
.agents/qa-project-context.md
文件——如果存在，直接使用并跳过已涵盖的问题。然后明确以下内容：
  1. 输入来源是什么？ PRD/规格文档、用户故事+验收标准、代码差异、Bug报告或API Schema。这决定了第1步的提取重点（见快速指南）。如果是LLM/AI功能规格文档，请停止——请在
    ai-system-testing
    中生成评估数据集，而非Playwright规格。
  2. 目标测试框架是什么？（针对Playwright）采用哪种代理集成模式？
    • 端到端（E2E）： Playwright（首选）、Cypress。单元测试： Jest、Vitest、pytest。API测试： Playwright 
      APIRequestContext
      、Supertest、requests。
    • Playwright CLI + 代理（推荐用于Claude Code/Codex/Cursor）：
      npx playwright init-agents --loop=claude
      会在
      .claude/agents/
      目录中生成规划器/生成器/修复器代理的Markdown文件。它们是交互式开发工具，可生成标准Playwright测试，在CI中无需修改即可运行。令牌效率高；在代理循环内运行。
    • Playwright MCP（
      npx @playwright/mcp@latest
      ）：开销较高，适用于代理必须在长时间会话中交互式驱动真实浏览器的场景。
    • 都不选——手动编写测试，将AI用作辅助工具。
  3. 有哪些可用的项目上下文？ 现有测试模式、页面对象/辅助工具、数据工厂/夹具、CI约束（超时、并行性）。上下文越丰富，后续清理工作越少。
  4. 审核工作流是什么？ 完整流程→人工审核→合并（默认）；仅生成场景→人工编写代码；或生成代码→人工迭代优化。
  5. 需要哪些领域知识？ 受监管行业（医疗、金融）的合规要求、领域不变量（金额不能为负、预约不能重叠）、过往事件中的已知风险区域。

Core Principles

核心原则

  1. Pipeline before code. Never generate test code before establishing what to test, why, and how to verify it. The seven-step pipeline exists to prevent premature code generation that targets the wrong things.
  2. Structured intermediates are the product. The assumptions document, coverage matrix, and oracle definitions are more valuable than the test code itself. They are reviewable, traceable, and reusable.
  3. Separate what from how. Scenario generation (what to test) and oracle design (how to verify) are distinct cognitive tasks. Mixing them produces scenarios biased toward what is easy to assert, with assertions tacked on as afterthoughts.
  4. AI generates the first draft; a human reviews and refines. Never ship AI-generated tests without human review. The AI accelerates — it does not replace judgment.
  5. Context is everything. Feed the LLM your conventions, existing patterns, selector strategy, and data setup. The more context, the less cleanup.
  6. Quality over quantity. Each test has a maintenance cost. Focus on critical paths, complex logic, and known risk areas — not test count.
  1. 先流程后代码。在确定测试内容、原因以及验证方式之前，绝不要生成测试代码。七步流程的存在是为了避免过早生成针对错误目标的代码。
  2. 结构化中间产物是核心产出。假设文档、覆盖矩阵和验证机制定义比测试代码本身更有价值。它们可审核、可追溯且可复用。
  3. 分离测试内容与验证方式。场景生成（测试什么）和验证机制设计（如何验证）是不同的认知任务。将两者混合会导致场景偏向于容易断言的内容，断言则沦为事后补充。
  4. AI生成初稿，人工审核优化。绝不要直接发布AI生成的测试。AI只是加速流程——无法替代人工判断。
  5. 上下文决定一切。向LLM提供你的规范、现有模式、选择器策略和数据设置。上下文越丰富，后续清理工作越少。
  6. 质量优先于数量。每个测试都有维护成本。专注于关键路径、复杂逻辑和已知风险区域——而非测试数量。

The Pipeline

执行流程

Mandatory workflow — agents MUST follow this order:
Step 1: Extract   → Requirements, entities, business rules from input
Step 2: Analyze   → Risks, invariants, edge cases, ambiguities
Step 3: Map       → Coverage matrix (requirement → scenario → priority)
Step 4: Generate  → Candidate scenarios (happy + boundary + negative + security + a11y)
Step 5: Design    → Assertions and oracles SEPARATELY from scenarios
Step 6: Code      → Test code (only after all above exist)
Step 7: Review    → Human review with traceability back to source
Full prompt templates for every step (extraction, risk analysis, scenario, oracle, code) live in 
references/prompt-patterns.md
. Below is the shape of each step's output.
强制工作流——代理必须遵循此顺序：
Step 1: 提取   → 从输入中提取需求、实体、业务规则
Step 2: 分析   → 风险、不变量、边缘场景、模糊点
Step 3: 映射   → 覆盖矩阵（需求→场景→优先级）
Step 4: 生成   → 候选场景（正向→边界→负向→安全→无障碍）
Step 5: 设计   → 单独于场景的断言和验证机制
Step 6: 编码   → 测试代码（仅在上述所有步骤完成后生成）
Step 7: 审核   → 人工审核，可追溯至原始输入
每个步骤（提取、风险分析、场景、验证机制、编码）的完整提示模板位于
references/prompt-patterns.md
中。以下是每个步骤输出的格式说明。

Step 1: Extract Requirements and Entities

Step 1: 提取需求与实体

Parse the input into structured elements: Entities (with roles/states/attributes), Business Rules (numbered), Explicit Requirements (
[REQ-N]
, stated in source), and Implicit Requirements (
[IMP-N]
, inferred — flag every one for human confirmation). Separating explicit from inferred is the rule that prevents testing assumptions as if they were specifications.
将输入解析为结构化元素：实体（包含角色/状态/属性）、业务规则（编号）、明确需求（
[REQ-N]
，来自输入原文）和隐含需求（
[IMP-N]
，推断得出——需标记所有隐含需求供人工确认）。区分明确需求与推断需求是防止将假设当作规格进行测试的关键规则。

Step 2: Risk Analysis and Invariants

Step 2: 风险分析与不变量

Derive what can go wrong, what must always be true, and where the source is silent.
  • Risks — table of 
    Risk | Likelihood | Impact | Source Requirement
    (e.g. race condition on stock decrement, email delay > 30s).
  • Invariants (must ALWAYS hold) — 
    stock >= 0
    , 
    order total = sum(items) + tax + shipping
    , 
    user sees only their own orders
    .
  • Ambiguities (need human answers) — "Does free shipping apply before or after discount codes?" Capture these explicitly; do not silently pick one.
  • Edge cases derived from risks — two users buy the last item, payment succeeds but email service is down.
推导可能出现的问题、必须始终成立的规则以及输入未明确说明的内容。
  • 风险 — 表格格式：
    风险 | 可能性 | 影响 | 来源需求
    （例如：库存递减时的竞态条件、邮件延迟超过30秒）。
  • 不变量（必须始终成立） — 
    库存 >= 0
    、
    订单总额 = 商品总和 + 税费 + 运费
    、
    用户仅能查看自己的订单
    。
  • 模糊点（需人工解答） — “免运费是在折扣码之前还是之后适用？” 需明确记录，不要自行选择。
  • 从风险推导的边缘场景 — 两位用户购买最后一件商品、支付成功但邮件服务故障。

Step 3: Coverage Matrix

Step 3: 覆盖矩阵

The single most important artifact — it prevents both gaps and duplicates. Map every requirement to scenarios with category, priority, and oracle type:
RequirementScenarioCategoryPriorityOracle Type
REQ-1Add single item to empty cartHappy pathP0State: cart count = 1
REQ-1Add out-of-stock itemNegativeP0UI: error message, cart unchanged
REQ-2Complete checkout with valid cardHappy pathP0State: order created, stock decremented
REQ-2Two users checkout last itemRace conditionP1One succeeds, one gets stock error
INV-1Stock never goes negativeInvariantP0Data: stock >= 0 after any operation
After building it, verify: every requirement has ≥1 happy and ≥1 negative scenario; every invariant has a direct test; every Step-2 risk has a scenario; no two rows test the same thing.
这是最重要的产物——它能防止测试遗漏和重复。将每个需求映射到场景，包含类别、优先级和验证机制类型：
需求场景类别优先级验证机制类型
REQ-1向空购物车添加单个商品正向路径P0状态：购物车数量 = 1
REQ-1添加缺货商品负向场景P0UI：显示错误消息，购物车无变化
REQ-2使用有效卡片完成结账正向路径P0状态：订单已创建，库存已递减
REQ-2两位用户结账最后一件商品竞态条件P1一位成功，一位收到库存不足错误
INV-1库存绝不会为负不变量P0数据：任何操作后库存 >= 0
构建完成后，验证：每个需求至少有1个正向和1个负向场景；每个不变量都有对应的测试；每个Step-2的风险都有对应的场景；没有两行测试相同的内容。

Step 4: Generate Candidate Scenarios

Step 4: 生成候选场景

For each matrix row, write the full scenario in Given/When/Then with explicit test-data requirements (
Given: user with 99 items in cart (max 100); When: adds one more; Then: count = 100
). Cover these categories systematically:
CategoryDescription
Happy pathThe user does exactly what the feature is designed for
BoundaryEdge of valid input ranges — use the BOUNDARIES framework (
references/prompt-patterns.md
)
NegativeInvalid inputs, unauthorized actions
SecurityAuth bypass, injection, privilege escalation
AccessibilityScreen reader, keyboard-only, contrast
State transitionValid and invalid moves between states
ConcurrencyTwo users acting simultaneously
针对矩阵中的每一行，用Given/When/Then格式编写完整场景，并明确测试数据要求（
Given: 购物车中有99件商品（上限100）；When: 再添加一件；Then: 数量 = 100
）。系统覆盖以下类别：
类别描述
正向路径用户完全按照功能设计进行操作
边界场景有效输入范围的边缘——使用BOUNDARIES框架（
references/prompt-patterns.md
）
负向场景无效输入、未授权操作
安全场景认证绕过、注入攻击、权限提升
无障碍场景屏幕阅读器、纯键盘操作、对比度
状态转换有效和无效的状态切换
并发场景两位用户同时操作

Step 5: Design Assertions and Oracles

Step 5: 设计断言与验证机制

Deliberately separate from Step 4. Scenarios describe behavior; oracles describe how to verify it. For each scenario, define oracles across categories — a single assertion is rarely enough to prove a behavior:
Oracle categoryAssertsExample
UI stateVisible text / element state
cart badge toHaveText('1')
DataPersisted state via API/DB
GET /api/cart
returns 1 item, correct total
NegativeWhat should NOT happenno error toast; no navigation away
Side effectAsync/external outcomesanalytics 
add_to_cart
fired; email in inbox < 30s
Oracle quality rules: assert business outcomes not implementation details; use the most specific assertion available (
toHaveText('$29.99')
, not 
toBeTruthy()
); include negative assertions; verify data integrity, not just UI; assert accessibility (focus management, live-region announcements).
务必与Step 4分离。场景描述行为；验证机制描述如何验证行为。针对每个场景，跨类别定义验证机制——单个断言通常不足以证明行为正确：
验证机制类别断言内容示例
UI状态可见文本/元素状态
cart badge toHaveText('1')
数据通过API/DB持久化的状态
GET /api/cart
返回1件商品，总价正确
负向验证不应发生的情况无错误提示框；未跳转页面
副作用异步/外部结果触发了
add_to_cart
分析事件；30秒内收到邮件
验证机制质量规则： 断言业务结果而非实现细节；使用最具体的断言（
toHaveText('$29.99')
，而非
toBeTruthy()
）；包含负向断言；验证数据完整性，而非仅UI；断言无障碍性（焦点管理、实时区域通知）。

Step 6: Generate Test Code

Step 6: 生成测试代码

Only after Steps 1-5 produce reviewed artifacts. Code is a mechanical translation of scenarios + oracles into framework syntax, with traceability comments linking back to the requirement and scenario:
typescript
/**
 * Scenario: SC-001 — Add single item to empty cart
 * Requirement: REQ-1 (User can add items to cart)
 * Priority: P0
 */
test('add single item to empty cart', async ({ page, testProduct }) => {
  await page.goto(`/products/${testProduct.id}`);                       // Given
  await page.getByRole('button', { name: 'Add to cart' }).click();      // When
  await expect(page.getByTestId('cart-badge')).toHaveText('1');         // Then
  await expect(page.getByTestId('error-toast')).not.toBeVisible();      // Negative oracle
});
Code generation rules: match project conventions (from 
qa-project-context.md
); reuse existing Page Objects, fixtures, and data factories; include traceability comments (
Scenario: SC-XXX
, 
Requirement: REQ-XX
); follow the project's selector strategy; put setup/teardown in fixtures, not inline.
仅在Step 1-5产出审核通过的产物后执行。代码是场景+验证机制到框架语法的机械转换，包含可追溯至需求和场景的注释：
typescript
/**
 * Scenario: SC-001 — Add single item to empty cart
 * Requirement: REQ-1 (User can add items to cart)
 * Priority: P0
 */
test('add single item to empty cart', async ({ page, testProduct }) => {
  await page.goto(`/products/${testProduct.id}`);                       // Given
  await page.getByRole('button', { name: 'Add to cart' }).click();      // When
  await expect(page.getByTestId('cart-badge')).toHaveText('1');         // Then
  await expect(page.getByTestId('error-toast')).not.toBeVisible();      // Negative oracle
});
代码生成规则： 匹配项目规范（来自
qa-project-context.md
）；复用现有页面对象、夹具和数据工厂；包含可追溯注释（
Scenario: SC-XXX
、
Requirement: REQ-XX
）；遵循项目的选择器策略；将设置/清理逻辑放在夹具中，而非内联编写。

Step 7: Human Review

Step 7: 人工审核

Not optional — a mandatory pipeline step. This reviews the tests this pipeline just generated, before they merge. (To audit a pre-existing suite you did not just generate, use 
ai-qa-review
instead.) Run every generated test against this checklist:
  • Traces to requirement: test → scenario → coverage row → requirement is followable.
  • Tests behavior, not implementation: survives a harmless refactor.
  • Correct abstraction level: right test type (unit vs integration vs E2E).
  • Test naming and readability: the test name states the behavior; a reader sees intent without decoding the body.
  • Test isolation / no shared state: the test creates and cleans up its own data, holds no order dependency on sibling tests, and passes when run alone or in any order.
  • Realistic test data: plausible, diverse, using 
    example.com
    .
  • Meaningful assertions: matches the oracle definition; specific, not 
    toBeTruthy()
    .
  • Matches project conventions: naming, structure, selector strategy.
  • No flakiness risks: no hardcoded timeouts, race conditions, or order dependence.
  • Edge cases included: goes beyond the happy path.
  • Assumptions validated: Step-2 ambiguities were resolved before coding.
Review outcome per test: KEEP (merge as-is) · MODIFY (fix listed issues, then merge) · REJECT (wrong requirement, wrong abstraction, hallucinated API) · DEFER (blocked on ambiguity).
非可选——这是流程中的强制步骤。审核本流程刚生成的测试，然后再合并。（若要审核非本次生成的现有测试套件，请使用
ai-qa-review
。）针对每个生成的测试，对照以下清单检查：
  • 可追溯至需求： 测试→场景→覆盖矩阵行→需求的链路清晰可查。
  • 测试行为而非实现： 在无破坏性重构时仍能通过。
  • 正确的抽象级别： 测试类型正确（单元测试vs集成测试vs端到端测试）。
  • 测试命名与可读性： 测试名称明确说明行为；读者无需解读代码主体即可理解意图。
  • 测试隔离/无共享状态： 测试创建并清理自己的数据，与其他测试无顺序依赖，单独运行或任意顺序运行均可通过。
  • 真实的测试数据： 合理、多样，使用
    example.com
    相关数据。
  • 有意义的断言： 匹配验证机制定义；具体明确，而非
    toBeTruthy()
    。
  • 符合项目规范： 命名、结构、选择器策略均匹配。
  • 无不稳定风险： 无硬编码超时、竞态条件或顺序依赖。
  • 包含边缘场景： 不止覆盖正向路径。
  • 假设已验证： Step-2中的模糊点已在编码前解决。
每个测试的审核结果： 保留（直接合并）· 修改（修复列出的问题后合并）· 拒绝（需求错误、抽象错误、幻觉API）· 推迟（因模糊点受阻）。

Guardrails

防护规则

Hard rules. Agents MUST follow them.
  • Code before coverage is forbidden. Never emit test code before Steps 1-3 (requirements, risk analysis with documented assumptions, coverage matrix) exist. If an agent skips to code: STOP, go back.
  • Assert outcomes, not implementation. 
    expect(screen.getByRole('progressbar')).toBeVisible()
    , not 
    expect(component.state.isLoading).toBe(true)
    . 
    expect(page.getByTestId('cart-badge')).toHaveText('1')
    , not 
    expect(store.dispatch).toHaveBeenCalledWith(...)
    .
  • Scenarios (Step 4) before oracles (Step 5), always. Scenario = WHAT happens; oracle = HOW to verify. Mixing them biases scenarios toward easy assertions.
  • Always produce the intermediates — assumptions document, uncovered ambiguities, oracle candidates, and the traceability chain — even in abbreviated form.
Flag these when detected:
  • Hallucinated APIs — endpoints, selectors, methods, or imports that do not exist in the codebase. Verify mechanically (see Verification) before human review.
  • Duplicate scenarios — same behavior, trivially different data. Consolidate or parametrize.
  • Low-value assertions — 
    expect(response).toBeTruthy()
    , 
    expect(page).toHaveURL(/.*/)
    .
  • Missing negative cases — if every scenario is a happy path, the coverage matrix is incomplete.
  • Unrealistic test data — 
    test@test.com
    , 
    John Doe
    , 
    password123
    . Use diverse, plausible data on 
    example.com
    .
硬性规则。代理必须遵守。
  • 禁止先编码后覆盖。在Step 1-3（需求、带文档化假设的风险分析、覆盖矩阵）完成之前，绝不能生成测试代码。如果代理跳过直接编码：停止，返回重新执行。
  • 断言结果而非实现。
    expect(screen.getByRole('progressbar')).toBeVisible()
    ，而非
    expect(component.state.isLoading).toBe(true)
    。
    expect(page.getByTestId('cart-badge')).toHaveText('1')
    ，而非
    expect(store.dispatch).toHaveBeenCalledWith(...)
    。
  • 始终先场景（Step4）后验证机制（Step5）。场景=发生了什么；验证机制=如何验证。混合两者会导致场景偏向于容易断言的内容。
  • 始终生成中间产物——假设文档、未解决的模糊点、候选验证机制以及可追溯链路，即使是简化形式也需生成。
检测到以下情况时需标记：
  • 幻觉API——代码库中不存在的端点、选择器、方法或导入。在人工审核前需通过机械方式验证（见验证环节）。
  • 重复场景——行为相同，仅数据略有不同。需合并或参数化。
  • 低价值断言——
    expect(response).toBeTruthy()
    、
    expect(page).toHaveURL(/.*/)
    。
  • 缺少负向场景——如果所有场景都是正向路径，则覆盖矩阵不完整。
  • 不真实的测试数据——
    test@test.com
    、
    John Doe
    、
    password123
    。使用
    example.com
    上多样、合理的数据。

Model selection per step

各步骤模型选择

Route by difficulty, not habit. Use a cheap model for mechanical extraction (Step 1) and the coverage-matrix bookkeeping (Step 3) — Haiku 4.5 or Sonnet 4.6 are plenty. Escalate to Opus 4.8 for oracle design (Step 5) and hallucination-sensitive code generation (Step 6), where a wrong inference is expensive; reach for Fable 5 only on genuinely hard reasoning (subtle invariants, regulated-domain logic). Running the strongest model on every step is wasteful; running the cheapest on Step 6 produces fabricated APIs.
根据难度选择模型，而非习惯。机械提取（Step1）和覆盖矩阵记录（Step3）使用低成本模型即可——Haiku 4.5或Sonnet 4.6足够。验证机制设计（Step5）和对幻觉敏感的代码生成（Step6）需升级到Opus 4.8，错误推断的代价很高；仅在真正复杂的推理（微妙的不变量、受监管领域逻辑）时才使用Fable 5。每个步骤都用最强模型是浪费；在Step6使用最便宜模型会生成伪造API。

Verification

验证环节

Convert the "hallucinated APIs" warning into a mechanical gate. After Step 6, before human review:
  1. Resolve imports / types. TypeScript: 
    npx tsc --noEmit
    — fabricated imports and wrong signatures fail here. Python: 
    python -m pyflakes <files>
    or 
    ruff check
    .
  2. Grep generated selectors/endpoints against the codebase. Confirm every 
    getByTestId('...')
    id and every API path the test calls actually exists in source:
    bash
    grep -roE "getByTestId\('([^']+)'\)" generated/ | sed -E "s/.*'([^']+)'.*/\1/" | sort -u \
  | while read id; do grep -rq "$id" src/ || echo "MISSING testid: $id"; done
  3. Run the suite once. Tests that reference nonexistent routes/selectors fail fast; quarantine those before review rather than reviewing dead code.
Any 
MISSING
line or 
tsc
error is a hallucination to fix before a human spends review time.
将“幻觉API”警告转化为机械校验关卡。在Step6之后、人工审核之前：
  1. 解析导入/类型。TypeScript：
    npx tsc --noEmit
    ——伪造的导入和错误的签名会在此处失败。Python：
    python -m pyflakes <files>
    或
    ruff check
    。
  2. 在代码库中搜索生成的选择器/端点。确认测试中调用的每个
    getByTestId('...')
    和API路径确实存在于源代码中：
    bash
    grep -roE "getByTestId\('([^']+)'\)" generated/ | sed -E "s/.*'([^']+)'.*/\1/" | sort -u \
  | while read id; do grep -rq "$id" src/ || echo "MISSING testid: $id"; done
  3. 运行一次测试套件。引用不存在的路由/选择器的测试会快速失败；在审核前隔离这些测试，而非审核无效代码。
任何
MISSING
行或
tsc
错误都是需要在人工审核前修复的幻觉问题。

Anti-Patterns

反模式

  1. Skipping to code. The most common failure: an agent gets a PRD and immediately writes tests. Without the coverage matrix it misses scenarios and duplicates others. The pipeline exists to prevent this.
  2. Asserting implementation instead of behavior. 
    expect(component.state.isLoading).toBe(true)
    breaks on any refactor. Assert 
    expect(screen.getByRole('progressbar')).toBeVisible()
    — what the user observes.
  3. Mixing scenarios and assertions. Writing "test this thing and check this value" as one step. Separate what to test from how to verify it.
  4. No project context in the prompt. Without conventions and existing patterns, you get generic tests. 
    qa-project-context.md
    exists for exactly this.
  5. Over-generating. AI will write 50 tests for a simple function. Each carries maintenance cost. Use the coverage matrix to bound generation to meaningful scenarios.
  6. Copy-paste without understanding. If you cannot explain what a generated test does and why, do not merge it. Tests you do not understand become tests you cannot debug.
  7. Shipping without review. Step 7 is not optional. AI tests routinely contain hallucinated APIs, wrong selectors, incorrect business logic, and flakiness only human review catches.
  8. Ignoring the feedback loop. When AI tests catch real bugs, note the prompt patterns that worked; when they false-positive, note what went wrong. Build a project-specific library of what works.
  1. 直接跳转到编码。最常见的错误：代理拿到PRD后立即编写测试。没有覆盖矩阵会导致场景遗漏和重复。流程的存在就是为了防止这种情况。
  2. 断言实现而非行为。
    expect(component.state.isLoading).toBe(true)
    会在任何重构时失效。应断言
    expect(screen.getByRole('progressbar')).toBeVisible()
    ——用户能观察到的内容。
  3. 混合场景与断言。将“测试这个内容并检查这个值”写成一步。需将测试什么与如何验证分离。
  4. 提示中不包含项目上下文。没有规范和现有模式，会得到通用测试。
    qa-project-context.md
    正是为此而生。
  5. 过度生成。AI会为简单函数编写50个测试。每个测试都有维护成本。使用覆盖矩阵限制生成有意义的场景。
  6. 复制粘贴而不理解。如果你无法解释生成的测试做了什么以及为什么，不要合并。你不理解的测试会成为无法调试的测试。
  7. 未审核就发布。Step7不是可选步骤。AI生成的测试通常包含幻觉API、错误选择器、不正确的业务逻辑以及只有人工审核才能发现的不稳定问题。
  8. 忽略反馈循环。当AI测试发现真实Bug时，记录有效的提示模式；当出现误报时，记录问题所在。构建项目特定的有效模式库。

Done When

完成标准

  • All seven artifacts exist: requirements document, risk & invariants, coverage matrix, scenario set, oracle definitions, test code, and review notes with a KEEP/MODIFY/REJECT/DEFER decision per test.
  • The coverage matrix was produced and reviewed before any test code file was written.
  • Verification passed: 
    tsc --noEmit
    (or language equivalent) exits 0 and the selector/endpoint grep reports zero 
    MISSING
    lines.
  • Each generated test has a recorded human review decision; no test is marked KEEP without one.
  • The suite's CI job exits 0 (green).
  • Reproducibility metadata recorded: the exact model ID (e.g. 
    claude-opus-4-8
    , 
    claude-sonnet-4-6
    , 
    claude-haiku-4-5-20251001
    ), input source hash, and the version of any skill / CLI / MCP server invoked.
  • 存在所有七个产物：需求文档、风险与不变量、覆盖矩阵、场景集、验证机制定义、测试代码以及每个测试带有保留/修改/拒绝/推迟决策的审核记录。
  • 在编写任何测试代码文件之前，已生成并审核覆盖矩阵。
  • 验证通过：
    tsc --noEmit
    （或对应语言的等效命令）返回0，选择器/端点搜索报告无
    MISSING
    行。
  • 每个生成的测试都有记录在案的人工审核决策；没有测试在无决策的情况下被标记为保留。
  • 测试套件的CI任务返回0（绿色）。
  • 记录可复现性元数据：精确的模型ID（例如
    claude-opus-4-8
    、
    claude-sonnet-4-6
    、
    claude-haiku-4-5-20251001
    ）、输入源哈希以及调用的任何技能/CLI/MCP服务器的版本。

Related Skills

相关技能

  • qa-project-context — Set up the context file that makes AI test generation dramatically better. Configure this first.
  • playwright-automation — Deep Playwright patterns (POM, fixtures, CI), plus the Test Agents (
    init-agents --loop=claude
    , scaffolded into 
    .claude/agents/
    ) and 
    @playwright/mcp
    modes chosen in Discovery Q2. Generated tests live inside this framework.
  • unit-testing — Jest, Vitest, pytest patterns for unit-level generated tests.
  • api-testing — Endpoint test patterns for tests generated from OpenAPI specs.
  • test-strategy — Decide what to test and at which level before generating.
  • test-reliability — Make generated tests reliable: flake classification, healing, video receipts.
  • ai-system-testing — When the input is an LLM feature spec, generate eval datasets here (Promptfoo, DeepEval, Ragas, Braintrust) instead of Playwright specs.
  • ai-qa-review — Audit a pre-existing suite you did not just generate (test smells, testability). Step 7 here only reviews this pipeline's own output.
  • ai-bug-triage — When generated tests find bugs, classify and report them through the triage pipeline.
  • qa-project-context — 设置上下文文件，大幅提升AI测试生成质量。请先配置此技能。
  • playwright-automation — 深入的Playwright模式（POM、夹具、CI），以及发现环节问题2中选择的Test Agents（
    init-agents --loop=claude
    ，生成到
    .claude/agents/
    ）和
    @playwright/mcp
    模式。生成的测试在此框架内运行。
  • unit-testing — 针对单元级生成测试的Jest、Vitest、pytest模式。
  • api-testing — 针对从OpenAPI规格生成的测试的端点测试模式。
  • test-strategy — 在生成测试前决定测试什么以及在哪个层级测试。
  • test-reliability — 提升生成测试的可靠性：不稳定测试分类、修复、视频记录。
  • ai-system-testing — 当输入是LLM功能规格文档时，请在此生成评估数据集（Promptfoo、DeepEval、Ragas、Braintrust），而非Playwright规格。
  • ai-qa-review — 审核非本次生成的现有测试套件（测试异味、可测试性）。本流程的Step7仅审核本流程的输出。
  • ai-bug-triage — 当生成的测试发现Bug时，通过分类流程进行分类和报告。

Reference Files (in 
references/
)

参考文件（位于
references/
）

  • prompt-patterns.md — Full prompt library aligned to the seven steps: extraction, risk analysis, scenario generation, oracle design, and code generation prompts, plus the BOUNDARIES edge-case framework used in Step 4.
  • prompt-patterns.md — 与七步流程对齐的完整提示库：提取、风险分析、场景生成、验证机制设计和代码生成提示，以及Step4中使用的BOUNDARIES边缘场景框架。