test-strategy
Compare original and translation side by side
🇺🇸
Original
English🇨🇳
Translation
Chinese<objective>
Generate an actionable QA strategy tailored to the product, team, and risk profile — a document that drives daily testing decisions, not a compliance artifact that collects dust. A team with 150 E2E tests and a 52-minute pipeline thinks it has good coverage; this skill diagnoses the inverted pyramid, prescribes the rebalance, and ties every element to a measurable KPI.
</objective>
<objective>
生成适配产品、团队和风险状况的可落地QA策略——一份能指导日常测试决策的文档，而非用于合规的闲置文件。某团队拥有150个E2E测试和52分钟的流水线，自认为覆盖度良好；本技能可诊断其倒置的测试金字塔问题，给出重新平衡的方案，并将每个环节与可量化的KPI挂钩。
</objective>
Discovery Questions
调研问题
Before writing a single line of strategy, gather context. Check first — if it exists, use it as the foundation and skip questions already answered there.
.agents/qa-project-context.md在撰写策略前，先收集上下文信息。首先查看——若该文件存在，以此为基础，跳过已解答的问题。
.agents/qa-project-context.mdProduct & Business Context
产品与业务背景
- What is the product? (SaaS, e-commerce, API platform, mobile app, content site)
- Who are the users? (consumers, enterprise, internal, developers)
- What are the business-critical flows? (signup, checkout, payment, data export)
- What is the release cadence? (continuous, weekly, bi-weekly, quarterly)
- What compliance requirements exist? (SOC2, HIPAA, PCI-DSS, GDPR, EU AI Act)
- 产品类型是什么？（SaaS、电商、API平台、移动应用、内容站点）
- 用户群体有哪些？（消费者、企业客户、内部员工、开发者）
- 哪些是业务核心流程？（注册、结账、支付、数据导出）
- 发布节奏如何？（持续发布、每周、每两周、每季度）
- 存在哪些合规要求？（SOC2、HIPAA、PCI-DSS、GDPR、欧盟AI法案）
Current Testing State
当前测试状态
- What test levels exist today, and the current count at each level?
- What frameworks and tools are in use?
- Current code coverage, and the target if any?
- How long does the CI pipeline take end-to-end?
- What is the current flakiness rate?
- 当前存在哪些测试级别，每个级别下的测试数量是多少？
- 使用了哪些框架和工具？
- 当前代码覆盖率是多少，是否有目标覆盖率？
- CI流水线端到端耗时多久？
- 当前测试不稳定率是多少？
Pain Points & Goals
痛点与目标
- Biggest quality pain points? (regressions, slow feedback, flaky tests, gaps)
- What broke in the last 3 releases? What escaped to production?
- What does "good enough quality" look like for this team?
- Appetite for investment in test infrastructure?
- 最大的质量痛点是什么？（回归问题、反馈缓慢、测试不稳定、覆盖缺口）
- 最近3次发布中出现了哪些问题？哪些问题漏到了生产环境？
- 对该团队而言，「足够好的质量」是什么标准？
- 对测试基础设施的投入意愿如何？
Team & Constraints
团队与约束条件
- Team size and composition (devs, QA, SDET, manual testers)
- Skill levels with automation tools
- Budget constraints for tooling
- Timeline pressure — is there a deadline driving this strategy?
Calibrate to team maturity (setinteam_maturity):.agents/qa-project-context.md
- startup — Minimal pyramid: unit tests + a handful of critical E2E paths. Skip contract testing and formal metrics until CI runs reliably. Phase 1 under 4 weeks.
- growing — Full pyramid with defined coverage targets, flakiness thresholds, and CI quality gates. Add risk-based prioritization.
- established — SLA-backed quality gates, multi-environment coverage, advanced tooling (contract testing, chaos, observability), and formal review cadence.
- 团队规模与人员构成（开发、QA、SDET、手动测试人员）
- 团队对自动化工具的掌握水平
- 工具预算限制
- 时间压力——是否有截止日期驱动本次策略制定？
根据团队成熟度调整（在中设置.agents/qa-project-context.md）：team_maturity
- 初创团队——极简金字塔：单元测试+少量核心E2E路径。在CI稳定运行前，跳过契约测试和正式指标。第一阶段周期不超过4周。
- 成长中团队——完整测试金字塔，定义覆盖目标、不稳定阈值和CI质量门禁。加入基于风险的优先级划分。
- 成熟团队——基于SLA的质量门禁、多环境覆盖、高级工具（契约测试、混沌工程、可观测性），以及正式的评审节奏。
Core Principles
核心原则
-
Risk-based prioritization over exhaustive coverage. Not all code is equal — a payment bug costs 1000x more than a tooltip typo. Allocate testing effort proportional to business risk, not code volume. The risk matrix drives where to invest; runfirst if no matrix exists yet.
risk-based-testing -
Test pyramid health is the leading indicator. A healthy suite is many fast unit tests, fewer integration, fewest E2E. When the shape inverts (ice cream cone) feedback is slow, maintenance is high, and confidence is paradoxically low. Diagnose the current shape before prescribing anything.
-
Shift-left: catch defects earlier. Every defect found later costs exponentially more. Push validation earlier — static analysis before tests, unit before integration, contract before E2E. Design reviews catch architecture bugs no test can find.
-
Every strategy element has a KPI. If you cannot measure it, you cannot improve it. Coverage targets, flakiness thresholds, escape-rate goals, MTTR limits — each section names a number and a tracking cadence.
-
Living document, not a shelf document. Reviewed quarterly at minimum. It carries a revision history, a named owner per section, and explicit re-evaluation triggers (new product area, team change, major incident, defect escape).
-
基于风险的优先级高于全面覆盖。并非所有代码都同等重要——支付功能的bug造成的损失是 tooltip 拼写错误的1000倍。根据业务风险分配测试资源，而非代码量。风险矩阵决定投入方向；若尚未建立风险矩阵，请先运行。
risk-based-testing -
测试金字塔健康度是首要指标。健康的测试套件包含大量快速单元测试、少量集成测试、极少E2E测试。当金字塔倒置（甜筒型）时，反馈缓慢、维护成本高，且测试信心反而低下。在给出方案前，先诊断当前的金字塔形态。
-
左移测试：更早发现缺陷。缺陷发现得越晚，修复成本呈指数级增长。将验证环节前置——测试前先做静态分析，单元测试早于集成测试，契约测试早于E2E测试。设计评审能发现任何测试都无法覆盖的架构缺陷。
-
策略的每个环节都对应KPI。无法衡量的内容，就无法改进。覆盖目标、不稳定阈值、缺陷逃逸率目标、MTTR限制——每个部分都明确数值和跟踪频率。
-
活文档，而非闲置文档。至少每季度评审一次。包含修订历史、各部分的指定负责人，以及明确的重新评估触发条件（新业务领域、团队变动、重大事故、缺陷逃逸）。
Strategy Document Template
策略文档模板
Walk through each section to produce the final document. Tailor depth to complexity — a 5-person startup needs 5 pages, not 50. The final document follows a 13-section structure (Executive Summary through Revision History); see for the copy-paste markdown skeleton, and for four fully worked examples (SaaS, e-commerce, API-first, media).
references/diagrams-and-worksheets.mdreferences/strategy-templates.md逐一完成以下部分，生成最终文档。根据复杂度调整内容深度——5人初创团队只需5页文档，而非50页。最终文档包含13个部分（从执行摘要到修订历史）；可查看获取可复制的Markdown骨架，查看获取4份完整示例（SaaS、电商、API优先、媒体）。
references/diagrams-and-worksheets.mdreferences/strategy-templates.md1. Scope & Objectives
1. 范围与目标
Define boundaries clearly. Ambiguity here causes gaps and wasted effort downstream.
- In scope: every product area, service, and integration this strategy covers; functional and non-functional types; platforms and browsers/devices.
- Out of scope: state what is NOT covered and why — third-party services tested only at the contract level, legacy systems slated for deprecation.
- Objectives: 3-5 measurable objectives with timelines, e.g. "Reduce defect escape rate from 12% to under 5% within two quarters," "Achieve 80% unit coverage on all services launched after Q1 2026."
明确界定边界。此处的模糊会导致后续出现覆盖缺口和资源浪费。
- 纳入范围：本策略覆盖的所有业务领域、服务和集成；功能与非功能测试类型；涉及的平台和浏览器/设备。
- 排除范围：说明未覆盖的内容及原因——仅在契约层测试的第三方服务、计划淘汰的遗留系统。
- 目标：3-5个可衡量的带时间线目标，例如「两个季度内将缺陷逃逸率从12%降至5%以下」、「2026年第一季度后上线的所有服务，单元覆盖率达到80%」。
2. Test Levels & Types
2. 测试级别与类型
Define each level, what it covers, who owns it, and expected volume.
| Level | What It Validates | Owner | Framework | Target Count | Run Frequency |
|---|---|---|---|---|---|
| Unit | Functions, business logic, edge cases | Developers | Vitest/Jest/pytest | 70-80% of all tests | Every commit |
| Integration | Service interactions, DB queries, API contracts | Developers + QA | Supertest/pytest + Testcontainers | 15-20% of all tests | Every PR |
| E2E | Critical user journeys through the full stack | QA/SDET | Playwright/Cypress | 5-10% of all tests | Pre-deploy + nightly |
| API | Contract compliance, schemas, error handling | Developers | Playwright APIRequestContext/Schemathesis | Per endpoint | Every PR |
| Visual | UI regression, layout shifts, responsive | QA | Playwright/Argos/Chromatic | Key pages | Nightly |
| Performance | Response times, throughput, resource usage | DevOps/QA | k6/Lighthouse | Critical paths | Weekly + pre-release |
| Security | OWASP Top 10, dep vulns, auth flows | Security/DevOps | OWASP ZAP/Snyk | Per release | Pre-release + scheduled |
| Accessibility | WCAG 2.2 AA, screen reader compat | QA/Frontend | axe-core | Key flows | Every PR |
Adjust to what the product actually needs. Not every product needs visual regression. Every product needs unit and integration tests.
定义每个测试级别、覆盖范围、负责人及预期数量。
| 级别 | 验证内容 | 负责人 | 框架 | 目标数量占比 | 运行频率 |
|---|---|---|---|---|---|
| 单元测试 | 函数、业务逻辑、边缘案例 | 开发人员 | Vitest/Jest/pytest | 70-80% | 每次提交 |
| 集成测试 | 服务交互、数据库查询、API契约 | 开发人员 + QA | Supertest/pytest + Testcontainers | 15-20% | 每个PR |
| E2E测试 | 核心用户全链路流程 | QA/SDET | Playwright/Cypress | 5-10% | 预部署 + nightly |
| API测试 | 契约合规性、Schema、错误处理 | 开发人员 | Playwright APIRequestContext/Schemathesis | 每个端点 | 每个PR |
| 视觉测试 | UI回归、布局偏移、响应式适配 | QA | Playwright/Argos/Chromatic | 核心页面 | nightly |
| 性能测试 | 响应时间、吞吐量、资源占用 | DevOps/QA | k6/Lighthouse | 核心路径 | 每周 + 预发布 |
| 安全测试 | OWASP Top 10、依赖漏洞、认证流程 | 安全团队/DevOps | OWASP ZAP/Snyk | 每次发布 | 预发布 + 定期 |
| 无障碍测试 | WCAG 2.2 AA、屏幕阅读器兼容性 | QA/前端团队 | axe-core | 核心流程 | 每个PR |
根据产品实际需求调整。并非所有产品都需要视觉回归测试，但所有产品都需要单元和集成测试。
3. Test Pyramid Analysis
3. 测试金字塔分析
Diagnose the current shape, then define the target.
Shapes. The suite takes one of four shapes — healthy pyramid (many unit, few E2E), ice cream cone (inverted, E2E-heavy), diamond (integration-heavy), or hourglass (unit-heavy and E2E-sparse with a missing integration middle). Each signals a different feedback/maintenance trade-off. See for the side-by-side ASCII diagram.
references/diagrams-and-worksheets.mdCurrent state. Count tests at each level, compute the percentage split, identify the shape, then capture CI duration, flaky rate, and pass rate. See the Current State Assessment Worksheet in the reference file.
Target state. Define target ratios (70-80% unit, 15-20% integration, 5-10% E2E) with concrete counts, plus target CI duration and flaky rate. See the Target State Worksheet.
Action plan — if ice cream cone or diamond:
- Freeze E2E growth — no new E2E tests unless covering a net-new critical path.
- Decompose existing E2E — find E2E tests validating logic testable at unit level (a checkout test asserting tax math becomes a unit test on the tax function), rewrite them down a level.
- Add unit requirements to the PR checklist — every PR touching business logic ships unit tests.
- Set CI gates — fail PRs where the unit:E2E ratio drops below threshold.
Before rebalancing, separate genuinely flaky E2E tests from ones exposing real bugs — quarantining a flaky test that hides a race condition is how the regression escapes. For flake root-cause triage and quarantine mechanics, see .
test-reliabilityAction plan — if hourglass:
- Invest in integration infrastructure — DB fixtures, service stubs, contract tests.
- Identify service boundaries — each boundary needs integration tests for happy path + error cases.
- Use contract testing (Pact) for inter-service communication.
诊断当前形态，定义目标形态。
形态类型：测试套件分为四种形态——健康金字塔（大量单元测试，少量E2E）、甜筒型（倒置，E2E占比过高）、钻石型（集成测试占比过高）、沙漏型（单元测试占比高但E2E缺失，中间集成层薄弱）。每种形态对应不同的反馈/维护权衡。可查看中的并列ASCII示意图。
references/diagrams-and-worksheets.md当前状态：统计每个级别的测试数量，计算占比，识别形态，记录CI耗时、不稳定率和通过率。可参考参考文件中的「当前状态评估工作表」。
目标状态：定义目标比例（70-80%单元测试、15-20%集成测试、5-10%E2E测试）及具体数量，同时设定目标CI耗时和不稳定率。可参考「目标状态工作表」。
调整方案——若为甜筒型或钻石型：
- 冻结E2E测试增长——除非覆盖全新核心路径，否则不再新增E2E测试。
- 拆分现有E2E测试——找出可在单元层验证逻辑的E2E测试（例如验证税费计算的结账测试，可改为针对税费函数的单元测试），将其改写为更低级别的测试。
- 在PR检查清单中添加单元测试要求——所有涉及业务逻辑的PR必须附带单元测试。
- 设置CI门禁——若单元测试与E2E测试比例低于阈值，PR不通过。
在重新平衡前，需区分真正不稳定的E2E测试和暴露真实bug的测试——隔离隐藏竞态条件的不稳定测试会导致回归问题漏到生产环境。关于不稳定测试的根因分析和隔离机制，可查看。
test-reliability调整方案——若为沙漏型：
- 投入集成测试基础设施——数据库 fixtures、服务桩、契约测试。
- 识别服务边界——每个边界需针对正常流程和异常场景编写集成测试。
- 使用契约测试（Pact）验证服务间通信。
4. Risk Assessment Matrix
4. 风险评估矩阵
Map features to risk levels — this directly determines testing depth. Score each feature as Impact (1 Negligible → 5 Catastrophic) × Likelihood (1 Rare → 5 Almost Certain); the product (1-25) maps to LOW/MED/HIGH/CRIT bands. See for the full 5x5 matrix with every cell labeled.
references/diagrams-and-worksheets.md| Risk Level | Testing Action | Automation | Monitoring |
|---|---|---|---|
| CRITICAL (15-25) | Full automation + manual exploratory + load test | Mandatory, every commit | Real-time alerts, synthetic monitoring |
| HIGH (10-14) | Full automation + periodic manual review | Mandatory, every PR | Dashboard + daily checks |
| MEDIUM (5-9) | Automation for happy path + key error cases | Recommended | Weekly review |
| LOW (1-4) | Manual testing or skip | Optional | None required |
Example mapping:
| Feature Area | Impact | Likelihood | Score | Testing Approach |
|---|---|---|---|---|
| Payment processing | 5 - Catastrophic | 3 - Possible | 15 - CRIT | Automated E2E + unit + contract + monitoring |
| User authentication | 5 - Catastrophic | 2 - Unlikely | 10 - HIGH | Automated E2E + security scan + unit |
| Product search | 3 - Moderate | 3 - Possible | 9 - MED | Unit + integration + happy-path E2E |
| Dashboard rendering | 2 - Minor | 3 - Possible | 6 - MED | Unit + visual regression |
| Email preferences | 1 - Negligible | 2 - Unlikely | 2 - LOW | Manual verification |
将功能映射到风险级别——直接决定测试深度。每个功能按影响程度（1 可忽略 → 5 灾难性）× 发生概率（1 罕见 → 5 几乎必然）评分；总分（1-25）对应LOW/MED/HIGH/CRIT四个风险等级。可查看中的完整5×5矩阵及每个单元格的说明。
references/diagrams-and-worksheets.md| 风险等级 | 测试动作 | 自动化要求 | 监控要求 |
|---|---|---|---|
| CRITICAL（15-25） | 全自动化测试 + 手动探索性测试 + 负载测试 | 强制，每次提交 | 实时告警、 synthetic监控 |
| HIGH（10-14） | 全自动化测试 + 定期手动评审 | 强制，每个PR | 仪表盘 + 每日检查 |
| MEDIUM（5-9） | 正常流程及关键异常场景自动化测试 | 推荐 | 每周评审 |
| LOW（1-4） | 手动测试或跳过 | 可选 | 无需监控 |
示例映射：
| 功能区域 | 影响程度 | 发生概率 | 总分 | 测试方法 |
|---|---|---|---|---|
| 支付处理 | 5 - 灾难性 | 3 - 可能 | 15 - CRIT | 自动化E2E + 单元 + 契约测试 + 监控 |
| 用户认证 | 5 - 灾难性 | 2 - 不太可能 | 10 - HIGH | 自动化E2E + 安全扫描 + 单元测试 |
| 产品搜索 | 3 - 中等 | 3 - 可能 | 9 - MED | 单元 + 集成 + 正常流程E2E测试 |
| 仪表盘渲染 | 2 - 轻微 | 3 - 可能 | 6 - MED | 单元 + 视觉回归测试 |
| 邮件偏好设置 | 1 - 可忽略 | 2 - 不太可能 | 2 - LOW | 手动验证 |
5. Environment Strategy
5. 环境策略
| Environment | Purpose | Test Types | Data | Deploy Trigger |
|---|---|---|---|---|
| Local | Developer feedback | Unit, integration | Mocked/seeded | On save |
| CI | Automated validation | Unit, integration, lint, SAST | Ephemeral | On push/PR |
| Staging | Pre-production validation | E2E, visual, performance, security | Production-like (anonymized) | On merge to main |
| Production | Monitoring & smoke | Smoke tests, synthetic monitoring | Live | On deploy |
Document: how test data is managed per environment, whether environments are ephemeral (preview deployments) or long-lived, who has access, and how environment-specific config is managed.
| 环境 | 用途 | 测试类型 | 数据 | 部署触发条件 |
|---|---|---|---|---|
| 本地环境 | 开发人员快速反馈 | 单元、集成测试 | 模拟/预填充数据 | 保存代码时 |
| CI环境 | 自动化验证 | 单元、集成、代码检查、SAST | 临时数据 | 推送/PR时 |
| 预发布环境 | 生产前验证 | E2E、视觉、性能、安全测试 | 类生产数据（匿名化） | 合并到主分支时 |
| 生产环境 | 监控与冒烟测试 | 冒烟测试、synthetic监控 | 真实数据 | 部署时 |
需记录：每个环境的测试数据管理方式，环境是临时的（预览部署）还是长期存在的，访问权限，以及环境特定配置的管理方式。
6. Tool Selection Rationale
6. 工具选择依据
Do not pick tools first. Understand needs, then select tools that fit. Score each candidate against weighted criteria.
| Criteria (weight) | Tool A | Tool B | Tool C |
|---|---|---|---|
| Fits tech stack (25%) | |||
| Team familiarity (20%) | |||
| Community & docs (15%) | |||
| CI integration (15%) | |||
| Maintenance cost (10%) | |||
| Speed of execution (10%) | |||
| License cost (5%) | |||
| Weighted total |
Score each 1-5, multiply by weight, sum for the weighted total. Beyond license fees, account for total cost of ownership: setup time (configure CI, write first tests, train team), writing time (time 5 real tests to measure), maintenance time (how often tests break on framework updates), debug time (good error messages cut this), and infrastructure cost (browser farms, parallel runners).
Common stack starting points — document why you chose or deviated:
| Product Type | Unit | Integration | E2E | API | Visual |
|---|---|---|---|---|---|
| React SaaS | Vitest | Testing Library + MSW | Playwright | Supertest | Playwright screenshots |
| Next.js | Vitest | Testing Library + MSW | Playwright | Supertest | Playwright screenshots |
| Python API | pytest | pytest + Testcontainers | pytest + requests | Schemathesis | N/A |
| Mobile (RN) | Jest | Testing Library + MSW | Detox / Maestro / Appium 3.x | Supertest | Appium screenshots |
| Vue SaaS | Vitest | Testing Library + MSW | Playwright | Supertest | Playwright screenshots |
| AI/LLM features | Vitest | DeepEval | Playwright + Promptfoo evals | Promptfoo / Ragas | N/A |
For AI/LLM features, add explicit risk testing for hallucinations, bias, prompt injection, and privacy — see and (EU AI Act).
ai-system-testingcompliance-testingReference frameworks:
- CTAL-AT v2.0 (ISTQB, Advanced Agile Tester, 2026) — a new Advanced-level certification requiring CTFL v4.0, not an update of an Advanced predecessor; it supersedes the retired Foundation-level CTFL-AT Agile extension (already absorbed into CTFL v4.0). Covers test strategy and approach, whole-team approach, shift-left, end-to-end testing, test smells, exploratory + AI-assisted testing.
- CT-GenAI v1.1 (ISTQB, released 2026-04-27) — formalizes LLM-powered test infrastructure as a discipline; defines AI-specific risk classes (hallucinations, reasoning errors, bias, privacy, AI regulations).
- CTFL v4.0 (ISTQB) — foundational vocabulary; useful when aligning teams from different testing traditions.
- HTSM v6.3 (Bach) — Heuristic Test Strategy Model; emphasizes state-based testing and boundary heuristics. Lightweight alternative to ISTQB framing.
- World Quality Report 2025-26 (Capgemini, 17th edition) — benchmark data: 43% of orgs experimenting with Gen AI in QA, 15% scaled. Useful for placing your AI-adoption stage during planning.
不要先选工具。先明确需求，再选择适配的工具。按加权标准对每个候选工具评分。
| 标准（权重） | 工具A | 工具B | 工具C |
|---|---|---|---|
| 适配技术栈（25%） | |||
| 团队熟悉度（20%） | |||
| 社区与文档（15%） | |||
| CI集成能力（15%） | |||
| 维护成本（10%） | |||
| 执行速度（10%） | |||
| 授权成本（5%） | |||
| 加权总分 |
每项评分1-5，乘以权重后求和得到加权总分。除授权费用外，还需考虑总拥有成本：设置时间（配置CI、编写首个测试、团队培训）、编写时间（编写5个真实测试的耗时）、维护时间（框架更新时测试失效的频率）、调试时间（清晰的错误信息可减少此耗时）、基础设施成本（浏览器集群、并行运行器）。
常见技术栈起点——记录选择或偏离的原因：
| 产品类型 | 单元测试 | 集成测试 | E2E测试 | API测试 | 视觉测试 |
|---|---|---|---|---|---|
| React SaaS | Vitest | Testing Library + MSW | Playwright | Supertest | Playwright截图 |
| Next.js | Vitest | Testing Library + MSW | Playwright | Supertest | Playwright截图 |
| Python API | pytest | pytest + Testcontainers | pytest + requests | Schemathesis | N/A |
| 移动端（RN） | Jest | Testing Library + MSW | Detox / Maestro / Appium 3.x | Supertest | Appium截图 |
| Vue SaaS | Vitest | Testing Library + MSW | Playwright | Supertest | Playwright截图 |
| AI/LLM功能 | Vitest | DeepEval | Playwright + Promptfoo evals | Promptfoo / Ragas | N/A |
针对AI/LLM功能，需新增幻觉、偏见、提示注入和隐私的专项风险测试——可查看和（欧盟AI法案）。
ai-system-testingcompliance-testing参考框架：
- CTAL-AT v2.0（ISTQB，高级敏捷测试师，2026）——新的高级认证，要求具备CTFL v4.0基础，并非旧高级认证的更新；取代已退役的基础级CTFL-AT敏捷扩展（已整合进CTFL v4.0）。涵盖测试策略与方法、全团队协作、左移测试、端到端测试、测试坏味道、探索性+AI辅助测试。
- CT-GenAI v1.1（ISTQB，2026-04-27发布）——将LLM驱动的测试基础设施正式化为一门学科；定义AI特定风险类别（幻觉、推理错误、偏见、隐私、AI法规）。
- CTFL v4.0（ISTQB）——基础术语体系；适用于协调不同测试背景的团队。
- HTSM v6.3（Bach）——启发式测试策略模型；强调基于状态的测试和边界启发法。是ISTQB框架的轻量替代方案。
- World Quality Report 2025-26（Capgemini，第17版）——基准数据：43%的企业在QA中尝试生成式AI，15%已规模化应用。可用于规划时定位自身AI adoption阶段。
7. CI Scaling Levers
7. CI扩展手段
When the suite or team grows, CI wall-clock time is the constraint that breaks the strategy. Pull these levers before deleting tests:
- Sharding — split the suite across N parallel runners (Playwright , Jest
--shard=1/4, pytest-xdist, Cypress parallelization). Linear speedup until per-shard fixed costs (install, build) dominate.--shard - Test impact analysis — run only tests affected by the diff instead of the whole suite on every PR. Driven by a dependency graph (Nx affected, Vitest , Bazel) or coverage-to-file maps. Keep the full suite on a nightly/merge gate so nothing rots.
--changed - Caching — cache dependencies, build artifacts, and browser binaries between runs.
- Selective E2E on PR — run smoke E2E on PRs, full E2E on merge/nightly.
Measure the payoff, do not assume it. Parallel efficiency = summed test-run time ÷ wall-clock time; target a value approaching the shard count (e.g. >3x on 4 shards). A low value means fixed setup costs or a long-pole test are eating the speedup. Track CI-minutes-per-PR to catch parallelization that cuts wall-clock time but balloons billed compute.
当测试套件或团队规模增长时，CI耗时会成为制约策略落地的瓶颈。在删除测试前，可采用以下手段：
- 分片执行——将测试套件拆分到N个并行运行器（Playwright 、Jest
--shard=1/4、pytest-xdist、Cypress并行化）。在每个分片的固定成本（安装、构建）占主导前，可实现线性提速。--shard - 测试影响分析——每个PR仅运行受代码变更影响的测试，而非全量套件。基于依赖图（Nx affected、Vitest 、Bazel）或覆盖率-文件映射实现。需保留全量套件在夜间/合并门禁运行，避免测试退化。
--changed - 缓存——在运行间缓存依赖、构建产物和浏览器二进制文件。
- PR阶段选择性运行E2E——PR阶段运行冒烟E2E测试，合并/夜间运行全量E2E测试。
需衡量收益，而非假设有效。并行效率 = 测试总耗时 ÷ 实际耗时；目标值接近分片数量（例如4分片时>3倍）。若数值较低，说明固定设置成本或长尾测试消耗了提速效果。跟踪每个PR的CI分钟数，避免并行化减少了实际耗时但增加了计费算力。
8. Entry/Exit Criteria
8. 准入/准出标准
Define what must be true before testing starts (entry) and before it is done (exit) at each level.
Unit — Entry: code compiles, function has a documented contract (inputs/outputs). Exit: all branches covered, edge cases tested, no skipped tests, coverage target met.
Integration — Entry: unit tests pass, dependent services available or stubbed, test data seeded. Exit: all service boundaries tested, error paths validated, no flaky tests.
E2E — Entry: integration tests pass, staging deployed, test accounts provisioned. Exit: all critical user journeys pass, no P0/P1 defects open, performance within SLA.
Release — Entry: all test levels pass, no CRITICAL/HIGH defects open, release notes drafted. Exit: smoke tests pass in production, monitoring shows no anomalies for an agreed bake window (30 min is a reasonable default — tune to your deploy frequency and alert latency), rollback plan verified.
定义每个测试级别开始前（准入）和完成后（准出）必须满足的条件。
单元测试——准入：代码编译通过，函数有文档化契约（输入/输出）。准出：所有分支覆盖，边缘案例测试完成，无跳过测试，达到覆盖率目标。
集成测试——准入：单元测试通过，依赖服务可用或已 stub，测试数据已预填充。准出：所有服务边界测试完成，异常路径验证通过，无不稳定测试。
E2E测试——准入：集成测试通过，预发布环境已部署，测试账号已创建。准出：所有核心用户流程通过，无P0/P1缺陷，性能符合SLA。
发布——准入：所有测试级别通过，无CRITICAL/HIGH级缺陷，发布说明已撰写。准出：生产环境冒烟测试通过，监控显示在约定观察窗口内无异常（默认30分钟——可根据发布频率和告警延迟调整），回滚计划已验证。
9. Quality Gates & Definition of Done
9. 质量门禁与完成定义
Automated gates that prevent bad code from moving forward.
PR gate (every PR): unit tests pass; integration tests pass; coverage does not decrease (or meets minimum); no new lint errors; SAST scan passes (no new high/critical); bundle size within threshold; at least one reviewer approval.
Merge gate (merge to main): all PR-gate checks pass; E2E smoke suite passes against preview deployment; no merge conflicts; branch up to date with main.
Deploy gate (before production): full E2E suite passes on staging; performance benchmarks within range; security scan passes; feature flags configured; rollback plan documented and tested.
Nightly gate (scheduled): full E2E including edge cases; visual regression; performance/load tests; accessibility scan; dependency vulnerability scan. Results reviewed by QA lead next morning.
Every gate names a concrete pass/fail threshold and is enforced in CI — a gate that can be clicked past is documentation, not a gate.
阻止不良代码进入下一环节的自动化门禁。
PR门禁（每个PR）：单元测试通过；集成测试通过；覆盖率不下降（或达到最小值）；无新代码检查错误；SAST扫描通过（无新增高/严重风险）；包大小在阈值内；至少一位评审人批准。
合并门禁（合并到主分支）：所有PR门禁检查通过；预览部署的E2E冒烟套件通过；无合并冲突；分支与主分支同步。
部署门禁（生产部署前）：预发布环境全量E2E套件通过；性能基准符合要求；安全扫描通过；功能开关已配置；回滚计划已记录并测试。
夜间门禁（定时运行）：全量E2E测试（含边缘案例）；视觉回归测试；性能/负载测试；无障碍扫描；依赖漏洞扫描。结果由QA负责人次日评审。
每个门禁需明确具体的通过/失败阈值，并在CI中强制执行——可手动跳过的门禁只是文档，而非真正的门禁。
10. Metrics & KPIs
10. 指标与KPI
| Metric | Definition | Target | Cadence |
|---|---|---|---|
| Code Coverage | Lines/branches covered by unit + integration | >80% critical services, >60% overall | Per PR |
| Test Pyramid Ratio | Unit:Integration:E2E split | 70:20:10 (±10% tolerance) | Monthly |
| Flakiness Rate | % of runs with non-deterministic failures | <2% | Weekly |
| Defect Escape Rate | % of defects found in prod vs. total | <5% | Per release |
| MTTR | Detection to fix deployed | <4h P0, <24h P1 | Per incident |
| CI Pipeline Duration | Push to green/red signal | <15 min PR, <30 min full | Weekly |
| CI Parallel Efficiency | Summed test time ÷ wall-clock time | Approaching shard count (>3x on 4 shards) | Weekly |
| CI-Minutes-per-PR | Billed compute minutes per PR run | Flat or decreasing | Monthly |
| Defect Density | Defects per 1000 LOC | Decreasing trend | Monthly |
| Automation Rate | % of test cases automated | >80% for regression suite | Quarterly |
| False Positive Rate | % of failures that are not real bugs | <5% | Weekly |
Using metrics: track trends over time, not absolute numbers — a team going 30%→60% coverage is doing great. Set realistic targets from current state (20%→90% in one quarter is a fantasy, not a plan). Review quarterly with leadership; celebrate improvements. Investigate spikes — a sudden flakiness jump signals infrastructure, not laziness. Never use metrics to punish teams. See for full KPI definitions, DORA metrics, and dashboards.
qa-metrics| 指标 | 定义 | 目标 | 跟踪频率 |
|---|---|---|---|
| 代码覆盖率 | 单元+集成测试覆盖的代码行/分支 | 核心服务>80%，整体>60% | 每个PR |
| 测试金字塔比例 | 单元:集成:E2E占比 | 70:20:10（±10%容忍度） | 每月 |
| 测试不稳定率 | 非确定性失败的运行占比 | <2% | 每周 |
| 缺陷逃逸率 | 生产环境发现的缺陷占总缺陷的比例 | <5% | 每次发布 |
| MTTR | 从发现到修复部署的时间 | P0<4小时，P1<24小时 | 每次事故 |
| CI流水线耗时 | 从推送到得到结果的时间 | PR<15分钟，全量<30分钟 | 每周 |
| CI并行效率 | 测试总耗时 ÷ 实际耗时 | 接近分片数量（4分片时>3倍） | 每周 |
| 每个PR的CI分钟数 | 每个PR运行的计费算力分钟数 | 持平或下降 | 每月 |
| 缺陷密度 | 每千行代码的缺陷数 | 呈下降趋势 | 每月 |
| 自动化率 | 自动化测试用例占比 | 回归套件>80% | 每季度 |
| 误报率 | 非真实bug的失败占比 | <5% | 每周 |
指标使用建议：跟踪趋势而非绝对数值——覆盖率从30%提升到60%的团队表现优秀。基于当前状态设定现实目标（一个季度从20%提升到90%是空想，而非计划）。每季度与管理层评审；庆祝改进。调查异常波动——不稳定率突然上升通常是基础设施问题，而非团队懈怠。切勿用指标惩罚团队。查看获取完整KPI定义、DORA指标和仪表盘。
qa-metrics11. Timeline & Milestones
11. 时间线与里程碑
Roll out in phases. Doing everything at once guarantees nothing gets done well.
Phase 1 — Foundation (Weeks 1-4): risk assessment for all product areas; CI pipeline with unit-test gate; baseline metrics (coverage, flakiness, pipeline time); unit tests for top 5 highest-risk areas; select and configure E2E framework. Exit: CI runs unit tests on every PR, baseline metrics documented.
Phase 2 — Coverage Expansion (Weeks 5-10): integration tests for all service boundaries; E2E for top 10 critical journeys; visual regression for key pages; test data management; nightly runs. Exit: all critical paths have E2E coverage, integration tests cover all APIs.
Phase 3 — Quality Gates (Weeks 11-14): coverage gates on PRs (no decrease); performance benchmarks in CI; security scanning; monitoring dashboards for all KPIs. Exit: all four gates (PR, merge, deploy, nightly) active and enforced.
Phase 4 — Optimization (Weeks 15-20): fix or quarantine flaky tests; CI scaling levers (sharding, caching, test impact analysis); synthetic monitoring in production; first quarterly strategy review. Exit: CI under 15 min, flakiness under 2%, first strategy revision published.
Ongoing: quarterly strategy review and revision; monthly metrics review; continuous maintenance (refactor, de-flake, retire).
分阶段推进。一次性完成所有工作必然导致所有工作都无法做好。
阶段1——基础搭建（第1-4周）：所有业务领域的风险评估；带单元测试门禁的CI流水线；基准指标（覆盖率、不稳定率、流水线耗时）；Top5高风险领域的单元测试；选择并配置E2E框架。阶段完成标志：CI在每个PR运行单元测试，基准指标已记录。
阶段2——覆盖扩展（第5-10周）：所有服务边界的集成测试；Top10核心流程的E2E测试；核心页面的视觉回归测试；测试数据管理；夜间运行。阶段完成标志：所有核心路径有E2E覆盖，集成测试覆盖所有API。
阶段3——质量门禁（第11-14周）：PR覆盖率门禁（不允许下降）；CI中的性能基准；安全扫描；所有KPI的监控仪表盘。阶段完成标志：四个门禁（PR、合并、部署、夜间）均已激活并强制执行。
阶段4——优化（第15-20周）：修复或隔离不稳定测试；CI扩展手段（分片、缓存、测试影响分析）；生产环境synthetic监控；首次季度策略评审。阶段完成标志：CI耗时<15分钟，不稳定率<2%，首次策略修订版发布。
持续阶段：每季度策略评审与修订；每月指标评审；持续维护（重构、修复不稳定测试、淘汰过时测试）。
Anti-Patterns
反模式
100% coverage targets. Diminishing returns past 80%. The last 20% means testing getters and trivial code while ignoring integration gaps where real bugs live. Set coverage per module by risk, not a blanket number.
Ice cream cone (inverted pyramid). Too many E2E, too few unit. Symptoms: CI 45+ minutes, tests break on every UI change, nobody trusts the suite. Fix by freezing E2E growth and decomposing existing E2E into lower levels.
Strategy as a one-time document. Written once and never updated is worse than none — it gives false confidence. Build in review triggers: quarterly calendar review, post-incident, new product area, team composition change.
Tool-first thinking. "We should use Playwright" is a tool choice masquerading as a plan. Start from what you need to validate, then pick tools that fit. The document justifies tool choices, never leads with them.
No metrics = no accountability. A strategy without measurable targets is a wish list. Every section connects to a KPI. If you cannot define success in numbers, question whether the element belongs.
Testing in isolation. A strategy living only in the QA wiki is invisible to developers. It must live in PR templates, CI gates, and the Definition of Done. If developers do not see it daily, it does not exist.
Copy-paste strategy. Taking another company's strategy verbatim ignores your risk profile, team skills, and constraints. Templates are starting points; every section is tailored.
Automating everything immediately. Manual exploratory testing has enormous value, especially early. Automate regression, keep exploration manual. The strategy specifies what stays manual and why.
100%覆盖率目标。超过80%后收益递减。最后20%的覆盖率意味着测试getter和无关代码，却忽略了真正容易出bug的集成缺口。按风险为每个模块设置覆盖率，而非统一数值。
甜筒型（倒置金字塔）。E2E测试过多，单元测试不足。症状：CI耗时45+分钟，UI变更导致大量测试失效，团队不信任测试套件。解决方法：冻结E2E测试增长，将现有E2E测试拆分为更低级别的测试。
一次性策略文档。撰写后从未更新的文档不如没有——会带来虚假的信心。内置评审触发条件：每季度日历评审、事故后评审、新业务领域上线、团队构成变动。
工具优先思维。「我们应该用Playwright」是伪装成计划的工具选择。从需要验证的内容出发，再选择适配的工具。文档应论证工具选择的合理性，而非以工具为起点。
无指标=无问责。没有可衡量目标的策略只是愿望清单。每个环节都需关联KPI。若无法用数值定义成功，需质疑该环节是否必要。
孤立的测试策略。仅存在于QA wiki的策略对开发人员不可见。必须融入PR模板、CI门禁和完成定义。若开发人员日常看不到，策略就等于不存在。
复制粘贴的策略。直接照搬其他公司的策略会忽略自身的风险状况、团队技能和约束条件。模板只是起点；每个部分都需定制。
立即自动化所有内容。手动探索性测试有巨大价值，尤其是早期阶段。自动化回归测试，保留手动探索。策略需明确哪些内容保留手动测试及原因。
Verification
验证
Prove the produced document is complete before calling it done. Run against the saved strategy file:
bash
DOC=docs/qa-strategy.md在完成前，先验证生成的文档是否完整。针对保存的策略文件运行以下命令：
bash
DOC=docs/qa-strategy.md1. All 13 numbered section headings present (Executive Summary → Revision History)
1. 确认所有13个带编号的章节标题存在（从执行摘要到修订历史）
grep -cE '^### [0-9]+.' "$DOC" # expect 13
grep -cE '^### [0-9]+.' "$DOC" # 预期输出13
2. Every row in the Metrics & KPIs table has a non-empty Target cell
2. 确认「指标与KPI」表格的每一行都有非空的目标单元格
(no "| | " gaps in the target column) — visually scan the table block:
（目标列无「| | 」空白）——可手动扫描表格块：
grep -nE '^|' "$DOC" | grep -iE 'target|coverage|flak|mttr|escape'
grep -nE '^|' "$DOC" | grep -iE 'target|coverage|flak|mttr|escape'
3. A revision history and a named owner exist
3. 确认存在修订历史和指定负责人
grep -niE 'revision history|owner:' "$DOC"
Then sanity-check the pyramid math by hand: target unit% + integration% + E2E% should sum to ~100%. If the document recommends sharding, confirm a parallel-efficiency or CI-minutes target appears in the Metrics table — an optimization with no metric is a guess.
---grep -niE 'revision history|owner:' "$DOC"
然后手动验证金字塔比例的合理性：目标单元测试占比+集成测试占比+E2E测试占比应约为100%。若文档建议分片执行，确认「指标」表格中包含并行效率或CI分钟数目标——无指标的优化只是猜测。
---Done When
完成标志
- A strategy document exists at an agreed path and returns 13 (all sections Executive Summary → Revision History populated).
grep -cE '^### [0-9]+\.' - Test pyramid target ratios are defined with concrete counts and a timeline to reach them; unit+integration+E2E percentages sum to ~100%.
- Entry and exit criteria are written for each level (unit, integration, E2E, release).
- Tool selection is documented with a scored, weighted rationale matrix — not just tool names.
- Quality gates are defined for all four stages (PR, merge, deploy, nightly), each with a concrete pass/fail threshold.
- Every Metrics & KPIs row has a non-empty Target and a tracking cadence.
- 策略文档已保存到约定路径，且返回13（所有从执行摘要到修订历史的章节已填充）。
grep -cE '^### [0-9]+\.' - 已定义测试金字塔目标比例及具体数量，并有达成时间线；单元+集成+E2E测试占比之和约为100%。
- 已为每个级别（单元、集成、E2E、发布）编写准入和准出标准。
- 工具选择已通过加权评分矩阵记录依据——而非仅列出工具名称。
- 已定义四个阶段（PR、合并、部署、夜间）的质量门禁，每个门禁都有具体的通过/失败阈值。
- 「指标与KPI」表格的每一行都有非空的目标值和跟踪频率。
Related Skills
相关技能
- risk-based-testing — run first; deep dive into risk-assessment methodology and AI/LLM-specific failure classes. This skill consumes its matrix.
- test-planning — go here for a single sprint or release plan; test-strategy is the multi-quarter umbrella above it.
- test-reliability — flake root-cause triage, self-healing locators, and quarantine mechanics referenced in the pyramid-rebalance step.
- qa-metrics — full KPI definitions, DORA metrics, Test Impact Analysis, dashboards, trend analysis.
- release-readiness — go/no-go checklists, canary analysis, release confidence scoring.
- ci-cd-integration — pipeline configuration, gate implementation, and smart sharding setup.
- shift-left-testing — techniques for moving validation earlier.
- ai-system-testing — when the strategy covers AI/LLM features, defines the eval-suite layer.
- compliance-testing — when the strategy serves a regulated audience (GDPR, EU AI Act, EAA, US state laws).
- risk-based-testing——优先运行；深入风险评估方法论和AI/LLM特定故障类别。本技能会使用其生成的风险矩阵。
- test-planning——用于单迭代或单版本计划；test-strategy是其上层的多季度框架。
- test-reliability——金字塔重新平衡步骤中提到的不稳定测试根因分析、自修复定位器和隔离机制。
- qa-metrics——完整KPI定义、DORA指标、测试影响分析、仪表盘、趋势分析。
- release-readiness——发布准入清单、金丝雀分析、发布信心评分。
- ci-cd-integration——流水线配置、门禁实现、智能分片设置。
- shift-left-testing——将验证环节前置的技术。
- ai-system-testing——当策略覆盖AI/LLM功能时，定义评估套件层。
- compliance-testing——当策略服务于受监管的受众（GDPR、欧盟AI法案、EAA、美国州级法律）时使用。
Reference Files (in references/
)
references/参考文件（位于references/
）
references/- diagrams-and-worksheets.md — pyramid shape diagrams, current/target state worksheets, the full 5x5 risk matrix, and the 13-section output skeleton.
- strategy-templates.md — four fully worked strategy documents (SaaS, e-commerce, API-first, media), plus a step-by-step pyramid analysis worksheet and a risk-matrix feature-inventory template.
- diagrams-and-worksheets.md——金字塔形态示意图、当前/目标状态工作表、完整5×5风险矩阵、13章节输出骨架。
- strategy-templates.md——4份完整的策略文档示例（SaaS、电商、API优先、媒体），以及分步金字塔分析工作表和风险矩阵功能清单模板。