Back to Details

testing-in-production

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese
<objective> Production is the only environment that is production. Every other environment is an approximation — staging never replicates real data volume, traffic, or third-party quirks. This skill covers how to validate quality in production safely: controlled blast radius, automated rollback, guardrail metrics, and smoke tests that catch problems before users do. The recurring failure it prevents: shipping to 100% of users with no flag to flip, no baseline to compare against, and no tested way back. </objective>
<objective> 生产环境是唯一的真实生产环境。其他所有环境都是近似模拟——staging环境永远无法复制真实的数据量、流量或第三方服务的特殊情况。本技能涵盖如何安全地在生产环境中验证质量：控制影响范围、自动回滚、防护指标，以及在用户发现问题前捕获问题的冒烟测试。它能避免的常见故障：在没有开关标志、没有基准可对比、没有经过测试的回退方案的情况下，直接向100%用户发布功能。 </objective>

Quick Route

快速指引

SituationGo to
Shipping a feature behind a flagFeature Flag Testing
Ramping traffic 1% → 100% with gatesProgressive Rollout + 
references/rollout-policy.md
Need post-deploy checks on every releaseProduction Smoke Tests + 
references/patterns.md
Deciding what numbers gate the rolloutGuardrail Metrics
New code path with no user-visible change yetDark Launches
Proving the rollback actually worksVerification
场景参考内容
在标志后发布功能Feature Flag Testing
按1% → 100%逐步放量并设置关卡渐进式发布 + 
references/rollout-policy.md
每次发布后需要执行部署后检查生产环境冒烟测试 + 
references/patterns.md
确定发布关卡对应的指标防护指标
新代码路径暂未对用户可见暗启动
验证回滚机制有效验证环节

Discovery Questions

调研问题

Check 
.agents/qa-project-context.md
first. If it exists, use it as context and skip questions already answered there.
Feature flag system:
  • Do you have a feature flag platform? (LaunchDarkly, Statsig — now part of OpenAI, GrowthBook, Unleash, Flagsmith, Harness FME — formerly Split, custom, none)
  • How are flags managed? (Dashboard, config file, environment variables)
  • Can flags target specific users, percentages, or segments?
  • How many active flags exist today? Is there a cleanup process?
Rollout capability:
  • Can you deploy to a subset of traffic? (Canary infrastructure, weighted routing, feature flags)
  • How long does a deployment take? How long does a rollback take?
  • Do you have blue-green or rolling deployments?
  • Can you route traffic by region, user cohort, or percentage?
Monitoring maturity:
  • What observability is in place? (APM, logging, error tracking, metrics)
  • Do you have dashboards for error rate, latency, and business metrics?
  • Are alerts configured with appropriate thresholds?
  • Can you compare metrics between canary and baseline in real time?
Production access and safety:
  • Who has production access? Is there an approval process?
  • Are there dedicated test accounts in production?
  • Can you run operations in production without affecting real user data?
  • Is there a production incident response process?
首先查看
.agents/qa-project-context.md
。如果该文件存在，以此为上下文并跳过已解答的问题。
功能标志系统：
  • 是否拥有功能标志平台？（LaunchDarkly、Statsig — 现属于OpenAI、GrowthBook、Unleash、Flagsmith、Harness FME — 前身为Split、自定义平台、无）
  • 标志如何管理？（仪表盘、配置文件、环境变量）
  • 标志是否可针对特定用户、百分比或用户群体？
  • 当前有多少个活跃标志？是否有清理流程？
发布能力：
  • 是否可向部分流量部署？（金丝雀基础设施、加权路由、功能标志）
  • 一次部署需要多长时间？回滚需要多长时间？
  • 是否采用蓝绿部署或滚动部署？
  • 是否可按地区、用户群组或百分比路由流量？
监控成熟度：
  • 已部署哪些可观测性工具？（APM、日志、错误追踪、指标）
  • 是否有错误率、延迟和业务指标的仪表盘？
  • 是否配置了合适阈值的告警？
  • 是否可实时对比金丝雀版本与基准版本的指标？
生产环境访问与安全：
  • 谁拥有生产环境访问权限？是否有审批流程？
  • 生产环境中是否有专用测试账号？
  • 是否可在生产环境中执行操作而不影响真实用户数据？
  • 是否有生产环境事件响应流程？

Core Principles

核心原则

1. Production is the final test environment

1. 生产环境是最终测试环境

Staging approximates production. It does not replicate production's data volume, traffic patterns, third-party integrations, infrastructure quirks, or user behavior. Testing in production is not reckless — it is realistic. The question is not whether to test in production, but how to do it safely.
预发布环境是生产环境的近似模拟，但无法复制生产环境的数据量、流量模式、第三方集成、基础设施特殊情况或用户行为。在生产环境中测试并非鲁莽，而是务实。问题不在于是否要在生产环境测试，而在于如何安全地进行测试。

2. Safety through blast radius control

2. 通过控制影响范围保障安全

Every production test must answer: "If this goes wrong, how many users are affected?" The answer must be as small as possible. Feature flags, canary deploys, and traffic splitting exist to shrink the blast radius from 100% to 1% or less.
每一项生产环境测试都必须回答：「如果出现问题，会影响多少用户？」答案应尽可能小。Feature flags、金丝雀部署和流量拆分的作用就是将影响范围从100%缩小到1%甚至更低。

3. Always have a tested rollback plan

3. 始终要有经过测试的回滚方案

Before any production test begins, the rollback mechanism must be identified, tested, and fast. "Disable the flag" is a good rollback plan. "Redeploy the previous version" is acceptable. "We'll figure it out" is not a plan. A rollback you have never fired is a hypothesis, not a plan — see Verification for how to prove it works.
在任何生产环境测试开始前，必须确定、测试并确保回滚机制快速有效。「关闭标志」是一个好的回滚方案，「重新部署上一版本」也可接受，但「我们到时再想办法」不算方案。从未执行过的回滚只是假设，而非方案——请查看验证环节了解如何证明回滚机制有效。

4. Monitoring is a prerequisite, not a nice-to-have

4. 监控是必要前提，而非可选功能

You cannot test in production without monitoring. If you cannot measure error rates, latency, and business metrics in real time, you cannot detect problems. Fix monitoring gaps before adding production tests.
没有监控就无法在生产环境中测试。如果无法实时测量错误率、延迟和业务指标，就无法检测问题。在添加生产环境测试前，先修复监控缺口。

5. Production tests must be non-destructive

5. 生产环境测试必须是非破坏性的

Production tests must never corrupt real user data, send real notifications to real users, charge real payment methods, or create side effects that require manual cleanup. Synthetic accounts, test flags, and isolated resources are mandatory.
生产环境测试绝不能损坏真实用户数据、向真实用户发送通知、扣除真实支付费用，或产生需要手动清理的副作用。必须使用模拟账号、测试标志和隔离资源。

Feature Flag Testing

Feature Flag Testing

Feature flags are the safest mechanism for production testing. They decouple deployment from release and provide instant rollback.
Feature flags是生产环境测试最安全的机制，它将部署与发布解耦，并提供即时回滚能力。

Test with flags ON and OFF

测试标志开启与关闭状态

Every flagged feature needs tests in both states. The flag-off path is the rollback path and must work flawlessly — use 
setFeatureFlag(name, true|false, { userId: TEST_USER_ID })
to drive both. See 
references/patterns.md
for the full ON/OFF test pair.
每个带标志的功能都需要在两种状态下进行测试。标志关闭路径就是回滚路径，必须确保其完美运行——使用
setFeatureFlag(name, true|false, { userId: TEST_USER_ID })
来驱动两种状态的测试。完整的开启/关闭测试对请参考
references/patterns.md
。

Flag lifecycle testing

标志生命周期测试

Flags are not just on or off. They transition through states, and each transition must be validated.
Flag lifecycle:
  Created → Targeting internal users → Canary (1%) → Partial (10-50%) → Full (100%) → Cleanup (removed)

Test at each stage:
  - Internal: Feature works for internal accounts, hidden from external
  - Canary: Metrics are comparable between flag-on and flag-off cohorts
  - Partial: No performance degradation at scale
  - Full: All user segments work correctly
  - Cleanup: Code with flag removed behaves identically to flag-on
标志并非只有开启或关闭两种状态，它会经历多个状态转换，每个转换都必须验证。
标志生命周期:
  创建 → 面向内部用户 → 金丝雀（1%） → 部分放量（10-50%） → 全量（100%） → 清理（移除）

每个阶段的测试:
  - 内部阶段：功能对内部账号可用，对外部用户隐藏
  - 金丝雀阶段：标志开启与关闭群体的指标可比
  - 部分放量阶段：大规模放量下无性能下降
  - 全量阶段：所有用户群体均可正常使用
  - 清理阶段：移除标志后的代码行为与标志开启时一致

Stale flag cleanup

过期标志清理

Flags left in code become technical debt. Run a weekly CI job that queries the flag provider for flags that are 100% rolled out and older than 14 days. These are candidates for code cleanup — remove the flag branching logic and retain only the enabled path. Removing the flag without removing the dead code is half a cleanup.
留在代码中的标志会变成技术债务。运行每周CI任务，查询标志提供商中已100%发布且存在超过14天的标志，这些标志是代码清理的候选对象——移除标志分支逻辑，仅保留启用路径。只移除标志而不删除无用代码不算完整清理。

Flag combination testing

标志组合测试

When multiple flags interact, test the combinations that matter. Do not test all 2^N combinations — focus on flags that affect the same user flow (e.g. 
new-checkout
, 
express-pay
, 
discount-engine-v2
all touch checkout). Pick the critical combinations: all-new, a representative mixed state, and all-legacy. See 
references/patterns.md
for the combination-test loop.
当多个标志相互作用时，测试关键组合即可。无需测试所有2^N种组合——重点关注影响同一用户流程的标志（例如
new-checkout
、
express-pay
、
discount-engine-v2
均涉及结账流程）。选择关键组合：全新版本、代表性混合状态、全旧版本。组合测试循环请参考
references/patterns.md
。

Progressive Rollout

渐进式发布

Vendor-native canary analysis. Before hand-rolling the rollout-policy YAML below, check whether your platform already does it: LaunchDarkly Guarded Rollouts (auto-monitored progressive rollouts with metric-based auto-rollback; uses a frequentist sequential-testing analysis model since early 2026), Statsig Auto-tune, Argo Rollouts AnalysisRun, Flagger, Harness Continuous Verification. If you have one, prefer it — the integration with your metrics and rollback mechanics is cheaper than maintaining a custom analysis loop.
AI feature rollout is its own pattern: model variant + prompt as a flag value, with cost guardrails and a kill switch. LaunchDarkly AI Configs / AgentControl (AI Configs rebranded under the AgentControl umbrella, announced May 2026) is the documented path for shipping LLM features behind progressive rollout. See 
release-readiness
for the full pattern.
厂商原生金丝雀分析：在手动编写以下发布策略YAML之前，请检查你的平台是否已支持该功能：LaunchDarkly Guarded Rollouts（基于指标自动监控的渐进式发布，支持自动回滚；自2026年初使用频率学派序贯测试分析模型）、Statsig Auto-tune、Argo Rollouts AnalysisRun、Flagger、Harness Continuous Verification。如果已有这些工具，请优先使用——它们与你的指标和回滚机制的集成比维护自定义分析循环成本更低。
AI功能发布是独立模式：将模型变体+提示作为标志值，搭配成本防护开关和紧急关闭机制。LaunchDarkly AI Configs / AgentControl（AI Configs于2026年5月更名为AgentControl）是发布LLM功能的渐进式发布标准路径。完整模式请参考
release-readiness
。

Canary stages: 1% to 100%

金丝雀阶段：1%到100%

A structured rollout with explicit promotion criteria at each stage.
StageTrafficHold TimeKey Checks
Canary1%15-30 minError rate, crash rate, exceptions
Early adopters10%1-2 hoursLatency P95, conversion rate
Partial50%2-4 hoursAll guardrails, business metrics
Full100%24 hours monitoringLong-tail issues, batch job compatibility
结构化发布流程，每个阶段都有明确的升级标准。
阶段流量占比等待时长关键检查项
金丝雀1%15-30分钟错误率、崩溃率、异常
早期尝鲜用户10%1-2小时P95延迟、转化率
部分放量50%2-4小时所有防护指标、业务指标
全量100%24小时监控长尾问题、批处理任务兼容性

Automated promotion and rollback

自动升级与回滚

Define machine-checkable conditions for advancing between stages — 
hold_duration
plus metric conditions (
error_rate_5xx < 0.5%
, 
latency_p95 < 500ms
, 
crash_rate == 0
). Automatic rollback fires when guardrails are breached, with no human approval needed: 
error_rate_5xx > 2x_baseline for 5m
, 
latency_p99 > 3x_baseline for 5m
, 
crash_rate > 0.1% for 2m
, each notifying on-call. Gate on error-budget burn rate, not only raw multipliers, so slow burns that still blow the SLO are caught. See 
references/rollout-policy.md
for the full promotion YAML, rollback triggers, and SLO-gate config.
定义阶段间升级的机器可验证条件——
hold_duration
加上指标条件（
error_rate_5xx < 0.5%
、
latency_p95 < 500ms
、
crash_rate == 0
）。当防护指标被触发时自动回滚，无需人工审批：
error_rate_5xx > 2x_baseline 持续5分钟
、
latency_p99 > 3x_baseline 持续5分钟
、
crash_rate > 0.1% 持续2分钟
，每种情况都会通知值班人员。以错误预算消耗率为判断标准，而非仅依赖原始倍数，这样可以捕获缓慢消耗但仍会突破SLO的情况。完整的升级YAML、回滚触发器和SLO关卡配置请参考
references/rollout-policy.md
。

Verify the rollback fired correctly

验证回滚是否正确执行

Auto-rollback firing is not the same as the incident being resolved. After a rollback triggers, do not declare "recovered" until you have confirmed the system is actually back:
  1. Re-run the health-check smoke test against production (
    GET /api/health
    returns 
    healthy
    and the previous 
    version
    ).
  2. Confirm the flag/deploy state reverted — query the flag platform that the flag is off (or the deploy that the previous build is serving), don't assume.
  3. Confirm guardrail metrics returned to baseline — error rate, P99 latency, and crash rate back within their pre-deploy windows.
  4. Confirm the rollback notification reached on-call (Slack/PagerDuty) so the incident is owned.
Only when all four hold is the rollback verified. See Verification for the staging dry-run that proves this chain before first production use.
自动回滚触发并不意味着事件已解决。回滚触发后，在确认系统确实恢复前，不要宣布「已恢复」：
  1. 重新运行健康检查冒烟测试（
    GET /api/health
    返回
    healthy
    和之前的
    version
    ）。
  2. 确认标志/部署状态已回退——查询标志平台确认标志已关闭（或部署已切换到上一版本），不要假设。
  3. 确认防护指标回到基准值——错误率、P99延迟和崩溃率回到部署前的正常范围。
  4. 确认回滚通知已送达值班人员（Slack/PagerDuty），确保事件有人负责。
只有当以上四项全部满足时，回滚才算验证完成。首次生产环境使用前，请通过验证环节的预发布环境演练来验证整个流程。

Production Smoke Tests

生产环境冒烟测试

Post-deploy critical path tests

部署后关键路径测试

Run immediately after every deployment as a pipeline stage (not only in pre-deploy CI). These verify core functionality works with production configuration, data, and infrastructure: a 
/api/health
check, the authentication flow with synthetic credentials, core data loading, and search. Configure 
retries: 1
and a 
timeout
so a flaky post-deploy run doesn't block the pipeline on the first blip. See 
references/patterns.md
for the full 
production-smoke.spec.ts
.
每次部署后立即作为流水线阶段运行（不仅在部署前CI中运行）。这些测试验证核心功能在生产环境配置、数据和基础设施下是否正常工作：
/api/health
检查、使用模拟凭证的认证流程、核心数据加载和搜索。配置
retries: 1
和
timeout
，避免首次运行时的不稳定情况阻塞流水线。完整的
production-smoke.spec.ts
请参考
references/patterns.md
。

Synthetic user accounts

模拟用户账号

Production test accounts must be clearly distinguishable from real users: a reserved email pattern (
smoke-test+{env}@yourcompany.com
), an 
is_synthetic = true
flag, and exclusion from analytics, billing, and email campaigns. Create them via admin API, and prefer short-lived OIDC / workload-identity tokens over long-lived passwords. See 
references/patterns.md
for the full conventions.
生产环境测试账号必须与真实用户明确区分：使用预留邮箱格式（
smoke-test+{env}@yourcompany.com
）、
is_synthetic = true
标志，并且排除在分析、计费和邮件营销之外。通过管理API创建，优先使用短期OIDC/工作负载身份令牌而非长期密码。完整约定请参考
references/patterns.md
。

Non-destructive assertions

非破坏性断言

Production smoke tests must read, not write. When writes are unavoidable, clean up in fixture teardown so cleanup runs whether the test passes or fails.
Do not call 
test.afterEach()
inside a 
test()
body — Playwright registers hooks at describe/file scope, so a hook registered mid-test never schedules teardown and throws 
test.afterEach() can only be called in a describe block
. The data leaks. Use an auto-cleanup fixture that records created resource IDs and deletes them on teardown (or 
try/finally
for a one-off script). See 
references/patterns.md
for the fixture-based create-verify-cleanup pattern.
生产环境冒烟测试应仅读取数据，不写入数据。当必须写入时，在测试夹具清理阶段进行清理，无论测试通过或失败都要执行。
不要在
test()
函数内部调用
test.afterEach()
——Playwright在describe/文件级别注册钩子，在测试中途注册的钩子永远不会执行清理，还会抛出
test.afterEach() can only be called in a describe block
错误，导致数据泄露。使用自动清理夹具记录创建的资源ID并在清理时删除（或对于一次性脚本使用
try/finally
）。基于夹具的创建-验证-清理模式请参考
references/patterns.md
。

Guardrail Metrics

防护指标

What to monitor during rollout

发布过程中需要监控的内容

CategoryMetricComparison MethodAlert Threshold
ErrorsHTTP 5xx ratevs. pre-deploy baseline>2x baseline for 5 min
ErrorsUnhandled exception countvs. pre-deploy baselineAny new exception type
LatencyP50 response timevs. pre-deploy baseline>1.5x baseline
LatencyP95 response timevs. pre-deploy baseline>2x baseline
LatencyP99 response timevs. pre-deploy baseline>3x baseline
BusinessConversion ratevs. 7-day averageDrop >5%
BusinessRevenue per sessionvs. 7-day averageDrop >10%
ClientCrash rate (mobile)vs. previous release>0.1% increase
ClientJavaScript error ratevs. pre-deploy baseline>2x baseline
InfraCPU utilizationabsolute>80% sustained
InfraMemory utilizationabsolute>85% sustained
类别指标对比方式告警阈值
错误HTTP 5xx错误率与部署前基准对比超过基准2倍并持续5分钟
错误未处理异常数量与部署前基准对比出现任何新异常类型
延迟P50响应时间与部署前基准对比超过基准1.5倍
延迟P95响应时间与部署前基准对比超过基准2倍
延迟P99响应时间与部署前基准对比超过基准3倍
业务转化率与7天平均值对比下降超过5%
业务每会话收入与7天平均值对比下降超过10%
客户端崩溃率（移动端）与上一版本对比上升超过0.1%
客户端JavaScript错误率与部署前基准对比超过基准2倍
基础设施CPU使用率绝对值持续超过80%
基础设施内存使用率绝对值持续超过85%

Baseline comparison

基准对比

Compare canary metrics against a control group running the previous version, not against historical data alone.
Comparison approaches (best to worst):
  1. Canary vs. control: split traffic, compare groups in real time (best)
  2. Before/after: compare post-deploy metrics to pre-deploy window (good)
  3. Historical: compare to same time last week (acceptable for trends)
  4. Absolute thresholds: fixed thresholds regardless of baseline (fragile)
将金丝雀版本的指标与运行上一版本的对照组对比，而非仅与历史数据对比。
对比方式（从优到劣）:
  1. 金丝雀vs对照组：拆分流量，实时对比两组数据（最优）
  2. 部署前后对比：将部署后指标与部署前窗口数据对比（良好）
  3. 历史对比：与上周同一时间数据对比（适用于趋势分析）
  4. 绝对阈值：不考虑基准的固定阈值（脆弱）

Statistical significance

统计显著性

For business metrics (conversion, revenue), small sample sizes produce noisy results. Wait for statistical significance before drawing conclusions.
Minimum sample sizes for rollout decisions:
  - Error rate: 1,000 requests (errors are rare events, need volume)
  - Latency: 500 requests (more stable, converges faster)
  - Conversion rate: 5,000 sessions (business metrics have high variance)
  - Crash rate: 10,000 app launches (crashes are rare events)

Rule of thumb: if you don't have enough traffic at 1% to reach
significance in 30 minutes, increase to 5% or extend the hold window.
对于业务指标（转化率、收入），小样本量会产生噪声结果。在得出结论前，需等待数据达到统计显著性。
发布决策的最小样本量:
  - 错误率：1000次请求（错误是罕见事件，需要足够量）
  - 延迟：500次请求（更稳定，收敛更快）
  - 转化率：5000次会话（业务指标方差高）
  - 崩溃率：10000次应用启动（崩溃是罕见事件）

经验法则：如果1%流量在30分钟内无法达到足够样本量，可将流量提升至5%或延长等待窗口。

Dark Launches

暗启动

Dark launches deploy new functionality to production but hide it from users. Real production traffic exercises the new code path without user-visible impact.
暗启动是将新功能部署到生产环境但对用户隐藏，让真实生产流量触发新代码路径而不产生用户可见的影响。

Traffic shadowing

流量镜像

Duplicate incoming requests to the new service. Compare responses without returning the new response to the user.
Request flow:
  User → Load Balancer → Production Service (returns response to user)
                       ↘ Shadow Service (processes request, logs result, discards)

What to compare:
  - Response status codes: shadow should match production
  - Response body: diff for semantic equivalence (ignore timestamps, IDs)
  - Latency: shadow should not be significantly slower
  - Error rate: shadow should not produce more errors
将传入请求复制到新服务，对比响应但不将新服务的返回结果发送给用户。
请求流程:
  用户 → 负载均衡器 → 生产服务（向用户返回响应）
                       ↘ 镜像服务（处理请求、记录结果、丢弃响应）

需要对比的内容:
  - 响应状态码：镜像服务应与生产服务一致
  - 响应体：语义等价对比（忽略时间戳、ID）
  - 延迟：镜像服务不应明显慢于生产服务
  - 错误率：镜像服务不应产生更多错误

Parallel execution

并行执行

For migrations (new database, new algorithm, new service), run both the old and new path in production. The old path returns the result to the user; the new path runs asynchronously, logs differences, and discards its result. Track the match rate over time — target 99%+ match before cutting over.
Shadow launch timeline:
  Week 1: Deploy shadow, start comparing, expect <50% match
  Week 2: Fix mismatches, match rate should climb to 90%+
  Week 3: Match rate stable at 99%+, handle remaining edge cases
  Week 4: Cut over: shadow becomes primary, old becomes shadow
  Week 5: Remove old path after 1 week of stability
对于迁移场景（新数据库、新算法、新服务），在生产环境中同时运行旧路径和新路径。旧路径向用户返回结果；新路径异步运行，记录差异并丢弃结果。跟踪匹配率随时间的变化——切换前目标匹配率需达到99%以上。
镜像启动时间线:
  第1周：部署镜像服务，开始对比，预期匹配率<50%
  第2周：修复不匹配问题，匹配率应提升至90%+
  第3周：匹配率稳定在99%+，处理剩余边缘情况
  第4周：切换：镜像服务成为主服务，旧服务变为镜像
  第5周：稳定运行1周后移除旧路径

Anti-Patterns

反模式

Testing in production without monitoring

无监控情况下在生产环境测试

Running production tests without dashboards and alerts is flying blind. You will not know if your tests caused an issue until a user reports it. Fix: Monitoring is a prerequisite. Before adding any production test, verify you can see error rates, latency, and key business metrics in real time. Set up alerts before the first test runs.
没有仪表盘和告警的生产环境测试等同于盲目操作。只有当用户反馈问题时，你才会知道测试引发了故障。 **修复方案：**监控是必要前提。在添加任何生产环境测试前，确保可实时查看错误率、延迟和关键业务指标。在首次测试前配置好告警。

No rollback plan

无回滚方案

"We'll deploy a fix if something goes wrong" is not a rollback plan. Under pressure, fixes take longer, introduce new bugs, and extend the outage. Fix: Every production test or rollout must have a documented rollback mechanism that takes less than 5 minutes to execute — feature flag disable, previous deployment, or traffic reroute — and a verification step that confirms it actually recovered the system (see Verification).
「如果出问题我们就部署修复版本」不算回滚方案。在压力下，修复需要更长时间，还会引入新bug，延长故障时长。 **修复方案：**每一项生产环境测试或发布都必须有记录在案的回滚机制，执行时间需少于5分钟——关闭feature flag、回滚到上一版本或重新路由流量——并包含验证步骤确认系统已恢复（见验证环节）。

Destructive operations in production tests

生产环境测试中的破坏性操作

Production tests that create real orders, send real emails, or modify real user data are not tests — they are incidents waiting to happen. Fix: Use synthetic accounts flagged as test data. Use sandbox modes for payment and email. Clean up created data in fixture teardown so it always runs. If a test cannot be made non-destructive, it does not belong in production.
创建真实订单、发送真实邮件或修改真实用户数据的生产环境测试不是测试，而是随时可能发生的故障。 **修复方案：**使用标记为测试数据的模拟账号。对支付和邮件服务使用沙箱模式。在测试夹具清理阶段清理创建的数据，确保无论测试结果如何都会执行。如果测试无法做到非破坏性，则不应在生产环境中进行。

Cleanup that only runs on success

仅在测试成功时执行清理

A cleanup step placed after an assertion never runs when the assertion fails, leaking exactly the test data it was meant to remove. Calling 
test.afterEach()
inside a 
test()
body is the same trap — it throws or is ignored. Fix: Put teardown in an auto-cleanup fixture or a 
finally
block so it runs on pass and fail alike. See 
references/patterns.md
.
将清理步骤放在断言之后，当断言失败时清理永远不会执行，导致恰好需要清理的测试数据泄露。在
test()
函数内部调用
test.afterEach()
也是同样的陷阱——会抛出错误或被忽略。 **修复方案：**将清理步骤放在自动清理夹具或
finally
块中，确保测试通过或失败时都会执行。请参考
references/patterns.md
。

Testing in production instead of pre-production

用生产环境测试替代预生产环境测试

Production testing supplements pre-production testing. It does not replace it. If staging is broken and you are "testing in production" because it is the only working environment, fix staging first. Fix: Maintain a working pre-production environment. Use production testing for what only production can validate: real traffic, real data volumes, real third-party integrations.
生产环境测试是预生产环境测试的补充，而非替代。如果预发布环境损坏，你因为「只有生产环境可用」而在生产环境测试，应先修复预发布环境。 **修复方案：**维护可用的预生产环境。生产环境测试仅用于验证只有生产环境才能覆盖的场景：真实流量、真实数据量、真实第三方集成。

Canary deploys without comparison

无对比的金丝雀部署

Deploying to 1% of traffic but not comparing canary metrics against a control group misses the entire point. You are just deploying slowly, not detecting problems. Fix: Always compare canary metrics against a baseline. Use side-by-side dashboards or automated canary analysis tools (Kayenta, Argo Rollouts analysis).
向1%流量部署但不对比金丝雀版本与对照组的指标，完全失去了金丝雀部署的意义。你只是在缓慢部署，而不是检测问题。 **修复方案：**始终将金丝雀版本的指标与基准对比。使用并排仪表盘或自动化金丝雀分析工具（Kayenta、Argo Rollouts analysis）。

Stale feature flags

过期feature flags

Flags that are fully rolled out but never removed accumulate. After a year, you have 200 flags with unknown interactions, and every code path has branching logic that nobody understands. Fix: Every flag gets an expiration date at creation time. After full rollout + 2 weeks of stability, remove the flag and its dead branch. Track flag age and alert when flags exceed their expiration.
已全量发布但从未移除的标志会不断累积。一年后，你会有200个交互未知的标志，每条代码路径都有无人理解的分支逻辑。 **修复方案：**每个标志在创建时都设置过期日期。全量发布并稳定运行2周后，移除标志及其无用分支。跟踪标志存在时长，当标志超过过期时间时触发告警。

Verification

验证环节

Prove the rollback path actually fires before the first production use — a rollback you have never triggered is a hypothesis. Smallest check first:
  1. Trip a guardrail in staging. Inject failure (e.g. force 
    error_rate_5xx
    above 
    2x_baseline
    , or fail the health check 3 times) on a staged rollout wired to the same 
    automatic_rollback
    policy. Confirm the rollback action fires within its 
    for:
    window.
  2. Confirm the reverted state. Re-run the health-check smoke test and assert 
    status === 'healthy'
    and the previous 
    version
    . Query the flag platform/deploy that the previous build is serving — don't assume.
  3. Confirm metrics recovered. Error rate, P99 latency, and crash rate are back inside their pre-deploy windows.
  4. Confirm the notification fired. The rollback alert reached the on-call channel (Slack/PagerDuty).
  5. Smoke tests are wired into the pipeline. Run the deploy job against staging and confirm the post-deploy smoke stage executes and gates promotion — 
    npx playwright test production-smoke.spec.ts
    exits 0.
If steps 1–4 cannot be demonstrated in staging, the rollback is unverified and the rollout is not ready.
Agent shortcut: vendor MCP servers exist for LaunchDarkly, GrowthBook, Unleash, Flagsmith, Statsig, and Harness FME, letting an AI agent flip flags and read rollout metrics directly during these checks rather than driving the dashboard by hand.
在首次生产环境使用前，必须证明回滚路径确实有效——从未触发过的回滚只是假设。按以下顺序逐步验证：
  1. 在预发布环境触发防护指标。在配置了相同
    automatic_rollback
    策略的预发布发布流程中注入故障（例如强制
    error_rate_5xx
    超过
    2x_baseline
    ，或连续3次健康检查失败），确认回滚操作在
    for:
    窗口内触发。
  2. 确认回退状态。重新运行健康检查冒烟测试，断言
    status === 'healthy'
    且版本为上一版本。查询标志平台/部署确认已切换到上一版本——不要假设。
  3. 确认指标已恢复。错误率、P99延迟和崩溃率回到部署前的正常范围。
  4. 确认通知已触发。回滚告警已送达值班频道（Slack/PagerDuty）。
  5. 冒烟测试已接入流水线。针对预发布环境运行部署任务，确认部署后冒烟测试阶段已执行并作为升级关卡——
    npx playwright test production-smoke.spec.ts
    返回0。
如果步骤1-4无法在预发布环境中演示，说明回滚未经过验证，发布流程未准备就绪。
**Agent快捷方式：**针对LaunchDarkly、GrowthBook、Unleash、Flagsmith、Statsig和Harness FME存在厂商MCP服务器，允许AI agent在这些检查中直接切换标志和读取发布指标，无需手动操作仪表盘。

Done When

完成标准

  • Feature flag rollout plan is documented with explicit percentage steps (1% → 10% → 50% → 100%) and named guardrail metrics at each stage.
  • Canary analysis is configured with automated pass/fail criteria so promotion and rollback decisions do not require manual metric comparison.
  • Production smoke tests run as a pipeline stage on every deploy (not only in CI pre-deploy) and the stage exits 0 against production.
  • Rollback trigger conditions are defined, documented, and demonstrated to fire correctly via the Verification staging dry-run (guardrail tripped → rollback fired → reverted state and recovered metrics confirmed) before first production use.
  • Production test data strategy is documented, specifying whether synthetic users or anonymized real users are used and how they are excluded from analytics and billing.
  • Feature flag发布计划已记录在案，包含明确的百分比步骤（1% → 10% → 50% → 100%）和每个阶段对应的防护指标。
  • 金丝雀分析已配置自动化通过/失败标准，升级和回滚决策无需手动对比指标。
  • 生产环境冒烟测试作为流水线阶段在每次部署时运行（不仅在CI预部署阶段），且该阶段在生产环境中返回0。
  • 回滚触发条件已定义、记录，并通过验证环节的预发布环境演练证明可正确触发（防护指标触发→回滚执行→确认回退状态和指标恢复），之后才能首次在生产环境使用。
  • 生产环境测试数据策略已记录在案，明确说明使用模拟用户还是匿名真实用户，以及如何将其排除在分析和计费之外。

Reference Files (in 
references/
)

参考文件（位于
references/
目录）

  • rollout-policy.md — full promotion-criteria YAML, automatic-rollback triggers, and the error-budget / SLO-gate config.
  • patterns.md — flag ON/OFF and combination tests, the 
    production-smoke.spec.ts
    suite, synthetic-account conventions, and the fixture-based non-destructive create-verify-cleanup pattern.
  • rollout-policy.md — 完整的升级标准YAML、自动回滚触发器，以及错误预算/SLO关卡配置。
  • patterns.md — 标志开启/关闭和组合测试、
    production-smoke.spec.ts
    测试套件、模拟账号约定，以及基于夹具的非破坏性创建-验证-清理模式。

Related Skills

相关技能

  • release-readiness — Go/no-go for the whole release; production testing is the post-deploy verification step inside it. Go there for the release checklist, not the rollout mechanics.
  • synthetic-monitoring — Scheduled probes that run continuously after the rollout is complete. Go there for ongoing SLA validation, not in-flight rollout safety.
  • observability-driven-testing — Uses prod traces and logs as the input to design new tests. Go there when telemetry tells you what to test, not when you need to ship safely.
  • qa-metrics — Where guardrail metrics and rollout criteria feed into dashboards and KPIs.
  • ci-cd-integration — Wiring the smoke-test stage and rollout gates into the pipeline.
  • test-environments — Pre-production environments that production testing complements, never replaces.
  • release-readiness — 整个发布的通过/不通过判定；生产环境测试是其中的部署后验证步骤。如需发布清单，请参考该技能，而非发布机制细节。
  • synthetic-monitoring — 发布完成后持续运行的定时探测。如需持续SLA验证，请参考该技能，而非发布过程中的安全保障。
  • observability-driven-testing — 将生产追踪和日志作为输入来设计新测试。当遥测数据告诉你需要测试什么时，请参考该技能，而非安全发布场景。
  • qa-metrics — 防护指标和发布标准在此汇总到仪表盘和KPI中。
  • ci-cd-integration — 将冒烟测试阶段和发布关卡接入流水线。
  • test-environments — 生产环境测试是预生产环境的补充，而非替代。