synthetic-monitoring
Compare original and translation side by side
🇺🇸
Original
English🇨🇳
Translation
Chinese<objective>
Synthetic monitoring runs scripted tests against production on a schedule, 24/7. It catches outages, performance degradation, and broken flows before real users report them — at 3 AM when traffic is zero, probes are the only thing checking your app works. A login page that always returns 200 but never authenticates passes a naive uptime check; a synthetic probe that asserts the dashboard loads catches it. This skill covers probe design, alerting integration, SLA validation, multi-region execution, and the runbook discipline that keeps a 3 AM page actionable.
</objective>
<objective>
合成监控会按计划全天候（24/7）对生产环境运行脚本化测试。它能在真实用户报告问题之前——比如凌晨3点流量为零时——发现停机、性能下降和流程故障。一个始终返回200但无法完成认证的登录页能通过简单的可用性检查，但一个验证仪表盘是否加载的合成探测任务能发现这个问题。本技能涵盖探测设计、告警集成、SLA验证、多区域执行，以及确保凌晨3点告警可快速处理的运行手册规范。
</objective>
Quick Route
快速导航
| Situation | Go to |
|---|---|
| Picking a platform (Checkly, Datadog, Grafana, CloudWatch…) | Platform Options |
| Writing a probe (login, API health, search) | Probe Design → |
| Probe runs but users still report outages | Failure Modes |
| Alerts too noisy or missing real outages | Alerting Integration |
| Calculating downtime budget for an SLA | SLA Validation |
| Probe failed at 3 AM and on-call is lost | runbook template in |
| 场景 | 跳转至 |
|---|---|
| 选择平台（Checkly、Datadog、Grafana、CloudWatch…） | 平台选项 |
| 编写探测任务（登录、API健康检查、搜索） | 探测设计 → |
| 探测任务正常运行但用户仍报告故障 | 故障模式 |
| 告警过于嘈杂或遗漏真实故障 | 告警集成 |
| 计算SLA的停机预算 | SLA验证 |
| 凌晨3点探测任务失败，值班人员不知所措 | |
Discovery Questions
调研问题
Check first. If it exists, use it as context and skip questions already answered there. Each answer changes the probe set, the alerting config, or the SLA math.
.agents/qa-project-context.mdCritical flows (decides which probes you write):
- What are the 5-10 most important user journeys (login, search, checkout, signup, core workflow)? These become your probe list.
- Which flows, if broken, cause immediate revenue loss or churn? These get the shortest frequency and page on-call.
- Are there flows that break silently (data sync, background jobs, webhooks)? Silent failures need probes most — no user reports them.
- Do you have documented SLAs/SLOs for availability and response time? They set the downtime budget and the alert thresholds.
Current monitoring (decides where the gaps are):
- What exists today (uptime checks, APM, error tracking, dashboards)? Avoid duplicating; find the gap.
- Are there gaps between what monitoring covers and what users experience? That gap is where probes earn their keep.
- How do you learn about production issues today (alerts, user reports, social media)? If it's user reports, detection time is your first metric to fix.
- What was the last outage, and how long before it was detected? Sets the detection-time target probes must beat.
Infrastructure (decides regions and CDN assertions):
- Is the app served from multiple regions or one? Multi-region serving needs multi-region probing.
- Are there CDN, caching, or edge layers that could mask origin failures? If yes, probes must assert on /origin headers.
x-cache - Do third-party dependencies (payment, auth, email) have their own monitoring? Their status pages feed your runbooks.
- What alerting systems exist (PagerDuty, OpsGenie, Slack, email)? Decides the routing config.
Test accounts (decides data safety):
- Do dedicated synthetic test accounts exist in production? Without them, probes pollute real data and analytics.
- Can test accounts be excluded from analytics, billing, and email campaigns? If not, probe traffic skews every downstream number.
- Is there API access for programmatic account setup and data cleanup? Decides whether create-then-delete probes are viable.
首先查看。如果该文件存在，以此为上下文并跳过已回答的问题。每个答案都会改变探测任务集、告警配置或SLA计算方式。
.agents/qa-project-context.md关键流程（决定编写哪些探测任务）：
- 最重要的5-10个用户旅程是什么（登录、搜索、结账、注册、核心工作流）？这些将成为你的探测任务列表。
- 哪些流程一旦中断会立即导致收入损失或用户流失？这些流程需要设置最短的探测频率，并触发值班告警。
- 是否存在无声中断的流程（数据同步、后台任务、Webhook）？无声故障最需要探测任务——没有用户会报告这类问题。
- 是否有记录在案的可用性和响应时间SLA/SLO？它们决定了停机预算和告警阈值。
当前监控情况（决定存在哪些缺口）：
- 当前已有哪些监控手段（可用性检查、APM、错误跟踪、仪表盘）？避免重复，找出缺口。
- 监控覆盖范围与用户实际体验之间是否存在差距？这个差距就是探测任务的价值所在。
- 目前如何获知生产环境问题（告警、用户报告、社交媒体）？如果依赖用户报告，检测时间是第一个需要优化的指标。
- 上一次停机是什么时候，多久后才被检测到？这为探测任务设定了必须超越的检测时间目标。
基础设施（决定探测区域和CDN验证逻辑）：
- 应用是从多个区域还是单个区域提供服务？多区域服务需要多区域探测。
- 是否存在可能掩盖源站故障的CDN、缓存或边缘层？如果有，探测任务必须验证/源站请求头。
x-cache - 第三方依赖（支付、认证、邮件）是否有自己的监控？它们的状态页面可作为运行手册的参考。
- 现有哪些告警系统（PagerDuty、OpsGenie、Slack、邮件）？这决定了告警路由配置。
测试账户（决定数据安全性）：
- 生产环境中是否有专用的合成测试账户？没有的话，探测任务会污染真实数据和分析结果。
- 测试账户能否被排除在分析、计费和邮件营销之外？如果不能，探测流量会扭曲所有下游数据。
- 是否有程序化创建账户和清理数据的API权限？这决定了“创建后删除”类探测任务是否可行。
Core Principles
核心原则
1. Synthetic tests validate the user experience continuously
1. 合成测试持续验证用户体验
RUM tells you what happened; synthetic tells you what is happening right now, whether or not real users are active. At 3 AM when traffic is zero, synthetic probes are the only thing checking the app works. Complement RUM, never replace it: synthetic covers known paths with predictable inputs, RUM discovers the creative ways real users break things.
RUM告诉你已经发生了什么；合成监控告诉你当前正在发生什么，无论是否有真实用户活跃。凌晨3点流量为零时，合成探测任务是唯一检查应用是否正常运行的手段。补充而非替代RUM：合成监控用可预测的输入覆盖已知路径，RUM则发现真实用户非常规操作导致的问题。
2. Keep probes simple and fast
2. 保持探测任务简单快速
A probe that takes 2 minutes and touches 15 pages is not a probe — it is an E2E suite running in production. Probes are short (per-probe wall-clock budget under 30 seconds), focused (one critical path each), and stable (, zero flakiness tolerance). The 30-second ceiling is a hard budget: a slow probe that "still passes" is masking a degradation users feel.
retries: 0一个耗时2分钟、涉及15个页面的探测任务不是真正的探测——它是在生产环境运行的端到端测试套件。探测任务应简短（单个任务耗时不超过30秒）、聚焦（每个任务对应一条关键路径）且稳定（，零容错）。30秒是硬性限制：一个“仍能通过”的缓慢探测任务，实则掩盖了用户能感知到的性能下降。
retries: 03. Alert on trends, not single failures
3. 基于趋势告警，而非单次失败
A single probe failure is noise — network blips and DNS hiccups cause them constantly. Two consecutive failures are a signal; failures from 2+ regions confirm it is not local. Configure consecutive-failure and multi-region thresholds, or alert fatigue teaches the team to ignore the pager.
单次探测失败属于噪音——网络波动和DNS故障经常导致此类情况。连续两次失败是信号；2个及以上区域同时失败则确认这不是本地问题。配置连续失败和多区域阈值，否则告警疲劳会让团队忽视告警。
4. Probes must not affect production data
4. 探测任务不得影响生产数据
Probes run every few minutes, 24/7. Even small side effects (a created record, an incremented counter) compound. Probes must be non-destructive, created data cleaned up immediately, and synthetic traffic excluded from analytics, billing, and — critically — from SLO/error-budget math itself, or probes inflate your own reliability numbers.
探测任务每分钟运行一次，全天候不间断。即使是微小的副作用（创建一条记录、递增一个计数器）也会不断累积。探测任务必须是非破坏性的，创建的数据需立即清理，合成流量需排除在分析、计费之外——关键是要排除在SLO/错误预算计算之外，否则探测任务会虚高可靠性指标。
5. Assert on the goal, not the status code
5. 验证目标达成，而非仅状态码
A login page that returns 200 but never authenticates is broken. A search page that returns 200 with zero results is broken. Probes assert that the user can accomplish their goal — data loads, auth succeeds, results appear — not merely that the page returns a 2xx.
返回200但无法完成认证的登录页是故障的；返回200但没有结果的搜索页也是故障的。探测任务应验证用户能否达成目标——数据加载完成、认证成功、结果显示——而非仅验证页面返回2xx状态码。
Probe Design
探测设计
Design probes around critical user journeys, not infrastructure components. Users do not care if your load balancer is healthy — they care if they can log in and use the product.
| Probe | What It Validates | Frequency | Timeout |
|---|---|---|---|
| Homepage load | DNS, CDN, server, basic rendering | 1 min | 10s |
| Login flow | Authentication service, session management | 5 min | 15s |
| Core workflow | Primary value-delivering action (create document, run report) | 5 min | 20s |
| API health | Backend services, database connectivity | 1 min | 5s |
| Search | Search index, query processing, result rendering | 5 min | 15s |
| Checkout (if applicable) | Payment integration (sandbox mode), cart, order creation | 10 min | 25s |
| Third-party integrations | OAuth providers, email delivery, file storage | 10 min | 15s |
围绕关键用户旅程而非基础设施组件设计探测任务。用户不关心你的负载均衡器是否健康——他们关心能否登录并使用产品。
| 探测任务 | 验证内容 | 频率 | 超时时间 |
|---|---|---|---|
| 首页加载 | DNS、CDN、服务器、基础渲染 | 1分钟 | 10秒 |
| 登录流程 | 认证服务、会话管理 | 5分钟 | 15秒 |
| 核心工作流 | 核心价值交付操作（创建文档、生成报告） | 5分钟 | 20秒 |
| API健康检查 | 后端服务、数据库连接 | 1分钟 | 5秒 |
| 搜索功能 | 搜索索引、查询处理、结果渲染 | 5分钟 | 15秒 |
| 结账流程（如适用） | 支付集成（沙箱模式）、购物车、订单创建 | 10分钟 | 25秒 |
| 第三方集成 | OAuth提供商、邮件投递、文件存储 | 10分钟 | 15秒 |
Probe implementations
探测任务实现
Three probe shapes cover most needs: a login flow (Playwright browser), an API health check (status + auth + latency budget), and a search probe (fill query → submit → assert on results, not just status). Keep each to one critical path, a tight timeout, and . See for runnable login-flow, API-health, and search probes plus the environment-aware config.
retries: 0references/probe-implementations.md三种探测模式可覆盖大多数需求：登录流程（Playwright浏览器）、API健康检查（状态+认证+延迟预算）、搜索探测（填写查询→提交→验证结果，而非仅状态码）。每个任务对应一条关键路径，设置严格超时时间，且。可查看获取可运行的登录流程、API健康检查、搜索探测任务代码，以及环境感知配置。
retries: 0references/probe-implementations.mdNon-destructive probe patterns
非破坏性探测模式
Safe patterns:
- Read-only operations: GET requests, page loads, searches
- Sandbox transactions: payment in test mode, email to internal addresses
- Create-then-delete: create a draft, verify, delete immediately
- Synthetic flag: operations tagged as synthetic, excluded from processing
Dangerous patterns (avoid):
- Creating real orders, tickets, or user-facing records
- Triggering real notifications (email, SMS, push)
- Modifying shared resources (config, permissions, settings)
- Operations that cannot be automatically cleaned up安全模式:
- 只读操作：GET请求、页面加载、搜索
- 沙箱事务：测试模式下的支付、发送至内部地址的邮件
- 创建后删除：创建草稿、验证、立即删除
- 合成标记：标记为合成的操作，排除在业务处理之外
危险模式（避免）:
- 创建真实订单、工单或面向用户的记录
- 触发真实通知（邮件、短信、推送）
- 修改共享资源（配置、权限、设置）
- 无法自动清理的操作Dedicated test accounts
专用测试账户
Account requirements:
- Clearly identifiable: email contains "synthetic" or "monitor"
- Flagged in database with an is_synthetic flag (is_synthetic = true)
- Excluded from analytics, billing, email campaigns, support queues
- Excluded from RUM and SLO/error-budget pipelines (synthetic traffic is not real traffic)
- Pre-populated with stable test data that probes can rely on
- Credentials in a secrets manager (process.env), rotated quarterly
- Separate accounts per concurrent probe (avoid state conflicts)账户要求:
- 可清晰识别：邮箱包含"synthetic"或"monitor"
- 在数据库中标记`is_synthetic`字段（is_synthetic = true)
- 排除在分析、计费、邮件营销、支持队列之外
- 排除在RUM和SLO/错误预算流程之外（合成流量不是真实流量）
- 预填充探测任务可依赖的稳定测试数据
- 凭证存储在密钥管理器中（process.env），每季度轮换一次
- 每个并发探测任务使用独立账户（避免状态冲突）Platform Options
平台选项
| Platform | Strengths | When to Use |
|---|---|---|
| Checkly | Playwright-native, code-first, Git integration; Rocky AI agent (GA 2026) now does automated root-cause analysis across Playwright/API/Multistep/TCP/DNS/ICMP checks; CLI access from any AI agent; MCP server | Teams already using Playwright for E2E |
| Datadog Synthetic | Deep APM integration, browser and API tests | Teams on the Datadog platform |
| Grafana Synthetic Monitoring | Open source, integrates with Grafana dashboards; pairs with k6 2.x; pin a version channel (v1.x/v2.x) for reproducibility | Teams using the Grafana stack |
| AWS CloudWatch Synthetics | Blueprints (heartbeat, API, broken-link, visual diff); Python or Node Puppeteer canaries | Teams already on AWS |
| New Relic Synthetics | Full-stack observability integration | Teams on the New Relic platform |
| Better Stack | Lightweight uptime + status pages + on-call | SMB-friendly, fast setup |
| Uptime Kuma | OSS, self-hosted, lightweight | Self-hosting requirement, small surface area |
| Custom (Playwright / k6 / Puppeteer + cron) | Full control, no vendor lock-in | Budget-constrained or custom requirements |
Probes can be authored in Playwright (TS/JS), Puppeteer, k6 (JS — k6 2.0 shipped May 2026 with AI-assisted test authoring and a clearer Assertions API; first-class for synthetic), or Python (CloudWatch Synthetics, Checkly). Pick what your team already maintains. Avoid: Pingdom for new setups — it is a legacy uptime tool; prefer code-first alternatives (Checkly, Grafana, custom Playwright) that version-control probes alongside your app.
| 平台 | 优势 | 适用场景 |
|---|---|---|
| Checkly | 原生支持Playwright、代码优先、Git集成；Rocky AI Agent（2026年正式发布）可跨Playwright/API/多步骤/TCP/DNS/ICMP检查执行自动化根因分析；支持从任意AI Agent访问CLI；提供MCP服务器 | 已使用Playwright进行端到端测试的团队 |
| Datadog Synthetic | 深度APM集成，支持浏览器和API测试 | 使用Datadog平台的团队 |
| Grafana Synthetic Monitoring | 开源，与Grafana仪表盘集成；可搭配k6 2.x；支持固定版本通道（v1.x/v2.x）以保证可重复性 | 使用Grafana栈的团队 |
| AWS CloudWatch Synthetics | 提供蓝图（心跳、API、断链、视觉对比）；支持Python或Node Puppeteer金丝雀测试 | 已使用AWS的团队 |
| New Relic Synthetics | 全栈可观测性集成 | 使用New Relic平台的团队 |
| Better Stack | 轻量级可用性监控+状态页面+值班管理 | 面向中小企业，快速部署 |
| Uptime Kuma | 开源、自托管、轻量 | 有自托管需求、监控范围较小的场景 |
| 自定义（Playwright / k6 / Puppeteer + cron） | 完全可控，无供应商锁定 | 预算有限或有定制需求的场景 |
探测任务可使用Playwright（TS/JS）、Puppeteer、k6（JS——k6 2.0于2026年5月发布，支持AI辅助测试编写和更清晰的Assertions API；是合成监控的首选工具）或Python（CloudWatch Synthetics、Checkly）编写。选择团队已在维护的技术栈。避免：新部署使用Pingdom——它是传统可用性工具；优先选择代码优先的替代方案（Checkly、Grafana、自定义Playwright），可将探测任务与应用代码一起纳入版本控制。
Custom and managed implementations
自定义与托管实现
Self-managed: schedule Playwright probes with a GitHub Actions cron (every 5 minutes), inject prod credentials via secrets, and report results to a monitoring webhook. Managed: Checkly is Playwright-native and runs from multiple locations on a fixed frequency. Either way, make probes environment-aware so the same code runs against staging and production with different base URLs and thresholds. See for the GitHub Actions workflow, the Checkly config, the alert-routing rules, and the runbook template.
schedulereferences/platforms-and-ci.md自托管：使用GitHub Actions的 cron（每5分钟一次）调度Playwright探测任务，通过密钥注入生产环境凭证，并将结果上报至监控Webhook。托管：Checkly原生支持Playwright，可在多个位置按固定频率运行。无论哪种方式，都要让探测任务具备环境感知能力，以便同一代码可在 staging 和生产环境运行，只需修改基础URL和阈值。可查看获取GitHub Actions工作流、Checkly配置、告警路由规则和运行手册模板。
schedulereferences/platforms-and-ci.mdAlerting Integration
告警集成
Not every probe failure is an incident. Configure rules that cut noise while catching real problems.
Alerting rules:
- Single failure: log, do not alert (transient network issue)
- 2 consecutive failures from same region: warning (possible issue)
- 2 consecutive failures from 2+ regions: alert on-call (confirmed outage)
- Latency >2x baseline for 10 minutes: warning (performance degradation)
- Latency >3x baseline for 5 minutes: alert on-call (severe degradation)
- Any probe timeout: alert if 3 consecutive (service unresponsive)Routing. Critical failures on revenue paths (login, checkout, api-health) page on-call (PagerDuty + Slack incidents) with a short repeat interval; warnings on secondary probes go to a Slack monitoring channel; info-level events use a long repeat interval. The probe must tag each result with a and label for these routes to match — see the routing note in for the full and the tagging step.
severityprobereferences/platforms-and-ci.mdalerting-rules.yamlSuppress synthetic alerts during planned maintenance. A maintenance window should silence synthetic paging (the probes will fail by design) and exclude that window from error-budget math, or scheduled work burns budget and pages on-call for nothing.
Alert message format — include enough context to start investigating immediately:
Alert template:
Title: [SYNTHETIC] {probe_name} failing from {region}
Severity: {critical|warning|info}
Consecutive failures: {count}
Last success: {timestamp}
Error: {error_message}
Duration: {last_response_time_ms}ms (threshold: {threshold}ms)
Dashboard: {link_to_dashboard}
Runbook: {link_to_runbook}
Regions affected: {list_of_failing_regions}The points at a per-probe runbook (six lines: what it tests, first checks, manual repro, escalation, dashboard, owner). See the template in .
{link_to_runbook}references/platforms-and-ci.md并非所有探测失败都是事件。配置规则以减少噪音，同时捕捉真实问题。
告警规则:
- 单次失败：记录日志，不触发告警（临时网络问题）
- 同一区域连续2次失败：警告（可能存在问题）
- 2个及以上区域连续2次失败：触发值班告警（确认停机）
- 延迟超过基线2倍达10分钟：警告（性能下降）
- 延迟超过基线3倍达5分钟：触发值班告警（严重性能下降）
- 探测任务超时：连续3次超时触发告警（服务无响应）路由。关键收入路径（登录、结账、API健康检查）的严重故障需通过PagerDuty + Slack事件触发值班告警，重复间隔较短；次要探测任务的警告发送至Slack监控频道；信息级事件使用较长的重复间隔。探测任务必须为每个结果标记和标签，以便路由规则匹配——可查看中的路由说明，获取完整的和标签配置步骤。
severityprobereferences/platforms-and-ci.mdalerting-rules.yaml计划维护期间屏蔽合成告警。维护窗口内应屏蔽合成告警（探测任务会因设计原因失败），并将该窗口从错误预算计算中排除，否则计划内工作会消耗预算并无故触发值班告警。
告警消息格式——包含足够上下文以立即开始排查：
告警模板:
标题: [SYNTHETIC] {probe_name} 在 {region} 区域失败
级别: {critical|warning|info}
连续失败次数: {count}
上次成功时间: {timestamp}
错误信息: {error_message}
响应时长: {last_response_time_ms}ms（阈值: {threshold}ms）
仪表盘: {link_to_dashboard}
运行手册: {link_to_runbook}
受影响区域: {list_of_failing_regions}{link_to_runbook}references/platforms-and-ci.mdSLA Validation
SLA验证
Availability calculation
可用性计算
Availability = (total_minutes - downtime_minutes) / total_minutes × 100
Where:
- total_minutes = calendar month in minutes (43,200 for a 30-day month)
- downtime_minutes = minutes where synthetic probes detected failure
SLA tiers (downtime/month, monthly basis on 43,200 min):
99.0% = 432 min = 7h 12min (basic web app)
99.9% = 43.2 min = 43min 12s (business application)
99.95% = 21.6 min = 21min 36s (critical SaaS)
99.99% = 4.32 min = 4min 19s (infrastructure/platform)These are common availability targets, not prescriptive tiers. Pick targets from a user-impact analysis, not by tier name. Modern practice (Google SRE Workbook, OpenSLO) favors explicit SLO + error-budget policies over labelled tiers — define what user-visible failure looks like, set the budget user impact tolerates, and let the SLO follow. References: https://sre.google/workbook/ ; https://openslo.com/
可用性 = (总分钟数 - 停机分钟数) / 总分钟数 × 100
其中:
- 总分钟数 = 当月日历分钟数（30天月份为43200分钟）
- 停机分钟数 = 合成探测任务检测到故障的分钟数
SLA等级（每月停机时间，基于43200分钟计算）:
99.0% = 432分钟 = 7小时12分钟 (基础Web应用)
99.9% = 43.2分钟 = 43分钟12秒 (业务应用)
99.95% = 21.6分钟 = 21分钟36秒 (关键SaaS应用)
99.99% = 4.32分钟 = 4分钟19秒 (基础设施/平台)这些是常见的可用性目标，而非强制等级。应基于用户影响分析选择目标，而非仅看等级名称。现代实践（《Google SRE工作手册》、OpenSLO）更倾向于明确的SLO + 错误预算策略，而非标签化等级——定义用户可见的故障是什么，设置用户可容忍的预算，然后据此制定SLO。参考资料：https://sre.google/workbook/ ; https://openslo.com/
Response time percentiles
响应时间百分位
Track percentiles, not averages — averages hide the worst experiences.
SLA response time targets (example):
Homepage load: P50 < 1s, P95 < 3s, P99 < 5s
API response: P50 < 200ms, P95 < 500ms, P99 < 1s
Search results: P50 < 500ms, P95 < 2s, P99 < 4s
Login flow: P50 < 2s, P95 < 5s, P99 < 8s跟踪百分位而非平均值——平均值会掩盖最差的用户体验。
SLA响应时间目标（示例）:
首页加载: P50 < 1秒, P95 < 3秒, P99 < 5秒
API响应: P50 < 200毫秒, P95 < 500毫秒, P99 < 1秒
搜索结果: P50 < 500毫秒, P95 < 2秒, P99 < 4秒
登录流程: P50 < 2秒, P95 < 5秒, P99 < 8秒Error budget tracking
错误预算跟踪
Error budget connects SLA targets to engineering decisions.
Error budget calculation:
SLO: 99.9% availability on a 43,200-min month
Budget: 0.1% of total time = 43.2 minutes/month
Budget consumed this month: 12 minutes (28%)
Budget remaining: 31.2 minutes (72%)
Actions by budget status:
>50% remaining: normal operations, ship features
25-50% remaining: caution, review recent changes
<25% remaining: freeze non-critical deploys, focus on reliability
Budget exhausted: incident mode, every deploy needs extra scrutinyExclude synthetic-probe downtime caused by your own maintenance windows from this calculation, or planned work shows as budget burn.
错误预算将SLA目标与工程决策关联起来。
错误预算计算:
SLO: 月度43200分钟内达到99.9%的可用性
预算: 总时间的0.1% = 43.2分钟/月
本月已消耗预算: 12分钟（28%）
剩余预算: 31.2分钟（72%）
根据预算状态采取的行动:
>50%剩余: 正常运营，发布新功能
25-50%剩余: 谨慎行事，审查近期变更
<25%剩余: 冻结非关键部署，聚焦可靠性
预算耗尽: 进入事件模式，所有部署需额外审查将计划维护期间的合成探测停机时间从计算中排除，否则计划内工作会显示为预算消耗。
Multi-Region Monitoring
多区域监控
Run probes from regions where your users are. A service that works from us-east-1 but is broken from ap-southeast-1 is broken for APAC users.
Region selection strategy:
- Minimum 3 regions for global services
- Always include: closest to primary infrastructure, largest user base, farthest from primary
- Example for US-primary service: us-east-1, eu-west-1, ap-southeast-1
- Example for EU-primary service: eu-west-1, us-east-1, ap-northeast-1Track regional latency separately — a global average hides regional degradation.
Regional latency dashboard:
Region | P50 | P95 | Status
us-east-1 | 120ms | 340ms | healthy
eu-west-1 | 280ms | 620ms | healthy
ap-southeast | 450ms | 1200ms | warning (P95 above threshold)
Alert when:
- Any region's P95 exceeds its regional threshold
- Latency difference between regions exceeds 5x (CDN or routing issue)
- A region that was healthy becomes consistently degradedCDN validation. Synthetic probes verify CDN caching by checking response headers (, ) for and confirming the header matches the expected provider. This catches CDN misconfigurations — and origin failures masked by a stale cache — before users hit slow uncached responses.
x-cachecf-cache-statusHITserver在用户所在区域运行探测任务。一个在us-east-1正常运行但在ap-southeast-1故障的服务，对亚太用户来说就是故障的。
区域选择策略:
- 全球服务至少选择3个区域
- 必须包含：离主基础设施最近的区域、用户量最大的区域、离主基础设施最远的区域
- 以美国为主基础设施的服务示例：us-east-1、eu-west-1、ap-southeast-1
- 以欧洲为主基础设施的服务示例：eu-west-1、us-east-1、ap-northeast-1单独跟踪区域延迟——全局平均值会掩盖区域性能下降。
区域延迟仪表盘:
区域 | P50 | P95 | 状态
us-east-1 | 120毫秒 | 340毫秒 | 健康
eu-west-1 | 280毫秒 | 620毫秒 | 健康
ap-southeast | 450毫秒 | 1200毫秒 | 警告（P95超过阈值）
告警触发条件:
- 任意区域的P95超过其区域阈值
- 区域间延迟差异超过5倍（CDN或路由问题）
- 原本健康的区域持续出现性能下降CDN验证。合成探测任务通过检查响应头（、）是否为，并确认头与预期提供商匹配，来验证CDN缓存。这能在用户遇到缓慢的未缓存响应之前，发现CDN配置错误——以及被 stale 缓存掩盖的源站故障。
x-cachecf-cache-statusHITserverAnti-Patterns
反模式
Complex probes that break often
频繁故障的复杂探测任务
A probe that navigates 10 pages, fills 5 forms, and asserts on 20 elements is an E2E test, not a synthetic probe. When it breaks, you cannot tell if the app is down or the probe is flaky. Fix: one critical path per probe, under 30 seconds, under 5 assertions. A failure should make it immediately clear what is broken.
一个导航10个页面、填写5个表单、验证20个元素的探测任务是端到端测试，而非合成探测任务。当它故障时，你无法判断是应用停机还是探测任务不稳定。修复方案：每个探测任务对应一条关键路径，耗时不超过30秒，验证项不超过5个。故障应能立即明确指出问题所在。
Alerting on every single failure
每次失败都触发告警
Network blips, DNS hiccups, and transient cloud issues cause occasional failures. Alerting on every one produces noise that teaches the team to ignore alerts. Fix: require 2-3 consecutive failures and failures from 2+ regions before paging. Escalating severity: first failure logs, second warns, third pages.
网络波动、DNS故障和临时云问题会导致偶尔的失败。每次失败都触发告警会产生噪音，让团队忽视告警。修复方案：触发值班告警前，要求连续2-3次失败且2个及以上区域同时失败。分级告警：首次失败记录日志，第二次发送警告，第三次触发值班告警。
No test account isolation
测试账户未隔离
Probes sharing one account interfere — one probe changes a setting, another fails because it expected the default. Fix: one dedicated synthetic account per concurrent probe, flagged synthetic, excluded from analytics and billing.
共享一个账户的探测任务会相互干扰——一个探测任务修改了设置，另一个因预期默认设置而失败。修复方案：每个并发探测任务使用独立的专用合成账户，标记为合成账户，排除在分析和计费之外。
Monitoring only the happy path
仅监控正常路径
Probes that only check "page loads, returns 200" miss broken functionality behind a loading page. A login that returns 200 but never authenticates is not working. Fix: assert on meaningful content — data loads, auth succeeds, the core action completes, search returns results. One "can the user accomplish their goal" probe is worth ten "does the page return 200" probes.
仅检查“页面加载、返回200”的探测任务会遗漏加载页面背后的故障。返回200但无法完成认证的登录页是无法正常使用的。修复方案：验证有意义的内容——数据加载完成、认证成功、核心操作完成、搜索返回结果。一个“用户能否达成目标”的探测任务，价值胜过十个“页面是否返回200”的探测任务。
No runbook for probe failures
探测故障无运行手册
An alert fires at 3 AM. On-call sees "Login probe failing" but has no idea what to check first, what the probe does, or how to tell a real outage from a probe issue. Fix: every probe links a runbook (what it tests; first checks — third-party status, recent deploys, app telemetry; manual repro; escalation; dashboard link). See the template in .
references/platforms-and-ci.mdEmerging pattern (2026): agent-driven first response. Tools like Checkly Rocky (GA 2026, automated root-cause analysis across check types) and Honeycomb Canvas Skills (Agent Observability, launched May 2026; open-source repo ships a honeycomb-investigator and instrumentation-advisor for Claude Code and Cursor) read probe context + linked runbook + telemetry and post a candidate diagnosis to chat. Treat this as triage assist — it shortens MTTR for routine failures, but it does not replace on-call judgment for novel incidents.
honeycombio/agent-skill凌晨3点触发告警。值班人员看到“登录探测任务失败”，但不知道首先检查什么、探测任务的作用是什么，也无法区分真实停机和探测任务问题。修复方案：每个探测任务都链接一个运行手册（测试内容；首次检查项——第三方状态、近期部署、应用遥测；手动复现步骤；升级流程；仪表盘链接）。可查看中的模板。
references/platforms-and-ci.md2026年新兴模式：Agent驱动的首次响应。Checkly Rocky（2026年正式发布，可跨检查类型执行自动化根因分析）和Honeycomb Canvas Skills（Agent可观测性，2026年5月发布；开源仓库为Claude Code和Cursor提供honeycomb-investigator和instrumentation-advisor）等工具，会读取探测上下文+链接的运行手册+遥测数据，并在聊天中发布候选诊断结果。将其视为分诊辅助——它能缩短常规故障的平均修复时间，但无法替代值班人员对新型事件的判断。
honeycombio/agent-skillFailure Modes
故障模式
| Symptom | Likely cause | Fix or check |
|---|---|---|
| Probe green but users report an outage | Probe asserts only HTTP 200, not the goal | Add content/state assertions (dashboard heading, search results count, |
| Flapping alerts (fire/resolve repeatedly) | No consecutive-failure or multi-region rule | Require 2+ consecutive failures and 2+ regions before paging |
| Probe passes locally, fails in CI/region | Region-specific outage or CDN routing | Compare per-region results; check latency-difference and |
| Probe failures with no real outage | Synthetic-account state drift (shared account) | One isolated account per concurrent probe; reset/seed stable data |
| Origin is down but probe stays green | CDN serving stale cache | Assert |
| Error budget burns with no incident | Maintenance windows counted as downtime | Suppress synthetic paging during maintenance; exclude window from budget math |
| On-call paged but can't act | Missing/empty runbook on the alert | Populate the six-line runbook; verify |
| Reliability numbers look too good | Synthetic traffic counted as real in SLO/RUM | Exclude |
| 症状 | 可能原因 | 修复或检查项 |
|---|---|---|
| 探测任务显示正常但用户报告故障 | 探测任务仅验证HTTP 200，未验证目标达成 | 添加内容/状态验证（仪表盘标题、搜索结果数量、 |
| 告警反复触发/恢复 | 未配置连续失败或多区域规则 | 触发值班告警前要求连续2次及以上失败且2个及以上区域同时失败 |
| 本地探测通过，但CI/区域探测失败 | 区域特定故障或CDN路由问题 | 对比各区域结果；检查延迟差异和 |
| 探测任务失败但无真实停机 | 合成账户状态漂移（共享账户） | 每个并发探测任务使用独立隔离账户；重置/填充稳定数据 |
| 源站停机但探测任务显示正常 | CDN提供stale缓存 | 验证 |
| 错误预算消耗但无事件记录 | 维护窗口被计入停机时间 | 维护期间屏蔽合成告警；将窗口从预算计算中排除 |
| 触发值班告警但无法采取行动 | 告警中缺少/为空的运行手册 | 填写6行运行手册；验证 |
| 可靠性指标过高 | 合成流量被计入SLO/RUM的真实流量 | 将 |
Verification
验证
Prove the monitoring works before trusting it — smallest check first.
- Probes pass against staging. Run against staging and confirm every probe passes within its declared timeout (no probe exceeds the 30s ceiling).
npx playwright test probes/ --reporter=list - A failure actually pages. Point one probe at a known-bad URL (a or a wrong path), let it run the configured consecutive-failure count, and confirm the alert routes to on-call within the SLA detection time — and resolves when you revert.
503 - Assertions are meaningful, not status-only. Temporarily break the asserted content (rename the dashboard heading on staging) and confirm the probe fails. A probe that stays green here is asserting on the wrong thing.
- Synthetic traffic is excluded. Query the analytics/RUM/billing pipeline for traffic and confirm it is filtered out.
is_synthetic
在信任监控之前先证明其有效——从最小检查开始。
- 探测任务在staging环境通过。运行针对staging环境，确认每个探测任务在声明的超时时间内通过（无探测任务超过30秒限制）。
npx playwright test probes/ --reporter=list - 故障确实会触发值班告警。将一个探测任务指向已知故障的URL（如或错误路径），让其运行配置的连续失败次数，确认告警在SLA检测时间内路由至值班人员——且恢复后告警解除。
503 - 验证项有意义，而非仅状态码。临时破坏验证内容（在staging环境修改仪表盘标题），确认探测任务失败。如果探测任务仍显示正常，说明验证的对象错误。
- 合成流量已被排除。查询分析/RUM/计费流程中的流量，确认已被过滤。
is_synthetic
Done When
完成标准
- A probe exists for every critical user journey identified in discovery (not only homepage uptime or a single health endpoint), and each asserts on the goal, not just a 2xx.
- Each probe's wall-clock budget is under 30s, with .
retries: 0 - Alerting config encodes a ≥2 consecutive-failure rule AND a multi-region confirmation rule, and probe results are tagged with +
severityso routes match.probe - The SLA dashboard shows current availability and error-budget consumption, and synthetic traffic is excluded from RUM, billing, and error-budget pipelines.
- A test alert (probe pointed at a failing URL) reaches on-call within the SLA detection time — confirmed once, not assumed.
- Every probe links a non-empty runbook with first checks, manual repro, and an escalation path.
- A scheduled review job or recurring calendar item for monitoring health exists (not just an intention to review).
- 调研阶段确定的每个关键用户旅程都有对应的探测任务（不仅是首页可用性或单个健康端点），且每个任务都验证目标达成，而非仅2xx状态码。
- 每个探测任务的耗时不超过30秒，且。
retries: 0 - 告警配置包含≥2次连续失败规则和多区域确认规则，且探测结果标记有+
severity标签以匹配路由规则。probe - SLA仪表盘显示当前可用性和错误预算消耗情况，且合成流量已从RUM、计费和错误预算流程中排除。
- 测试告警（探测任务指向故障URL）在SLA检测时间内送达值班人员——已确认，而非假设。
- 每个探测任务都链接一个非空的运行手册，包含首次检查项、手动复现步骤和升级路径。
- 存在监控健康状况的定期审查任务或 recurring 日历事项（而非仅意向）。
Reference Files (in references/
)
references/参考文件（位于references/
目录）
references/- probe-implementations.md — Runnable login-flow, API-health, and search probes, plus the environment-aware probe config.
- platforms-and-ci.md — GitHub Actions scheduling workflow, Checkly config, alert-routing rules + tagging note, and the per-probe runbook template.
- probe-implementations.md — 可运行的登录流程、API健康检查、搜索探测任务代码，以及环境感知探测配置。
- platforms-and-ci.md — GitHub Actions调度工作流、Checkly配置、告警路由规则+标签说明，以及每个探测任务的运行手册模板。
Related Skills
相关技能
- testing-in-production — safe-release techniques (flags, canary, guardrail metrics) applied during a rollout; synthetic monitoring is the schedule-driven validation that runs continuously after.
- release-readiness — a one-shot post-deploy smoke gate tied to a specific release lives there; come here for the continuous, schedule-driven version that keeps running long after the release.
- observability-driven-testing — uses production telemetry (including signals from these probes) as input to design new tests; this skill instead produces the probes and their telemetry.
- performance-testing — load tests measure capacity on demand; synthetic probes track production performance trends between those load tests.
- qa-metrics — availability, response-time percentiles, and error-budget consumption from probes feed the quality dashboards defined there.
- ci-cd-integration — go there to wire synthetic probes into a pipeline as a post-deploy verification stage.
- testing-in-production — 发布期间应用的安全发布技术（功能开关、金丝雀发布、 guardrail 指标）；合成监控是发布后持续运行的计划驱动验证。
- release-readiness — 与特定发布绑定的一次性部署后冒烟检查属于该技能；本技能是发布后长期持续运行的计划驱动版本。
- observability-driven-testing — 使用生产遥测数据（包括这些探测任务产生的信号）作为输入设计新测试；本技能则是生成探测任务及其遥测数据。
- performance-testing — 负载测试按需测量容量；合成探测任务在负载测试之间跟踪生产性能趋势。
- qa-metrics — 探测任务提供的可用性、响应时间百分位和错误预算消耗数据，为该技能定义的质量仪表盘提供数据。
- ci-cd-integration — 如需将合成探测任务接入流水线作为部署后验证阶段，请查看该技能。