Back to Details

observability

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Observability

Observability

Decide what to measure so a system can be seen, alerted on, and debugged in production. Getting this wrong is failure mode #6 — ignoring failure: a design that works on the whiteboard but goes dark under load, where the first signal of an outage is a user complaint instead of a page.
确定需要测量的内容，以便在生产环境中能够查看、告警和调试系统。在这方面出错属于第6种故障模式——忽略故障：这种设计在白板上可行，但在负载下会“失联”，系统中断的第一个信号不是告警通知，而是用户投诉。

When to reach for this

何时使用本Skill

Any production design needs an answer to "how would we know this broke, and how fast?" Reach for this when defining what the system measures, what pages a human, what an acceptable level of service is (SLO), or how a request is traced across services. It is the design move that makes every other block's stress section real — you cannot mitigate a thundering herd or a hot shard you can't see.
任何生产环境设计都需要回答“我们如何知道系统故障，以及故障发生的速度有多快？”这个问题。当定义系统需要测量的内容、人员的告警通知规则、可接受的服务水平（SLO），或是请求跨服务的追踪方式时，都可以使用本Skill。它能让其他模块的压力测试部分变得真实——你无法缓解一个看不到的“惊群效应”或热点分片问题。

When NOT to

何时不使用

Do not build a full metrics-logs-traces stack for a prototype or an internal tool with no users to disappoint (YAGNI) — a health check and error logging are enough. Do not invent SLOs nobody will defend, or wire alerts before knowing the symptom that matters; an alert with no owner and no runbook is noise that trains the team to ignore pages. This skill owns what to measure and alert on; the high-volume log pipeline (collect → buffer → ship → index → retain) lives in 
distributed-logging
— summarize and link, don't rebuild it here.
不要为原型或无外部用户的内部工具搭建完整的metrics-logs-traces栈（遵循YAGNI原则）——健康检查和错误日志就足够了。不要制定无人维护的SLO，也不要在明确关键症状前配置告警；没有负责人和运行手册的告警只是噪音，会让团队养成忽略告警通知的习惯。本Skill负责定义需要测量和告警的内容；而高容量日志管道（收集→缓冲→传输→索引→留存）属于
distributed-logging
模块——只需总结并链接到该模块，无需在此重复构建。

Clarify first

先明确以下问题

  • What is "healthy" from a user's view? The symptom that defines a bad experience (slow checkout, failed upload) — alerts target this, not CPU.
  • SLO target and window? e.g. 99.9% of requests < 300 ms over 30 days. This sets the error budget and the alert thresholds. (→ 
    back-of-the-envelope
    for the nines.)
  • Request volume and cardinality? QPS drives metric/trace sample rates; high-cardinality labels (user ID, URL) blow up a metrics store.
  • Is this request-driven or resource-driven? Picks RED (services) vs USE (CPU/disk/queues) as the measurement frame.
  • Multi-service request path? If a request crosses services, tracing earns its keep; a single service may not need it yet.
  • 从用户视角看，什么是“健康”状态？ 定义糟糕体验的症状（如结账缓慢、上传失败）——告警应针对这些症状，而非CPU使用率等指标。
  • SLO目标和时间窗口？ 例如：30天内99.9%的请求响应时间<300毫秒。这会设定错误预算和告警阈值。（可参考
    back-of-the-envelope
    模块了解“几个9”的计算方法。）
  • 请求量和基数？ QPS（每秒请求数）决定了metric/trace的采样率；高基数标签（如用户ID、URL）会撑爆指标存储。
  • 系统是请求驱动还是资源驱动？ 这会决定选择RED（服务端）还是USE（CPU/磁盘/队列）作为测量框架。
  • 请求路径是否跨多个服务？ 如果请求跨服务，追踪就有存在的价值；单一服务可能暂时不需要追踪。

The options

可选方案

The three pillars (complementary, not either/or):
  • Metrics — cheap numeric time series (counters, gauges, histograms). Use for dashboards, trend analysis, and alerting — the always-on signal.
  • Logs — discrete events with context. Use for debugging the specific failure after an alert fires. (The pipeline that moves them is 
    distributed-logging
    .)
  • Traces — one request's journey across services with timing per hop. Use to find which service or dependency is the latency/error source.
What to measure (pick a frame per component):
  • RED (request-driven services): Rate, Errors, Duration. Use for APIs, web tiers, anything serving requests.
  • USE (resources): Utilization, Saturation, Errors. Use for CPU, memory, disk, connection pools, queues.
  • Four golden signals (latency, traffic, errors, saturation) — the superset; use as the default service dashboard.
Health checks (the signal, consumed by 
load-balancing
/orchestrator):
  • Liveness — is the process alive / not deadlocked? Failure ⇒ restart. Keep it cheap; don't check dependencies. Use to recover stuck processes.
  • Readiness — can this instance serve traffic now (deps reachable, warmup done)? Failure ⇒ pull from rotation, don't restart. Use to gate cold/struggling instances.
Alerting:
  • Symptom-based (SLO burn) — page when the user-facing SLO is at risk. Use as the default; it is actionable and low-noise.
  • Cause-based (resource thresholds) — ticket/warn on CPU, disk, saturation. Use for capacity planning, not paging.
三大支柱（互补而非互斥）：
  • Metrics —— 低成本的数值时间序列（计数器、仪表盘、直方图）。用于仪表板展示、趋势分析和告警——持续在线的信号。
  • Logs —— 带有上下文的离散事件。用于告警触发后调试具体故障。（传输日志的管道属于
    distributed-logging
    模块。）
  • Traces —— 单个请求跨服务的完整链路，包含每个节点的耗时。用于定位延迟/错误来源的服务或依赖项。
测量内容（为每个组件选择一个框架）：
  • RED（请求驱动型服务）：Rate（请求率）、Errors（错误数）、Duration（耗时）。适用于API、Web层等处理请求的服务。
  • USE（资源型组件）：Utilization（使用率）、Saturation（饱和度）、Errors（错误数）。适用于CPU、内存、磁盘、连接池、队列等资源。
  • 四大黄金信号（延迟、流量、错误、饱和度）——上述框架的超集；可作为服务仪表板的默认方案。
健康检查（信号由
load-balancing
/编排系统消费）：
  • Liveness —— 进程是否存活/未死锁？检测失败⇒重启进程。检查逻辑要轻量；不要检查依赖项。用于恢复卡住的进程。
  • Readiness —— 该实例当前是否可以处理流量（依赖项可达、预热完成）？检测失败⇒从流量池中移除，不重启。用于隔离冷启动或状态异常的实例。
告警：
  • 基于症状（SLO消耗） —— 当用户侧SLO面临风险时触发告警通知。作为默认方案；可执行性强且噪音低。
  • 基于原因（资源阈值） —— 当CPU、磁盘、饱和度等指标达到阈值时生成工单/警告。用于容量规划，而非触发告警通知。

Trade-offs

权衡取舍

OptionWhat it solvesWhat it worsensChange it when
MetricsCheap, always-on alerting + trendsNo per-request detail; high-cardinality labels explode storage/costYou need to debug a specific request → add traces/logs
LogsRich context for debugging an incidentVolume + cost; needs the 
distributed-logging
pipeline to scale		Volume is unmanageable → sample, or shift detail to metrics
TracesPinpoints the slow/failing hop across servicesInstrumentation effort; sampling needed at high QPSSingle-service or low volume → defer; metrics suffice
REDRight frame for request servicesMisses resource exhaustion that hasn't yet hurt requestsComponent is a resource (queue, disk) → use USE
USECatches saturation before it hurts usersResource-centric, not user-centric; can be noisyAlerting on it → switch to symptom/SLO-based
Liveness checkRecovers deadlocked processesToo aggressive ⇒ restart loops, mask real bugsRestarts hide a crash-loop → fix readiness/root cause
Readiness checkKeeps cold/broken instances out of rotationFlapping if it checks flaky deps ⇒ capacity yo-yoAll instances fail readiness together → it's a dep outage
SLO/error budgetTies alerting + release pace to user painEffort to define + defend; wrong SLO misleadsBudget never burns (too loose) or always burns (too tight)
Symptom alertingLow-noise, actionable pagesSlightly slower to localize root causeNeed faster localization → add cause-based tickets (not pages)
方案解决的问题带来的问题何时调整
Metrics低成本、持续在线的告警+趋势分析无单请求细节；高基数标签会激增存储/成本需要调试特定请求时→添加traces/logs
Logs为事件调试提供丰富上下文数据量+成本；需要
distributed-logging
管道才能规模化		数据量无法管理时→采样，或将细节转移到metrics
Traces跨服务定位缓慢/故障节点instrumentation成本高；高QPS时需要采样单一服务或低流量场景→延后使用；metrics足够
RED适合请求驱动型服务的框架无法捕捉尚未影响请求的资源耗尽问题组件是资源（队列、磁盘）时→使用USE
USE在影响用户前捕捉资源饱和问题以资源为中心，而非用户为中心；可能产生噪音基于USE告警时→切换为基于症状/SLO的告警
Liveness检查恢复死锁进程检查过于激进⇒重启循环，掩盖真实bug重启隐藏崩溃循环时→修复readiness/根本原因
Readiness检查将冷启动/故障实例排除在流量池外若检查不稳定依赖项会导致实例频繁上下线⇒容量波动所有实例都无法通过readiness检查时→问题出在依赖项上
SLO/错误预算将告警+发布节奏与用户体验关联定义和维护成本高；错误的SLO会误导决策预算从未消耗（过于宽松）或总是耗尽（过于严格）时
基于症状的告警低噪音、可执行的告警通知定位根本原因的速度稍慢需要更快定位时→添加基于原因的工单（而非告警通知）

Behavior under stress

压力下的表现

Observability is most needed exactly when it's most likely to break or mislead.
  • The monitoring amplifies the outage. Synchronous logging on the request path turns a slow log backend into request latency. Per-request trace export with no sampling melts the collector under a spike. Keep telemetry async and sampled so it degrades the signal, never the service.
  • Health-check stampede. Aggressive liveness probes restart instances that are merely slow, and readiness flaps yank capacity mid-spike — making the spike worse. The gating/recovery behavior is owned by 
    load-balancing
    and 
    resilience-failure
    ; this block owns defining a stable signal (sane thresholds, consecutive-failure counts, deps in readiness not liveness).
  • Cardinality blow-up. A label like 
    user_id
    or raw URL multiplies time series into the millions and OOMs the metrics store under load — drop or bucket high-cardinality dimensions.
  • Alert storm. One root cause (a DB outage) fires fifty correlated alerts; the on-call drowns. Alert on the user-facing symptom; group/inhibit downstream alerts.
  • Blind spot on recovery. After an incident, dashboards must show whether the fix worked and whether recovery traffic is overwhelming a cold tier.
Monitor: the four golden signals per service (rate, errors, latency p50/p95/p99, saturation), SLO error-budget burn rate, healthy-host count and probe flap rate, and the telemetry pipeline's own lag/drop rate (watch the watcher).
可观测性恰恰在最需要它的时候最容易失效或产生误导。
  • 监控放大故障：请求路径上的同步日志会将缓慢的日志后端转化为请求延迟。无采样的单请求trace导出会在流量峰值时压垮收集器。要确保遥测数据是异步且经过采样的，这样即使信号降级，服务也不会受影响。
  • 健康检查踩踏：过于激进的liveness探针会重启只是响应缓慢的实例，而readiness的频繁上下线会在流量峰值时移除容量——让峰值问题更严重。限流/恢复逻辑由
    load-balancing
    和
    resilience-failure
    模块负责；本模块负责定义稳定的信号（合理的阈值、连续失败次数、readiness中检查依赖项而非liveness）。
  • 基数爆炸：
    user_id
    或原始URL这类标签会将时间序列数据激增到数百万条，在负载下导致指标存储OOM——要删除或归并高基数维度。
  • 告警风暴：一个根本原因（如数据库中断）会触发五十个相关告警；值班人员会被淹没。应针对用户侧症状告警；对下游告警进行分组/抑制。
  • 恢复阶段的盲区：事件结束后，仪表板必须显示修复是否有效，以及恢复流量是否会压垮冷启动的服务层。
需监控的内容：每个服务的四大黄金信号（请求率、错误数、延迟p50/p95/p99、饱和度）、SLO错误预算消耗率、健康主机数量和探针波动率，以及遥测管道自身的延迟/丢包率（监控监控系统本身）。

How to apply

如何应用

  1. Clarify the inputs — pin the user-facing symptom, the SLO target + window, request volume/cardinality, and whether the path is multi-service (see Clarify first). No SLO yet? Define the symptom first; the number follows.
  2. Pick the pillars and frame from the trade-off table — metrics for alerting always; RED for services and USE for resources; add traces only when a request crosses services or volume justifies it.
  3. Set the key knobs — define liveness vs readiness endpoints (deps in readiness, not liveness), consecutive-failure thresholds, trace sample rate, and the SLO + error-budget policy. Drop high-cardinality labels up front.
  4. Stress-test the choice — walk Behavior under stress: confirm telemetry is async/sampled, probes won't stampede, cardinality is bounded, and alerts group by root cause so one outage isn't fifty pages.
  5. Size it with numbers — sanity-check metric series count and retention, trace/log sample volume against the QPS, and that the SLO's nines match the architecture's redundancy (→ Numbers that matter).
  6. Pick a provider — default to the generic recipe; open a provider file only if the user named a cloud (see Choosing a provider).
  1. 明确输入条件 —— 确定用户侧症状、SLO目标+时间窗口、请求量/基数，以及请求路径是否跨服务（参考“先明确以下问题”部分）。还没有SLO？先定义症状；数值可以后续确定。
  2. 从权衡表中选择支柱和框架 —— Metrics始终用于告警；服务端用RED，资源用USE；只有当请求跨服务或流量规模需要时才添加traces。
  3. 设置关键参数 —— 定义liveness和readiness端点（在readiness中检查依赖项，而非liveness）、连续失败阈值、trace采样率，以及SLO+错误预算策略。提前删除高基数标签。
  4. 压力测试选择方案 —— 回顾“压力下的表现”部分：确认遥测是异步/采样的、探针不会导致踩踏、基数可控，且告警按根本原因分组，避免一次故障触发五十条告警通知。
  5. 用数值量化规模 —— 验证指标序列数量和留存时间、trace/log采样量与QPS的匹配度，以及SLO的“几个9”与架构冗余度是否一致（参考Numbers that matter部分）。
  6. 选择供应商 —— 默认使用上述通用方案（Prometheus + Grafana + Loki + Jaeger，由OpenTelemetry整合）。如果用户指定了云厂商，请查看
    references/providers/<provider>.md
    获取托管服务映射、配额/限制和厂商特定的权衡点。如果该厂商没有对应的文档，通用方案即为答案。

Dos and don'ts

注意事项

Do
  • Alert on user-facing symptoms (SLO burn), and make every page actionable with a runbook.
  • Put dependency checks in readiness and keep liveness cheap and dependency-free.
  • Emit telemetry asynchronously and sampled so it never adds latency or melts the collector.
  • Define an SLO + error budget you will actually defend, and tie release pace to it.
  • Standardize a per-service dashboard on the four golden signals.
Don't
  • Don't page on causes (CPU > 80%) — ticket those; pages are for user pain.
  • Don't use high-cardinality labels (user ID, raw URL) as metric dimensions.
  • Don't check downstream dependencies in liveness — you'll restart-loop the fleet.
  • Don't build a full stack for a prototype, or wire alerts no one owns (YAGNI / alert fatigue).
  • Don't reimplement the log pipeline here — that's 
    distributed-logging
    .
建议
  • 针对用户侧症状（SLO消耗）告警，并为每条告警通知配备可执行的运行手册。
  • 将依赖项检查放在readiness中，保持liveness轻量且不检查依赖项。
  • 异步且采样式地发送遥测数据，确保不会增加请求延迟或压垮收集器。
  • 定义一个实际会维护的SLO+错误预算，并将发布节奏与它关联。
  • 标准化每个服务的四大黄金信号仪表板。
禁止
  • 不要针对原因（如CPU>80%）触发告警通知——这类情况生成工单即可；告警通知只用于用户体验受损的场景。
  • 不要将高基数标签（如用户ID、原始URL）作为metric维度。
  • 不要在liveness中检查下游依赖项——这会导致整个集群进入重启循环。
  • 不要为原型搭建完整的可观测栈，也不要配置无人负责的告警（遵循YAGNI原则/避免告警疲劳）。
  • 不要在此重新实现日志管道——那是
    distributed-logging
    模块的职责。

Numbers that matter

关键数值

The SLO sets everything else: 99.9% over 30 days allows ~43 minutes of budget; if the architecture's redundancy can't deliver those nines, the SLO is fiction (→ 
back-of-the-envelope
for the availability-nines table). Alert on p95/p99, not averages — averages hide tail pain. Sample traces (often 1–10% at high QPS) so cost scales sub-linearly with traffic. Watch metric cardinality (series ≈ product of label values): one unbounded label can mean millions of series. Don't restate the latency/QPS/nines tables — they live in 
back-of-the-envelope
.
SLO决定了所有其他参数：30天内99.9%的可用性允许约43分钟的错误预算；如果架构冗余度无法达到这个“几个9”的水平，那么该SLO就是空谈（参考
back-of-the-envelope
模块的可用性“几个9”对照表）。告警应针对p95/p99延迟，而非平均值——平均值会掩盖尾部延迟问题。对traces进行采样（高QPS场景通常采样1–10%），让成本随流量呈次线性增长。关注metric的基数（序列数≈标签值的乘积）：一个无界标签可能导致数百万条序列。无需重复延迟/QPS/“几个9”的表格——这些内容在
back-of-the-envelope
模块中。

Interface sketch

接口示例

Telemetry has contracts worth pinning down. A metric: name + label set + type (
http_requests_total{service,method,status}
counter) — labels are low-cardinality. A health endpoint: 
GET /livez
→ 200/503 (process only), 
GET /readyz
→ 200/503 (deps + warmup), with version/build info in the body for debugging. A trace context: a 
trace_id
+ 
span_id
propagated on every inbound/outbound call (e.g. W3C 
traceparent
header) so traces and logs correlate.
遥测数据有值得明确的契约。一个metric：名称+标签集+类型（如
http_requests_total{service,method,status}
计数器）——标签需为低基数。一个健康检查端点：
GET /livez
→ 返回200/503（仅检查进程状态），
GET /readyz
→ 返回200/503（检查依赖项+预热状态），响应体中包含版本/构建信息用于调试。一个trace上下文：
trace_id
+ 
span_id
在每个入站/出站调用中传播（如W3C 
traceparent
头），以便traces和logs关联。

Choosing a provider

选择供应商

Default to the generic recipe above (Prometheus + Grafana + Loki + Jaeger, glued by OpenTelemetry). If the user names a cloud, read 
references/providers/<provider>.md
for the managed-service mapping, quotas/limits, and provider-specific trade-offs. If no file exists for that provider, the generic recipe is the answer.
默认使用上述通用方案（Prometheus + Grafana + Loki + Jaeger，由OpenTelemetry整合）。如果用户指定了云厂商，请查看
references/providers/<provider>.md
获取托管服务映射、配额/限制和厂商特定的权衡点。如果该厂商没有对应的文档，通用方案即为答案。

Diagram

图表

To visualize the telemetry flow (app → OpenTelemetry SDK → collector → metrics / logs / traces backends → dashboard + alertmanager) or the alert-to-page path, use the in-plugin 
architecture-diagram
skill. Telemetry export uses dashed arrows to show it is off the request path.
要可视化遥测流（应用→OpenTelemetry SDK→收集器→metrics/logs/traces后端→仪表板+alertmanager）或告警到通知的路径，请使用插件内的
architecture-diagram
Skill。遥测导出用虚线箭头表示其不在请求路径上。

Related building blocks

相关模块

  • distributed-logging
    — owned-concept lives in there: the high-volume log pipeline (collect → buffer → ship → index → retain). This skill decides what to log and alert on; that skill decides how to move and store it at scale.
  • resilience-failure
    — pairs with this: alerts and SLO burn trigger graceful degradation and circuit breaking, and the health-check signal defined here is the input its failover and 
    load-balancing
    health-gating consume.
  • scaling-evolution
    — feeds into it: saturation metrics and the golden signals reveal the next bottleneck that justifies the next scaling step.
  • back-of-the-envelope
    — feeds into this skill: it supplies the availability nines and QPS that turn an SLO and sample rates into real numbers.
  • system-design
    — owned-concept lives in the orchestrator: the reasoning loop, the trade-off method, and the ten failure modes (this block defends #6).
  • distributed-logging
    —— 相关概念归属该模块：高容量日志管道（收集→缓冲→传输→索引→留存）。本Skill决定需要记录和告警的内容；该Skill决定如何规模化地传输和存储这些数据。
  • resilience-failure
    —— 与本Skill配合使用：告警和SLO消耗会触发优雅降级和熔断，本模块定义的健康检查信号是其故障转移和
    load-balancing
    健康限流的输入。
  • scaling-evolution
    —— 为本Skill提供输入：饱和度指标和黄金信号会揭示下一个瓶颈，证明下一步扩容的合理性。
  • back-of-the-envelope
    —— 为本Skill提供支持：它提供可用性“几个9”和QPS计算方法，将SLO和采样率转化为实际数值。
  • system-design
    —— 相关概念归属编排器：推理循环、权衡方法和十种故障模式（本模块防御第6种）。

References

参考资料

  • references/deep-dive.md
     — pillar mechanics (histograms vs gauges, push vs pull, sampling), SLI/SLO/error-budget math and burn-rate alerts, RED/USE in detail, health-check tuning, and alert design. Read when designing the observability layer in depth.
  • references/providers/{generic,aws,azure,gcp}.md
     — service mappings, limits, and pitfalls per environment.
  • references/deep-dive.md
     —— 支柱原理（直方图vs仪表盘、推送vs拉取、采样）、SLI/SLO/错误预算的数学计算和消耗率告警、RED/USE详细说明、健康检查调优和告警设计。深入设计可观测性层时阅读。
  • references/providers/{generic,aws,azure,gcp}.md
     —— 各环境下的服务映射、限制和陷阱。