Back to Details

observability-sre-triage

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

SRE Service Triage

SRE服务排查

Decide whether a service is healthy, degraded, or unhealthy, and say why. Triage is a hierarchy, not a checklist: SLOs and alerts define whether the service is failing its contract, trace-derived golden signals describe how it is failing, dependencies and infrastructure explain where the failure comes from, and logs supply the sentence you put in the incident channel. Work down the hierarchy until the evidence supports a verdict, then stop.
For authoring and tuning SLO definitions, burn-rate rules, and alert thresholds, use the observability-service-reliability skill. This skill only reads that state. For Kubernetes workload, node, or control-plane diagnosis — restart loops, OOM kill confirmation, node pressure, admission rejections, stuck rollouts — hand off to the observability-k8s-investigation skill. This skill checks whether a Kubernetes-hosted service is saturated; it does not diagnose why the pod or the node behind it is failing.
<!-- begin-partial: preamble -->
判断服务处于健康、性能下降还是异常状态，并说明原因。排查遵循层级逻辑而非 checklist：SLO和告警定义了服务是否违反服务协议，链路追踪衍生的黄金信号描述了服务故障的表现形式，依赖服务和基础设施解释了故障的来源，日志则提供可直接发布到事件频道的结论。按照层级逐步排查，直到证据足以支持结论为止，然后停止。
若需编写和调整SLO定义、burn rate规则及告警阈值，请使用observability-service-reliability技能。本技能仅读取上述配置状态。针对Kubernetes工作负载、节点或控制平面诊断（如重启循环、OOM kill确认、节点压力、准入拒绝、部署停滞等），请转交至observability-k8s-investigation技能。本技能仅检查Kubernetes托管服务是否饱和，不诊断背后的Pod或节点故障原因。
<!-- begin-partial: preamble -->

Environment Configuration

环境配置

This skill executes Elasticsearch operations through the 
elastic
CLI. If the 
elastic
CLI is not installed, tell the user what it is needed for. Do not guess credentials, call the HTTP API directly, or attempt other workarounds.
This skill references operations in HTTP-shorthand form (e.g., 
GET /
, 
GET /_cat/indices
, 
GET /{index}/_mapping
, 
GET /{index}/_settings/index.mode
, 
POST /_query
). The Operations table at the end of this document maps each shorthand to the equivalent 
elastic
CLI command — always use the CLI rather than calling the HTTP API directly.
<!-- end-partial: preamble -->
本技能通过
elastic
CLI执行Elasticsearch操作。若未安装
elastic
CLI，请告知用户其用途。请勿猜测凭证、直接调用HTTP API或尝试其他变通方法。
本技能以HTTP简写形式引用操作（例如
GET /
、
GET /_cat/indices
、
GET /{index}/_mapping
、
GET /{index}/_settings/index.mode
、
POST /_query
）。本文档末尾的操作表格将每个简写映射为等效的
elastic
CLI命令——请始终使用CLI而非直接调用HTTP API。
<!-- end-partial: preamble -->

Analysis without cluster access

无集群权限时的分析

The CLI check above gates querying the cluster — it does not gate analysis. When the user has already supplied the evidence in their question (metric values, counts, status reasons, log lines, alert payloads, configuration), reason from that evidence and deliver the conclusion.
When you genuinely do need data the user has not provided, still say what you would check and how — name the specific query, index, and field that would settle the question — and then ask for CLI setup. An answer that names the check is useful without a cluster; one that only asks for setup is not.
Everything here is expressed in ES|QL (
POST /_query
) or the Kibana Observability APIs. Do not use Query DSL, and do not use the ES|QL 
KQL
search function — express predicates natively (
WHERE service.name == "checkout"
).
上述CLI检查仅限制查询集群——不限制分析工作。当用户已在问题中提供证据（指标值、计数、状态原因、日志行、告警负载、配置）时，请基于该证据进行推理并给出结论。
当确实需要用户未提供的数据时，请说明你要检查的内容及方式——指明能解决问题的具体查询、索引和字段——然后请求用户配置CLI。明确说明检查内容的回答即使没有集群权限也有价值，仅要求配置CLI的回答则无意义。
本文档所有内容均以ES|QL（
POST /_query
）或Kibana可观测性API表述。请勿使用Query DSL，也请勿使用ES|QL的
KQL
搜索函数——请原生表达谓词（例如
WHERE service.name == "checkout"
）。

Jobs to be done

核心任务

  • Answer "is service X healthy?" with a verdict and the evidence behind it
  • Answer "why is service X slow / erroring / quiet?" by localizing the change to the service, a dependency, or its infrastructure
  • Read SLO status, burn rate, and remaining error budget during an incident
  • Determine which alerting rules currently apply to a service, including all-services rules
  • Funnel a noisy log stream down to the failures that explain the degradation
  • Identify which attribute (version, host, pod, region, route) distinguishes the failing or slow subpopulation
  • Distinguish a healthy service from a service with no telemetry
  • 回答“服务X是否健康？”，给出结论及背后的证据
  • 回答“服务X为何变慢/报错/无流量？”，将问题定位到服务自身、依赖服务或基础设施
  • 在事件期间查看SLO状态、burn rate及剩余错误预算
  • 确定当前适用于服务的告警规则，包括全局服务规则
  • 从嘈杂的日志流中筛选出能解释性能下降的故障信息
  • 识别区分失败或慢请求子群体的属性（版本、主机、Pod、区域、路由）
  • 区分健康服务与无遥测数据的服务

Output discipline

输出规范

Applies to every response produced under this skill.
  • Commit to the best-supported conclusion. When the evidence points one way, say so. Do not downgrade confidence to sound cautious — hedging on unambiguous evidence is a defect, not humility.
  • Commit to a verdict: healthy, degraded, or unhealthy, followed by the reason. A triage answer that does not name one of the three has not done the job.
  • State confidence once, in the conclusion. Do not restate it per bullet.
  • Do not speculate past the evidence. If the telemetry did not show a cause, it does not go in the answer. Name what is unknown and stop. Never offer a mechanism ("probably a GC pause", "likely a noisy neighbor") that no signal measured.
  • Report absence as absence. Zero rows means the data is missing or not collected; it never means the underlying condition is healthy. "No dependency metrics" is not "dependencies are fine".
  • Do not pad. No restating the question, no narrating which queries were run unless the result mattered, no summarizing the summary.
  • End on the finding. No trailing offers such as "want me to dig deeper?". Actionable follow-ups belong in a recommendations list, phrased as recommendations, not as questions.
适用于本技能生成的所有响应。
  • 基于最充分的证据给出结论。当证据指向明确方向时，请直接说明。请勿为了显得谨慎而降低置信度——对明确证据含糊其辞是缺陷，而非谦逊。
  • 明确给出结论：健康、性能下降或异常，随后说明原因。未明确三者之一的排查回答未完成任务。
  • 仅在结论中声明一次置信度。请勿逐条重复。
  • 请勿超出证据范围推测。若遥测数据未显示原因，则不得写入回答。说明未知内容后停止。切勿提出未被任何信号测量到的机制（例如“可能是GC停顿”、“可能是噪声邻居”）。
  • 将缺失报告为缺失。零行结果意味着数据缺失或未收集，绝不代表底层状态健康。“无依赖指标”不等于“依赖服务正常”。
  • 请勿冗余。勿重复问题，勿叙述执行了哪些查询（除非结果至关重要），勿总结总结内容。
  • 以发现内容结尾。请勿添加诸如“需要我进一步深入排查吗？”之类的后续提议。可操作的后续内容应列入建议列表，以建议形式表述，而非问题。

Signal hierarchy

信号层级

Signals disagree constantly. This ordering decides which one wins.
RankSignalAuthority
1SLO status and burn rateAuthoritative when SLOs exist. They encode the agreed definition of "good" for this service
2Active alerting rulesAuthoritative when no SLO covers the symptom. Sourced from the Alerting API
3Error rate, latency, throughputDescribes the degradation. Decisive only when nothing above it exists
4Dependency healthLocates the cause upstream or downstream; does not by itself set the verdict
5ML anomaliesDeviation from learned baseline, not from a target. Corroborates and time-bounds
6Infrastructure (CPU, memory, OOM)Explains a mechanism. A saturated pod with healthy golden signals is a risk, not an outage
7LogsExplain, never decide. Log volume is not health
Conflict rules:
  • SLO healthy, latency elevated → degraded but within error budget. The verdict follows the SLO; report the trend as a risk with the burn rate.
  • SLO violated, current-window metrics look fine → trust the SLO and check its window. SLOs are evaluated over hours or days; a 15-minute ES|QL window can look clean while the budget is already spent.
  • Alerts firing, no SLO defined → the alerts are the verdict. Resolve each rule's 
    params
    to confirm it actually targets this service before attributing it.
  • Logs noisy, golden signals flat → not degraded. High log volume without an error-rate or latency change is a logging-configuration finding, not a health finding.
  • Throughput collapsed, error rate flat → the caller stopped calling. Look upstream before blaming this service.
  • Any query returns zero rows → missing data. Say which signal is unavailable and lower the scope of the verdict accordingly; never convert silence into health.
信号时常相互矛盾。以下优先级决定以哪个信号为准。
优先级信号权威性说明
1SLO状态和burn rate当存在SLO时具有权威性。它们编码了该服务“良好状态”的约定定义
2激活的告警规则当SLO未覆盖相关症状时具有权威性。数据源为告警API
3错误率、延迟、吞吐量描述性能下降情况。仅在无更高优先级信号时起决定性作用
4依赖服务健康状况定位故障来自上游还是下游；本身不决定结论
5机器学习异常与学习到的基线（而非目标值）的偏差。用于佐证和确定时间范围
6基础设施（CPU、内存、OOM）解释故障机制。Pod饱和但黄金信号健康属于风险，而非故障
7日志用于解释，绝不用于决定结论。日志量不代表健康状况
冲突处理规则：
  • SLO健康，但延迟升高 → 性能下降但仍在错误预算范围内。结论遵循SLO；将趋势作为风险并结合burn rate报告。
  • SLO已违反，但当前窗口指标正常 → 信任SLO并检查其时间窗口。SLO基于数小时或数天评估；15分钟的ES|QL窗口可能显示正常，但错误预算已耗尽。
  • 告警触发，但未定义SLO → 告警即为结论。在归因前解析每个规则的
    params
    ，确认其确实针对该服务。
  • 日志嘈杂，但黄金信号平稳 → 未出现性能下降。日志量高但错误率或延迟无变化属于日志配置问题，而非健康问题。
  • 吞吐量骤降，但错误率平稳 → 请求方停止调用。在指责本服务前先检查上游。
  • 任何查询返回零行 → 数据缺失。说明哪个信号不可用，并相应缩小结论范围；绝不将静默视为健康。

Routing: symptom to first signal

路由：症状到首个信号

Presenting symptomPull firstReference
"Is X healthy?" / unclearSLO status, then active rules, then golden signalsslo-and-alerts.md
"X is slow"Latency percentiles versus the prior period, then dependency latencyapm-signals.md
"X is erroring" / 5xxError rate by route, then failed-transaction correlationapm-signals.md
"X is down" / no trafficThroughput, then confirm the service still ingests at allapm-signals.md
"Only some requests are bad"Subpopulation correlation over candidate attributesapm-signals.md
"An alert fired" / "the SLO is burning"Rule 
params
and SLO burn rate, then the metric the rule watches		slo-and-alerts.md
"What is in the logs?" / noisy logsThe log funnel — iterate with 
NOT
exclusions		log-investigation.md
Suspected OOM, throttling, restartsContainer CPU and memory limit utilizationapm-signals.md
"Is it saturated?" on a non-K8s hostHost CPU, memory, and load average from the hostmetrics receiverapm-signals.md
"Which downstream is hurting X?"Per-destination call volume, latency, and failure rateapm-signals.md
呈现症状优先拉取的信号参考文档
“X是否健康？” / 症状不明确SLO状态，然后是激活规则，最后是黄金信号slo-and-alerts.md
“X变慢了”延迟百分位数与之前时段对比，然后是依赖服务延迟apm-signals.md
“X报错了” / 5xx错误按路由统计错误率，然后是失败事务关联apm-signals.md
“X宕机了” / 无流量吞吐量，然后确认服务是否仍在接收流量apm-signals.md
“仅部分请求异常”候选属性对应的子群体关联对比apm-signals.md
“告警触发了” / “SLO正在消耗错误预算”规则
params
和SLO burn rate，然后是规则监控的指标		slo-and-alerts.md
“日志内容是什么？” / 日志嘈杂日志筛选——通过
NOT
排除逐步迭代		log-investigation.md
疑似OOM、限流、重启容器CPU和内存限制利用率apm-signals.md
“非K8s主机是否饱和？”主机metrics接收器提供的主机CPU、内存和负载平均值apm-signals.md
“哪个下游服务影响了X？”按目标统计调用量、延迟和失败率apm-signals.md

Data sources

数据源

OTel-native data streams, verified against Elasticsearch 9.6.0:
DataIndex pattern
Traces (spans, transactions)
traces-*.otel-*
; classic Elastic APM agent ingest also lands in 
traces*apm*
Logs
logs-*.otel-*
Raw metrics
metrics-*.otel-*
; classic APM agent ingest also lands in 
metrics*apm*
Service inventory (1m rollup)
metrics-service_summary.1m.otel-*
Transaction rollups (1m)
metrics-service_transaction.1m.otel-*
, 
metrics-transaction.1m.otel-*
Dependency rollups (1m)
metrics-service_destination.1m.otel-*
Kubernetes
metrics-kubeletstatsreceiver.otel-*
, 
metrics-k8sclusterreceiver.otel-*
, 
logs-k8seventsreceiver.otel-*
Host (VM, bare metal)
metrics-hostmetricsreceiver.otel-*
; the Elastic Agent system integration lands in 
metrics-system.*
service.name
is populated on traces, metrics, and logs, so it is the join key across all three. Use flat OTel field paths in ES|QL (
k8s.pod.name
, not 
resource.attributes.k8s.pod.name
). When analyzing OTel application metrics, the ES|QL 
TS
(time series) command gives more efficient metric queries. It is GA on Serverless; on Stack it is preview in 9.2 and GA in 9.4, so below 9.4 use 
FROM
with 
BUCKET
instead. 
TS
also rejects 
COUNT(*)
— count a field instead.
The recipes in this skill and its references are written against the OTel-native streams above. A service instrumented with the classic Elastic APM agent ships to 
traces-apm*
and 
metrics-apm*
under different field names (
transaction.duration.us
, 
event.outcome
), so these recipes return no rows for it. An empty result on a service that is otherwise clearly alive is therefore a scope boundary, not evidence of an outage: check which index family the service actually writes (
GET /_cat/indices
) and report the ingest path rather than concluding from silence.
原生OTel数据流，已针对Elasticsearch 9.6.0验证：
数据类型索引模式
链路追踪（spans、transactions）
traces-*.otel-*
；传统Elastic APM代理数据也会写入
traces*apm*
日志
logs-*.otel-*
原始指标
metrics-*.otel-*
；传统APM代理数据也会写入
metrics*apm*
服务清单（1分钟聚合）
metrics-service_summary.1m.otel-*
事务聚合（1分钟）
metrics-service_transaction.1m.otel-*
, 
metrics-transaction.1m.otel-*
依赖聚合（1分钟）
metrics-service_destination.1m.otel-*
Kubernetes
metrics-kubeletstatsreceiver.otel-*
, 
metrics-k8sclusterreceiver.otel-*
, 
logs-k8seventsreceiver.otel-*
主机（VM、裸金属）
metrics-hostmetricsreceiver.otel-*
；Elastic Agent系统集成数据写入
metrics-system.*
service.name
字段存在于链路追踪、指标和日志中，因此是三者的关联键。在ES|QL中使用扁平化的OTel字段路径（例如
k8s.pod.name
，而非
resource.attributes.k8s.pod.name
）。分析OTel应用指标时，ES|QL的
TS
（时间序列）命令可实现更高效的指标查询。该命令在Serverless环境中已GA；在Stack环境中，9.2版本为预览版，9.4版本及以上为GA版，因此9.4以下版本请使用
FROM
搭配
BUCKET
。
TS
命令也不支持
COUNT(*)
——请改为统计具体字段。
本技能及其参考文档中的示例基于上述原生OTel数据流编写。使用传统Elastic APM代理的服务会将数据写入
traces-apm*
和
metrics-apm*
，且字段名称不同（例如
transaction.duration.us
、
event.outcome
），因此这些示例对其无返回结果。若一个明显存活的服务返回空结果，这是范围边界问题，而非故障证据：请检查服务实际写入的索引族（
GET /_cat/indices
），并报告数据摄入路径，而非从静默中得出结论。

ES|QL feature availability

ES|QL功能可用性

Three features this skill uses are newer than its 8.11 base floor. Check 
GET /
before relying on them: 
build_flavor: "serverless"
means all three are available; otherwise compare 
version.number
against the Stack column. Never report "no data" when the real answer is that the query did not run — say which feature was unavailable and use the fallback.
FeatureServerlessStackLicenceUsed byFallback
FORK
GApreview 9.1-9.3, GA 9.4+anyThe log funnel, and the subpopulation comparisonRun each branch as a separate query and combine the results yourself
CATEGORIZE
GApreview 9.0, GA 9.1PlatinumMessage categorization inside the log funnelGroup by a truncated message prefix, or funnel on structured error fields
TS
GApreview 9.2, GA 9.4anyOTel application metric queries
FROM
with 
BUCKET
over the same data stream
The Platinum requirement on 
CATEGORIZE
is not a version check. A 9.6 Stack cluster on a Basic or Gold licence fails it exactly as an 8.11 cluster fails 
FORK
, and the error names the licence rather than the syntax. On Serverless the function is GA with no separate licence gate.
本技能使用的三个功能晚于其8.11基准版本。在依赖这些功能前，请先执行
GET /
检查：
build_flavor: "serverless"
表示三个功能均可用；否则请将
version.number
与Stack列对比。若查询无法运行，请勿报告“无数据”——请说明哪个功能不可用并使用替代方案。
功能ServerlessStack版本许可证使用场景替代方案
FORK
GA9.1-9.3预览版，9.4+GA版任意日志筛选和子群体对比分别运行每个分支查询，自行合并结果
CATEGORIZE
GA9.0预览版，9.1GA版铂金版日志筛选中的消息分类按截断的消息前缀分组，或基于结构化错误字段筛选
TS
GA9.2预览版，9.4GA版任意OTel应用指标查询使用
FROM
搭配
BUCKET
查询同一数据流
CATEGORIZE
功能的铂金版要求与版本无关。9.6版本的Stack集群若使用基础版或黄金版许可证，会像8.11版本集群不支持
FORK
一样报错，且错误信息会指明许可证问题而非语法问题。在Serverless环境中，该功能已GA且无额外许可证限制。

Process: triage a degraded service

流程：排查性能下降的服务

  1. Fix the service and the window. Resolve the service name and the time range from the request. Use the user's time range — do not silently assume the last hour when the complaint is historical. If no range is given, use the last hour and say so. Confirm the service actually exists in telemetry with a 
    COUNT(*) BY service.name
    over 
    traces-*.otel-*
    via 
    POST /_query
    ; if the name does not appear, resolve the ambiguity before querying further.
    Decision: which service and window every later query is scoped to. Data: distinct 
    service.name
    values in range.
  2. Read SLO status and burn rate. List SLOs with 
    GET kbn:/api/observability/slos
    and fetch the ones bound to this service with 
    GET kbn:/api/observability/slos/{id}
    . Read status, current SLI, burn rate, and remaining error budget.
    Decision: does an agreed contract exist, and is it being violated? If yes, the verdict is already determined and the remaining steps only explain it. If no SLO covers this service, say so once and fall through to step 3.
  3. Determine which alerting rules apply to this service, and which of them are firing. Call 
    GET kbn:/api/alerting/rules/_find
    with 
    per_page=100&filter=alert.attributes.enabled:true
    , paging with 
    page
    if 
    total
    exceeds what you received. Then filter the response client-side. Do not query 
    .alerts*
    indices to determine active state     — the Alerting API response is the source of truth. Fetch a rule's full definition with 
    GET kbn:/api/alerting/rule/{id}
    when its 
    params
    are needed.
    Do not narrow this call server-side. The 
    _find
    filter
    parameter is KQL over saved-object attributes, and 
    params
    is not among them — 
    filter=alert.attributes.params.serviceName:<name>
    returns zero rules on a cluster that has them. Narrowing by 
    search=apm&search_fields=tags
    , by 
    alertTypeId
    , or by 
    consumer
    is worse: it drops rules on a naming convention or a rule-type allowlist, and the rules it drops are disproportionately the all-services ones. See references/slo-and-alerts.md for the measured failure.
    From the fetched set, evaluate both rules whose 
    params.serviceName
    matches the service and rules where 
    params.serviceName
    is absent, because the latter are all-services rules that apply to it too. Read 
    execution_status.status
    on each: 
    active
    means the rule's last run produced alerts, 
    ok
    means it ran and produced none, and 
    error
    means it is not evaluating at all — a blind spot, not a pass.
    Decision: what covers this service, and is any of it currently firing? Data: rule 
    params.serviceName
    , rule type, and execution status.
  4. Check ML anomalies, if any jobs exist. List jobs with 
    GET /_ml/anomaly_detectors
    and confirm they are running with 
    GET /_ml/anomaly_detectors/_stats
    — a stopped job produces no anomalies, which is not the same as no anomaly. Pull scored records with 
    GET /_ml/anomaly_detectors/{id}/results/records
    .
    Decision: did latency, throughput, or error rate deviate from its learned baseline, and when? Use the anomaly window to narrow steps 5 and 6.
  5. Measure the golden signals. Run ES|QL over 
    traces-*.otel-*
    for throughput, latency (avg, p95, p99), and error rate, bucketed over the window and compared against the immediately preceding window of equal length. See references/apm-signals.md.
    Decision: is the service actually changed relative to itself, and in which dimension? Data: request count, latency percentiles, and failure ratio for the current and prior windows.
  6. Localize: dependencies, then subpopulation, then infrastructure.
    • Dependencies — aggregate 
      metrics-service_destination.1m.otel-*
      by 
      span.destination.service.resource
      for call volume, average latency, and failure rate. If this query returns zero rows for the service, the service is not APM-instrumented for dependencies; report insufficient dependency data and do not claim upstreams are healthy.
    • Subpopulation — when only part of the traffic is bad, compare the failure or slow rate per candidate attribute against the overall rate to find which attribute is over-represented. See references/apm-signals.md.
    • Infrastructure — read the resource attributes on the service's spans (
      k8s.pod.name
      , 
      container.id
      , 
      host.name
      ) first, then branch on what they contain. Pod and namespace attributes mean the service is Kubernetes-hosted: check 
      k8s.container.cpu_limit_utilization
      and 
      k8s.container.memory_limit_utilization
      in 
      metrics-kubeletstatsreceiver.otel-*
      . A 
      host.name
      with no pod attributes means the service runs on a VM or bare host, where every 
      k8s.*
      field is empty: check 
      system.cpu.utilization
      , 
      system.memory.utilization
      , and 
      system.cpu.load_average.1m
      in 
      metrics-hostmetricsreceiver.otel-*
      instead. OOM kills, CPU throttling, and host saturation degrade APM health directly. See references/apm-signals.md.
    • Recent change — a deploy is the most common cause of a step change. Search deploy annotations for the service with 
      GET kbn:/api/apm/services/{serviceName}/annotation/search
      over the incident window, and compare the failure or latency rate by 
      service.version
      in the subpopulation query. An annotation inside the onset window is a strong correlation; confirm it plausibly explains the symptom before attributing.
    Decision: is the cause inside this service, in something it calls, in one slice of its instances, under it, or in a change that landed?
    When the Kubernetes branch shows saturation, restarts, or an OOM kill, the mechanism is established and the remaining diagnosis — why the pod is being killed, whether the node is under pressure, whether a rollout is stuck — belongs to the observability-k8s-investigation skill. Hand off rather than continuing here.
  7. Explain with logs. Scope logs by 
    service.name
    , or by 
    trace.id
    when a specific failing trace is in hand, and run the funnel until the remaining set is small enough to read. See references/log-investigation.md. Logs confirm and articulate the cause; they do not overturn steps 2 and 3.
  8. State the verdict. Healthy, degraded, or unhealthy, with the reason and one statement of confidence, followed by recommendations. Name any signal that was unavailable.
  1. 确定服务和时间窗口。从请求中解析服务名称和时间范围。使用用户指定的时间范围——当用户反馈历史问题时，请勿默认为最近一小时。若未指定范围，请使用最近一小时并告知用户。通过
    POST /_query
    查询
    traces-*.otel-*
    的
    COUNT(*) BY service.name
    确认服务在遥测数据中存在；若服务名称未出现，请先解决歧义再进一步查询。
    决策：后续所有查询的服务和时间窗口范围。数据：指定时间范围内的
    service.name
    唯一值。
  2. 查看SLO状态和burn rate。通过
    GET kbn:/api/observability/slos
    列出SLO，并使用
    GET kbn:/api/observability/slos/{id}
    获取与该服务绑定的SLO。查看状态、当前SLI、burn rate及剩余错误预算。
    决策：是否存在约定的服务协议，且是否被违反？若是，结论已确定，后续步骤仅用于解释原因。若该服务无SLO覆盖，请说明一次并进入步骤3。
  3. 确定适用于该服务的告警规则及触发状态。调用
    GET kbn:/api/alerting/rules/_find
    ，参数为
    per_page=100&filter=alert.attributes.enabled:true
    ，若
    total
    超过返回数量，则使用
    page
    分页。然后在客户端过滤响应结果。请勿查询
    .alerts*
    索引确定告警激活状态    ——告警API响应为权威数据源。当需要规则的完整定义时，使用
    GET kbn:/api/alerting/rule/{id}
    获取。
    请勿在服务端缩小查询范围。
    _find
    的
    filter
    参数是针对保存对象属性的KQL查询，而
    params
    不在属性范围内——
    filter=alert.attributes.params.serviceName:<name>
    会返回零结果，即使集群中存在相关规则。通过
    search=apm&search_fields=tags
    、
    alertTypeId
    或
    consumer
    缩小范围更糟：会遗漏不遵循命名约定的规则，且遗漏的规则多为全局服务规则。请参考references/slo-and-alerts.md查看相关问题案例。
    在获取的规则集中，需同时评估
    params.serviceName
    匹配该服务的规则，以及
    params.serviceName
    不存在的规则（后者为适用于所有服务的全局规则）。查看每个规则的
    execution_status.status
    ：
    active
    表示规则上次运行触发了告警，
    ok
    表示运行未触发告警，
    error
    表示规则未正常评估——这是监控盲区，而非正常状态。
    决策：哪些规则适用于该服务，且哪些处于触发状态？数据：规则的
    params.serviceName
    、规则类型及执行状态。
  4. 检查机器学习异常（若存在相关任务）。通过
    GET /_ml/anomaly_detectors
    列出任务，并使用
    GET /_ml/anomaly_detectors/_stats
    确认任务正在运行——停止的任务不会产生异常，这与无异常不同。通过
    GET /_ml/anomaly_detectors/{id}/results/records
    获取评分记录。
    决策：延迟、吞吐量或错误率是否偏离学习基线，以及何时偏离？使用异常窗口缩小步骤5和6的范围。
  5. 测量黄金信号。在
    traces-*.otel-*
    上运行ES|QL查询，统计吞吐量、延迟（平均值、p95、p99）和错误率，按时间窗口分组并与前一个相同长度的窗口对比。请参考references/apm-signals.md。
    决策：服务自身状态是否发生变化，以及在哪些维度变化？数据：当前窗口和前一窗口的请求数、延迟百分位数和失败率。
  6. 定位问题：依赖服务 → 子群体 → 基础设施
    • 依赖服务 —— 在
      metrics-service_destination.1m.otel-*
      上按
      span.destination.service.resource
      聚合，统计调用量、平均延迟和失败率。若该查询针对服务返回零行，则表示服务未针对依赖服务进行APM埋点；请报告依赖数据不足，勿声称上游服务健康。
    • 子群体 —— 当仅部分流量异常时，按候选属性统计失败或慢请求率，并与整体比率对比，找出占比过高的属性。请参考references/apm-signals.md。
    • 基础设施 —— 先查看服务链路追踪的资源属性（
      k8s.pod.name
      、
      container.id
      、
      host.name
      ），再根据属性内容分支处理。若存在Pod和命名空间属性，表示服务运行在Kubernetes环境：请检查
      metrics-kubeletstatsreceiver.otel-*
      中的
      k8s.container.cpu_limit_utilization
      和
      k8s.container.memory_limit_utilization
      。若
      host.name
      存在但无Pod属性，表示服务运行在VM或裸金属主机上，所有
      k8s.*
      字段为空：请检查
      metrics-hostmetricsreceiver.otel-*
      中的
      system.cpu.utilization
      、
      system.memory.utilization
      和
      system.cpu.load_average.1m
      。OOM kill、CPU限流和主机饱和会直接影响APM健康状况。请参考references/apm-signals.md。
    • 最近变更 —— 部署是阶跃变化的最常见原因。通过
      GET kbn:/api/apm/services/{serviceName}/annotation/search
      在事件窗口内搜索服务的部署注解，并在子群体查询中按
      service.version
      对比失败率或延迟率。若注解出现在故障起始窗口内，相关性较强；在归因前请确认其能合理解释症状。
    决策：故障原因是在服务自身、其调用的服务、部分实例、底层基础设施还是最近的变更？
    若Kubernetes分支显示饱和、重启或OOM kill，机制已确定，剩余诊断（如Pod为何被杀死、节点是否存在压力、部署是否停滞）属于observability-k8s-investigation技能范畴。请转交任务而非继续排查。
  7. 用日志解释。按
    service.name
    筛选日志，若有特定失败链路则按
    trace.id
    筛选，运行筛选流程直到剩余日志量少到可直接阅读。请参考references/log-investigation.md。日志用于确认和阐述原因；不得推翻步骤2和3的结论。
  8. 给出结论。健康、性能下降或异常，说明原因并声明一次置信度，随后给出建议。说明任何不可用的信号。

Examples

示例

"Is checkout healthy?" — resolve the window, read its SLOs, then the active rules including all-services rules, then throughput, latency percentiles, and error rate over 
traces-*.otel-*
against the prior window. If the availability SLO is at 99.2% against a 99.5% target with a burn rate above 1, the verdict is unhealthy on SLO violation, and the golden signals are the explanation, not the verdict.
"Why is the frontend slow?" — compare p95 and p99 for the current window against the previous window of equal length. If service-level latency rose while per-destination latency in 
metrics-service_destination.1m.otel-*
is flat, the added time is inside the service; if one destination's average response time rose in step with it, the dependency is the cause and the frontend is a victim.
"Only some checkout requests fail" — run the subpopulation comparison: failure rate grouped by 
service.version
, 
k8s.pod.name
, 
host.name
, and 
cloud.region
alongside the overall failure rate. An attribute value whose failure rate is several times the overall rate, on a volume large enough to matter, is the correlated attribute. On live data, grouping frontend server spans by route showed a 3.8% slow rate for 
POST
against a 0.9% overall rate — a 4x lift that localizes the problem to write paths.
"The cart service logs look bad" — run the funnel over 
logs-*.otel-*
scoped to 
service.name == "cart"
: get trend, total, samples, and message categorization in one 
FORK
, then add 
NOT ... LIKE
exclusions for each dominant pattern and re-run with the full accumulated filter until fewer than 20 patterns remain. High log volume alone is not a health verdict — check the golden signals before calling the service degraded.
"Is the payment service's upstream healthy?" — query 
metrics-service_destination.1m.otel-*
for it. Zero rows means the service does not emit dependency metrics. Report that dependency data is unavailable for this service and give the verdict from the signals that do exist; do not report the upstreams as healthy.
"An alert fired on api-gateway" — fetch the enabled rules with no server-side narrowing, then match in memory on 
params.serviceName == "api-gateway"
and on rules with no 
params.serviceName
, reading 
execution_status.status
to see which are firing. Read the firing rule's threshold from 
GET kbn:/api/alerting/rule/{id}
, then query the same metric over the same window in ES|QL to confirm the rule is describing a real change rather than a threshold that is set too tight.
“checkout服务是否健康？” —— 确定时间窗口，查看其SLO，然后是激活规则（包括全局服务规则），最后是
traces-*.otel-*
上的吞吐量、延迟百分位数和错误率与前一窗口对比。若可用性SLO为99.2%，低于99.5%的目标，且burn rate高于1，则结论为因SLO违反导致异常，黄金信号为解释而非结论。
“前端为何变慢？” —— 对比当前窗口与前一相同长度窗口的p95和p99延迟。若服务级延迟升高，但
metrics-service_destination.1m.otel-*
中的按目标统计延迟平稳，则额外耗时来自服务内部；若某个目标的平均响应时间同步升高，则依赖服务是原因，前端为受害者。
“仅部分checkout请求失败” —— 运行子群体对比：按
service.version
、
k8s.pod.name
、
host.name
和
cloud.region
统计失败率，并与整体失败率对比。若某个属性值的失败率是整体的数倍，且流量足够大，则该属性为相关属性。在实际数据中，按路由分组前端服务链路追踪显示
POST
请求的慢请求率为3.8%，而整体为0.9%——4倍差异将问题定位到写入路径。
“cart服务日志看起来异常” —— 在
logs-*.otel-*
上按
service.name == "cart"
运行筛选流程：通过
FORK
一次获取趋势、总数、样本和消息分类，然后为每个主导模式添加
NOT ... LIKE
排除条件，使用累积的完整过滤器重新运行，直到剩余模式少于20种。仅日志量高并非健康结论——在判定服务性能下降前请检查黄金信号。
“payment服务的上游是否健康？” —— 查询
metrics-service_destination.1m.otel-*
。零行结果表示服务未上报依赖指标。请报告该服务的依赖数据不可用，并基于现有信号给出结论；勿称上游服务健康。
“api-gateway触发了告警” —— 在不缩小服务端范围的情况下获取启用的规则，然后在内存中匹配
params.serviceName == "api-gateway"
的规则以及无
params.serviceName
的规则，查看
execution_status.status
确定哪些规则触发。通过
GET kbn:/api/alerting/rule/{id}
获取触发规则的阈值，然后在ES|QL中查询同一时间窗口的同一指标，确认规则描述的是真实变化而非阈值设置过严。

Guidelines

指南

  • Work the signal hierarchy in order and stop when the evidence supports a verdict. Do not run every query in this document on every request.
  • Anchor to SLO status and burn rate when SLOs exist. When they do not, fall back to alerts, ML anomalies, throughput, latency, error rate, dependencies, infrastructure, and logs — and say that no SLO covers the service.
  • Use the Alerting API for active-alert state. Never query 
    .alerts*
    indices for it. Always evaluate both service-scoped rules and rules with no 
    params.serviceName
    .
  • Fetch alerting rules unnarrowed and filter client-side. 
    _find
    cannot filter on 
    params
    , tag search drops rules that do not follow a naming convention, and 
    executionStatus.status:active
    returns only rules that are firing right now — each of those silently hides the all-services rules the bullet above requires.
  • Always use the user's time range. Compare every metric against the immediately preceding window of equal length — absolute numbers without a baseline do not support a verdict.
  • Zero rows is missing data. Say which signal is unavailable rather than treating silence as a pass.
  • Scope every query by 
    service.name
    and a bounded 
    @timestamp
    range, and cap output with 
    LIMIT
    . Prefer coarse buckets when only a trend is needed.
  • Prefer 
    event.outcome == "failure"
    for failed spans; 
    status.code == "Error"
    is equivalent on OTel traces but is null on successes, so it cannot be counted directly.
  • Filter server-side traffic with 
    kind == "Server"
    when measuring a service's own throughput and latency, so client spans do not double-count.
  • Treat 
    log.level
    and 
    severity_text
    as hints, never as filters you rely on. On real OTel data most log records carry no level at all and those that do disagree on case and vocabulary (
    INFO
    , 
    Information
    , 
    SEVERE
    , 
    Normal
    ). In particular never write 
    log.level == "error"
     — the lowercase ECS vocabulary is not what the OTel SDKs emit, so it returns zero rows with no error even on a service that is logging errors, and reports the service healthy. Use the normalized numeric 
    severity_number >= 17
    if you need a severity predicate at all.
  • Logs explain; they do not decide. Never issue a verdict whose only support is log content.
  • Do not invent field names. If a field might not exist in this deployment, confirm the data stream exists with 
    GET /_resolve/index/{pattern}
    before building on it.
  • Establish where the service runs before checking saturation. Kubernetes and host telemetry share no field names, so a Kubernetes query against a VM-hosted service returns zero rows and says nothing about whether it is saturated.
  • Pass 
    --drop-null-columns
    on 
    POST /_query
    when a result is mostly empty columns. Infrastructure metrics are sparse by nature — limit utilization is absent wherever no limit is declared — and the flag collapses the noise while listing the suppressed column names under 
    all_columns
    , so nothing is hidden.
  • 按信号层级顺序排查，当证据支持结论时停止。请勿对每个请求运行本文档中的所有查询。
  • 当存在SLO时，以SLO状态和burn rate为核心。若不存在SLO，则 fallback到告警、机器学习异常、吞吐量、延迟、错误率、依赖服务、基础设施和日志——并说明该服务无SLO覆盖。
  • 使用告警API获取告警激活状态。绝不查询
    .alerts*
    索引获取该状态。请始终同时评估服务专属规则和无
    params.serviceName
    的规则。
  • 获取告警规则时不缩小范围，在客户端过滤。
    _find
    无法按
    params
    过滤，标签搜索会遗漏不遵循命名约定的规则，
    executionStatus.status:active
    仅返回当前触发的规则——上述操作均会静默隐藏前文要求的全局服务规则。
  • 始终使用用户指定的时间范围。将每个指标与前一相同长度的窗口对比——无基线的绝对数值无法支持结论。
  • 零行结果表示数据缺失。说明哪个信号不可用，而非将静默视为正常。
  • 所有查询均按
    service.name
    和有限的
    @timestamp
    范围筛选，并使用
    LIMIT
    限制输出。仅需趋势时优先使用粗粒度时间桶。
  • 统计失败链路时优先使用
    event.outcome == "failure"
    ；
    status.code == "Error"
    在OTel链路追踪中等效，但成功链路中该字段为null，无法直接统计。
  • 测量服务自身吞吐量和延迟时，通过
    kind == "Server"
    过滤服务端流量，避免客户端链路重复统计。
  • 将
    log.level
    和
    severity_text
    视为提示，绝不作为依赖的筛选条件。在实际OTel数据中，大多数日志记录无级别信息，且存在大小写和词汇差异（例如
    INFO
    、
    Information
    、
    SEVERE
    、
    Normal
    ）。尤其请勿编写
    log.level == "error"
    ——小写的ECS词汇并非OTel SDK输出内容，因此即使服务正在记录错误，该查询也会返回零行且无错误，进而错误报告服务健康。若确实需要严重性谓词，请使用标准化数值
    severity_number >= 17
    。
  • 日志用于解释；绝不用于决定结论。切勿仅基于日志内容给出结论。
  • 请勿虚构字段名称。若字段可能不存在于当前部署，请先通过
    GET /_resolve/index/{pattern}
    确认数据流存在，再基于其构建查询。
  • 在检查饱和度前确定服务运行环境。Kubernetes和主机遥测无共享字段名称，因此针对VM托管服务运行Kubernetes查询会返回零行，无法说明其是否饱和。
  • 当结果多为空列时，在
    POST /_query
    中添加
    --drop-null-columns
    参数。基础设施指标本质上是稀疏的——无限制配置的地方不存在限制利用率字段——该参数会折叠冗余内容，并在
    all_columns
    下列出被隐藏的列名，因此不会丢失信息。

Operations

操作

HTTP API (shorthand)
elastic
CLI command
GET /
elastic es info
POST /_query
elastic es esql query --format tsv --query '<esql>'
GET /_resolve/index/{pattern}
elastic es indices resolve-index --name '<pattern>'
GET /_ml/anomaly_detectors
elastic es ml get-jobs
GET /_ml/anomaly_detectors/_stats
elastic es ml get-job-stats
GET /_ml/anomaly_detectors/{id}/results/records
elastic es ml get-records --job-id '<id>'
GET kbn:/api/observability/slos
elastic kb slo find-slos-op --space-id '<space>' --kql-query '<kql>'
GET kbn:/api/observability/slos/{id}
elastic kb slo get-slo-op --space-id '<space>' --slo-id '<id>'
GET kbn:/api/alerting/rules/_find
elastic kb alerting get-alerting-rules-find --filter '<filter>'
GET kbn:/api/alerting/rule/{id}
elastic kb alerting get-alerting-rule-id --id '<id>'
GET kbn:/api/apm/services/{serviceName}/annotation/search
elastic kb apm-annotations get-annotation --service-name '<service>' --environment '<env>' --start '<iso8601>' --end '<iso8601>'
The SLO find command takes a KQL query string because that is the API's contract; it is not an exception to the ES|QL rule for data queries.
The annotation search route rejects a request that omits 
environment
, so pass 
ENVIRONMENT_ALL
when the service's environment is not known. Only the search direction is in scope: this skill is read-only, so the companion create-annotation operation is deliberately not bound.
HTTP API（简写）
elastic
CLI命令
GET /
elastic es info
POST /_query
elastic es esql query --format tsv --query '<esql>'
GET /_resolve/index/{pattern}
elastic es indices resolve-index --name '<pattern>'
GET /_ml/anomaly_detectors
elastic es ml get-jobs
GET /_ml/anomaly_detectors/_stats
elastic es ml get-job-stats
GET /_ml/anomaly_detectors/{id}/results/records
elastic es ml get-records --job-id '<id>'
GET kbn:/api/observability/slos
elastic kb slo find-slos-op --space-id '<space>' --kql-query '<kql>'
GET kbn:/api/observability/slos/{id}
elastic kb slo get-slo-op --space-id '<space>' --slo-id '<id>'
GET kbn:/api/alerting/rules/_find
elastic kb alerting get-alerting-rules-find --filter '<filter>'
GET kbn:/api/alerting/rule/{id}
elastic kb alerting get-alerting-rule-id --id '<id>'
GET kbn:/api/apm/services/{serviceName}/annotation/search
elastic kb apm-annotations get-annotation --service-name '<service>' --environment '<env>' --start '<iso8601>' --end '<iso8601>'
SLO查询命令接受KQL查询字符串，这是API约定；这不属于数据查询需使用ES|QL规则的例外情况。
注解搜索路由会拒绝未提供
environment
的请求，因此当服务环境未知时，请传入
ENVIRONMENT_ALL
。仅搜索方向在本技能范围内：本技能为只读，因此刻意未绑定对应的创建注解操作。