gke-alert-configuration

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

GKE Alert Configuration

GKE告警配置

This skill provides guidelines and best practices for creating robust, high-signal alerting policies for Google Kubernetes Engine workloads using Google Cloud Managed Service for Prometheus and Terraform. It ensures comprehensive coverage of the 4 Golden Signals and key cluster health metrics while minimizing alert noise.

本技能提供使用Google Cloud Managed Service for Prometheus和Terraform为Google Kubernetes Engine工作负载创建可靠、高信号告警策略的指南与最佳实践。它确保全面覆盖四大黄金信号和关键集群健康指标，同时最大限度减少告警噪音。

Critical Rules

关键规则

Negative Triggers and Scope Redirection for Non-GKE Standalone Runtimes:
- This skill is strictly scoped to Google Kubernetes Engine (GKE) workloads, clusters, and services using PromQL and Google Cloud Managed Service for Prometheus.
- Do not use for non-GKE compute runtimes, such as standalone Compute Engine virtual machines or standalone Cloud Run services without GKE.
- STOP AND RESPOND DIRECTLY (Do Not Edit Files): When the user requests alert configuration for non-GKE compute infrastructure:
  1. Do not write, create, edit, or validate any Terraform files on disk.
  2. Immediately stop and respond directly to the user in chat:
    - Explicitly Clarify Out-of-Scope: State clearly that standalone Compute Engine virtual machine monitoring or standalone Cloud Run monitoring is out of scope for this GKE-specific PromQL alerting skill, which is designed specifically for GKE workloads using Google Cloud Managed Service for Prometheus and PromQL.
    - Do Not Generate GKE PromQL Alerts: Do not create or generate Kubernetes PromQL alert policies or fabricate Kubernetes container, pod, or node resources for non-GKE infrastructure.
    - Redirect the User: Guide and redirect the user to standard Google Cloud Monitoring metrics, such as
      compute.googleapis.com/instance/cpu/utilization
      or
      run.googleapis.com/request_latencies
      , using standard
      google_monitoring_alert_policy
      with
      condition_threshold
      or MQL, or recommend the relevant specialized Cloud observability skill.
Mandatory
kube-state-metrics
(KSM) Cost Guardrail:
- Deploying open-source
```
kube-state-metrics
```
  in Google Cloud Managed Service for Prometheus incurs billable metric ingestion costs.
- STOP AND ASK PERMISSION FIRST (Do Not Edit Files): When a requested alert rule relies on Tier 2 KSM metrics (such as
```
kube_cronjob_*
```
  ,
```
kube_pod_status_phase
```
  ,
```
kube_persistentvolume_*
```
  ,
```
kube_deployment_*
```
  ,
```
kube_statefulset_*
```
  ,
```
kube_job_*
```
  , or
```
kube_daemonset_*
```
  ), do not write, create, edit, or validate any Terraform files or generate alert policies before obtaining user approval.
- Instead, you must immediately stop and respond directly to the user to:
  1. Alert the user that the requested alert requires
```
kube-state-metrics
```
    .
  2. Explain the cost impact: Detail that
```
kube-state-metrics
```
    incurs billable sample ingestion costs in Google Cloud Managed Service for Prometheus.
  3. Ask for explicit permission: Ask the user for explicit permission before assuming, enabling, or generating KSM-dependent alert configurations.
  4. Recommend filtering or allowlisting: Suggest and recommend filtering or allowlisting only the specific required metrics, such as using a
```
PodMonitoring
```
    resource with
```
metricRelabeling
```
    (
```
action: keep
```
    ) or KSM
```
--metric-allowlist
```
    to minimize ingestion costs. Provide a concrete allowlist example.
- Always prefer Non-KSM Native Alternatives (Tier 1 cAdvisor or native GKE metrics documented in metrics_and_alerts_catalog.md) whenever possible, such as using
```
container_memory_working_set_bytes
```
  and
```
container_spec_memory_limit_bytes
```
  instead of
```
kube_pod_container_resource_limits
```
  .
- Explicit Tier and Cost Surcharge Identification in Response: In every response where you generate or recommend an alerting policy, you must explicitly state its classification tier and cost impact:
  - Tier 1 native or standard metric (GKE built-in metrics, cAdvisor
```
container_*
```
    , kubelet volume stats, kubelet node conditions, and control-plane metrics; see metrics_and_alerts_catalog.md): State that it is a Tier 1 native or standard metric with zero KSM cost surcharge.
  - Tier 2 KSM metric: State that it is a Tier 2 KSM-dependent metric and follow the permission and allowlisting guardrail above. (Tip: Generally, metrics with the
    kube_
    prefix that represent resource state or metadata belong to Tier 2).
Plan-Validate-Execute Loop for Approved File Edits: When modifying, adding, or merging approved Terraform files on disk in a workspace, follow the three-phase workflow:
1. Plan: Draft a structured change plan (
```
changes.json
```
  ) containing proposed policy resource names, PromQL expressions, grouping labels, and durations.
2. Validate: Run the pre-edit validation script (
```
python3 scripts/validate_config.py --plan changes.json
```
  ) to verify PromQL grammar, lookback windows, duration rules, and ensure no duplicate signals exist.
3. Execute: After the plan passes validation, apply or merge changes in-place into the target Terraform configuration (
```
alerts.tf
```
  ).
4. Note: When answering questions or providing Terraform snippets directly in chat where no disk modification is requested, output the complete, valid Terraform HCL block in your response.
Configure the 4 Golden Signals and Cluster Health: Always ensure the target Kubernetes workload or service has the following alerting coverage:
1. Latency (P95 response time)
2. Errors (Multi-Window Multi-Burn-Rate SLO alerts, such as Fast Burn 1 hour / 5 minutes with factor 14.4, Slow Burn 6 hours / 30 minutes with factor 6.0; do not use simple static ratios)
3. Traffic (Sudden drop or complete metric disappearance using
```
absent()
```
  or
```
default 0
```
  syntax, or overload spikes)
4. Saturation (Memory Limit Utilization Only): When describing or configuring alert policies for a cluster or project, include ONLY Memory Saturation (
```
container_memory_working_set_bytes
```
  /
```
container_spec_memory_limit_bytes
```
  ). Do NOT include CPU saturation alerts or list
```
container_cpu_usage_seconds_total
```
  as an alert metric because CPU is compressible and throttled by CFS quotas rather than causing uncompressible fatal termination (OOM).
5. Cluster Health (Pod CrashLooping, Node NotReady)
PromQL Only (Managed Prometheus): You must use
```
condition_prometheus_query_language
```
with PromQL. Do NOT use MQL or standard
```
condition_threshold
```
unless explicitly requested. Google Cloud Managed Service for Prometheus is the standard telemetry ingestion path for GKE.
Terraform Only: Write the generated observability configuration ONLY as Terraform (
```
.tf
```
) files, such as
```
alerts.tf
```
and
```
variables.tf
```
.
Dynamic Multi-Resource Alerting (No Hardcoding): You must not hardcode specific pod names, node names, or service names in alerting conditions unless explicitly requested. Alerting policies must be written to cover resources dynamically:
- Always use grouping aggregations (
```
by (cluster, namespace, service, pod, container)
```
  ) instead of filtering to a single instance. This allows a single alert policy to dynamically track each service or pod separately.
- Always declare and use Terraform variables for
```
project_id
```
  ,
```
cluster_name
```
  , and
```
namespace
```
  (
```
var.project_id
```
  ,
```
var.cluster_name
```
  ,
```
var.namespace
```
  ) to make the configuration reusable across environments. Always define these variables in
```
variables.tf
```
  (or within the configuration) and reference all three in policies or PromQL label matchers.
No Redundant Duration Windows on Lookbacks:
- When PromQL expressions already use an aggregated lookback window (such as
```
increase(...[15m]) > 3
```
  or multi-window SLO burn rates), the query time window already smooths out transient spikes.
- Adding a Terraform duration on top of a PromQL lookback window increases the Mean Time to Detect (MTTD) without providing additional smoothing benefits.
- In these cases, set Terraform
```
duration = "0s"
```
  (or
```
"60s"
```
  ). Do not enforce
```
duration = "300s"
```
  on top of
```
[15m]
```
  , which delays critical crashloop alerts by up to 20 minutes total (15 minutes + 5 minutes).
- Use
```
duration = "300s"
```
  only on instantaneous gauge conditions, such as
```
kube_node_status_condition == 0
```
  .
Use SLO Burn Rates Instead of Simple Ratios: For error rate alerting, always generate Multi-Window Multi-Burn-Rate (MWMBR) SLO alerts (such as 14.4x burn rate over 1 hour and 5 minute windows for a 99% SLO) rather than simple error rate ratios (
```
rate(5xx)/rate(total) > 0.05
```
), which produce excessive false alarms on low traffic.
Robust Traffic Drop Detection (
absent()
/
default 0
): When monitoring for traffic drops to zero, do not use
```
rate(...) == 0
```
alone because Prometheus time series disappear completely when no requests occur (evaluating to an empty vector rather than 0). Use
```
default 0
```
syntax, such as
```
sum(rate(...[5m])) default 0 == 0
```
, or
```
absent(...) == 1
```
.
Notification Channels: By default, never configure any notification channels without user input. If the user explicitly provides a notification channel, configure the alerts to use it. Otherwise, you must prompt the user in your response to ask if they would like to configure one.
Consult GKE Metrics and Open-Source Alerts Catalog: When designing or generating evaluation suites or alerting policies, consult metrics_and_alerts_catalog.md for public GKE metrics (
```
kubernetes.io/
```
) and open-source Kubernetes alerts (
```
awesome-prometheus-alerts
```
).
Plain English Response: You must include a plain English explanation for what the alerts do in your response. Explain what the alert measures, what the threshold represents, and what a trigger indicates.

非GKE独立运行时的触发限制与范围重定向:
- 本技能严格适用于使用PromQL和Google Cloud Managed Service for Prometheus的Google Kubernetes Engine (GKE)工作负载、集群及服务。
- 请勿用于非GKE计算运行时，例如独立Compute Engine虚拟机或未关联GKE的独立Cloud Run服务。
- 立即停止并直接响应（请勿编辑文件）: 当用户请求为非GKE计算基础设施配置告警时：
  1. 请勿在磁盘上编写、创建、编辑或验证任何Terraform文件。
  2. 立即停止并在聊天中直接回复用户:
    - 明确说明超出范围: 清晰告知用户，独立Compute Engine虚拟机监控或独立Cloud Run监控属于本GKE专属PromQL告警技能的范围之外，本技能专为使用Google Cloud Managed Service for Prometheus和PromQL的GKE工作负载设计。
    - 请勿生成GKE PromQL告警: 请勿为非GKE基础设施创建或生成Kubernetes PromQL告警策略，也不要虚构Kubernetes容器、Pod或节点资源。
    - 引导用户: 引导用户使用标准Google Cloud监控指标，例如
      compute.googleapis.com/instance/cpu/utilization
      或
      run.googleapis.com/request_latencies
      ，搭配标准
      google_monitoring_alert_policy
      和
      condition_threshold
      或MQL，或推荐相关的云可观测性专项技能。
强制
kube-state-metrics
(KSM)成本防护:
- 在Google Cloud Managed Service for Prometheus中部署开源
```
kube-state-metrics
```
  会产生可计费的指标摄入成本。
- 立即停止并先请求许可（请勿编辑文件）: 当请求的告警规则依赖Tier 2 KSM指标（例如
```
kube_cronjob_*
```
  、
```
kube_pod_status_phase
```
  、
```
kube_persistentvolume_*
```
  、
```
kube_deployment_*
```
  、
```
kube_statefulset_*
```
  、
```
kube_job_*
```
  或
```
kube_daemonset_*
```
  ）时，在获得用户批准前，请勿编写、创建、编辑或验证任何Terraform文件，也不要生成告警策略。
- 相反，你必须立即停止并直接回复用户，以：
  1. 提醒用户：请求的告警需要
```
kube-state-metrics
```
    。
  2. 说明成本影响：详细告知用户
```
kube-state-metrics
```
    在Google Cloud Managed Service for Prometheus中会产生可计费的样本摄入成本。
  3. 请求明确许可：在假设、启用或生成依赖KSM的告警配置前，请求用户的明确许可。
  4. 建议过滤或白名单：建议仅过滤或白名单特定所需指标，例如使用带有
```
metricRelabeling
```
    （
```
action: keep
```
    ）的
```
PodMonitoring
```
    资源，或KSM的
```
--metric-allowlist
```
    参数，以最小化摄入成本。提供具体的白名单示例。
- 始终优先选择非KSM原生替代方案（Tier 1 cAdvisor或metrics_and_alerts_catalog.md中记录的原生GKE指标），例如使用
```
container_memory_working_set_bytes
```
  和
```
container_spec_memory_limit_bytes
```
  替代
```
kube_pod_container_resource_limits
```
  。
- 响应中明确标识层级与成本附加费: 在生成或推荐告警策略的每一次响应中，你必须明确说明其分类层级和成本影响:
  - Tier 1原生或标准指标（GKE内置指标、cAdvisor
```
container_*
```
    、kubelet卷统计、kubelet节点状态、控制平面指标；详见metrics_and_alerts_catalog.md）：说明这是Tier 1原生或标准指标，无KSM成本附加费。
  - Tier 2 KSM指标：说明这是Tier 2依赖KSM的指标，并遵循上述许可和白名单防护规则。（提示：通常，带有
```
kube_
```
    前缀、代表资源状态或元数据的指标属于Tier 2。）
已批准文件编辑的规划-验证-执行循环: 在工作区中修改、添加或合并已批准的Terraform文件时，遵循三阶段工作流：
1. 规划: 起草结构化变更计划(
```
changes.json
```
  )，包含拟议的策略资源名称、PromQL表达式、分组标签和时长。
2. 验证: 运行编辑前验证脚本(
```
python3 scripts/validate_config.py --plan changes.json
```
  )，以验证PromQL语法、回溯窗口、时长规则，并确保无重复信号。
3. 执行: 计划通过验证后，将变更应用或合并到目标Terraform配置(
```
alerts.tf
```
  )中。
4. 注意: 当直接在聊天中回答问题或提供Terraform代码片段且未请求修改磁盘文件时，在响应中输出完整、有效的Terraform HCL块。
配置四大黄金信号与集群健康: 始终确保目标Kubernetes工作负载或服务具备以下告警覆盖：
1. 延迟（P95响应时间）
2. 错误（多窗口多燃烧率SLO告警，例如快速燃烧1小时/5分钟，系数14.4；慢速燃烧6小时/30分钟，系数6.0；请勿使用简单静态比率）
3. 流量（使用
```
absent()
```
  或
```
default 0
```
  语法检测流量骤降或指标完全消失，或过载峰值）
4. 饱和（仅内存限制利用率）: 描述或配置集群或项目的告警策略时，仅包含内存饱和（
```
container_memory_working_set_bytes
```
  /
```
container_spec_memory_limit_bytes
```
  ）。请勿包含CPU饱和告警，也不要将
```
container_cpu_usage_seconds_total
```
  列为告警指标，因为CPU是可压缩资源，会被CFS配额限制，而非导致不可压缩的致命终止（OOM）。
5. 集群健康（Pod循环崩溃、Node NotReady）
仅使用PromQL（托管Prometheus）: 必须使用
```
condition_prometheus_query_language
```
搭配PromQL。除非明确要求，请勿使用MQL或标准
```
condition_threshold
```
。Google Cloud Managed Service for Prometheus是GKE的标准遥测摄入路径。
仅使用Terraform: 生成的可观测性配置仅需以Terraform(
```
.tf
```
)文件形式编写，例如
```
alerts.tf
```
和
```
variables.tf
```
。
动态多资源告警（禁止硬编码）: 除非明确要求，否则不得在告警条件中硬编码特定Pod名称、节点名称或服务名称。告警策略必须编写为动态覆盖资源：
- 始终使用分组聚合（
```
by (cluster, namespace, service, pod, container)
```
  ）而非过滤到单个实例。这允许单个告警策略动态跟踪每个服务或Pod。
- 始终为
```
project_id
```
  、
```
cluster_name
```
  和
```
namespace
```
  声明并使用Terraform变量（
```
var.project_id
```
  、
```
var.cluster_name
```
  、
```
var.namespace
```
  ），使配置可跨环境复用。始终在
```
variables.tf
```
  （或配置内）定义这些变量，并在策略或PromQL标签匹配器中引用全部三个变量。
回溯窗口上无冗余时长缓冲:
- 当PromQL表达式已使用聚合回溯窗口（例如
```
increase(...[15m]) > 3
```
  或多窗口SLO燃烧率）时，查询时间窗口已平滑瞬态峰值。
- 在PromQL回溯窗口之上添加Terraform时长会增加检测平均时间（MTTD），且无额外平滑效果。
- 在这些情况下，设置Terraform
```
duration = "0s"
```
  （或
```
"60s"
```
  ）。请勿在
```
[15m]
```
  之上强制设置
```
duration = "300s"
```
  ，这会将关键循环崩溃告警延迟最多20分钟（15分钟+5分钟）。
- 仅对瞬时 gauge 条件使用
```
duration = "300s"
```
  ，例如
```
kube_node_status_condition == 0
```
  。
使用SLO燃烧率而非简单比率: 对于错误率告警，始终生成多窗口多燃烧率（MWMBR）SLO告警（例如99% SLO对应的1小时和5分钟窗口内14.4倍燃烧率），而非简单错误率比率（
```
rate(5xx)/rate(total) > 0.05
```
），后者在低流量时会产生过多误报。
可靠的流量下降检测（
absent()
/
default 0
）: 监控流量降至零时，请勿仅使用
```
rate(...) == 0
```
，因为当无请求时，Prometheus时间序列会完全消失（计算为空向量而非0）。使用
```
default 0
```
语法，例如
```
sum(rate(...[5m])) default 0 == 0
```
，或
```
absent(...) == 1
```
。
通知渠道: 默认情况下，未获得用户输入时，请勿配置任何通知渠道。如果用户明确提供通知渠道，则配置告警使用该渠道。否则，必须在响应中提示用户询问是否需要配置通知渠道。
参考GKE指标与开源告警目录: 设计或生成评估套件或告警策略时，请参考metrics_and_alerts_catalog.md获取公共GKE指标（
```
kubernetes.io/
```
）和开源Kubernetes告警（
```
awesome-prometheus-alerts
```
）。
通俗易懂的响应: 响应中必须包含告警作用的通俗易懂说明。解释告警测量的内容、阈值代表的含义，以及触发告警时的指示信息。

Alerting Policy Structure in Terraform

Terraform中的告警策略结构

Alerting policies must be defined using the

google_monitoring_alert_policy

resource with

condition_prometheus_query_language

. Always declare variables in

variables.tf

for

project_id

cluster_name

, and

namespace

hcl

undefined

告警策略必须使用

google_monitoring_alert_policy

资源搭配

condition_prometheus_query_language

定义。始终在

variables.tf

中为

project_id

、

cluster_name

和

namespace

声明变量。

hcl

undefined

variables.tf

variable "project_id" { type = string description = "Google Cloud Project ID" }

variable "cluster_name" { type = string description = "GKE Cluster Name" }

variable "namespace" { type = string description = "Target Kubernetes Namespace" default = "default" }

variable "slo_target" { type = number description = "SLO Target fraction (for example 0.99 for 99%)" default = 0.99 }


```hcl

variable "project_id" { type = string description = "Google Cloud Project ID" }

variable "cluster_name" { type = string description = "GKE Cluster Name" }

variable "namespace" { type = string description = "Target Kubernetes Namespace" default = "default" }

variable "slo_target" { type = number description = "SLO Target fraction (for example 0.99 for 99%)" default = 0.99 }


```hcl

alerts.tf

Example: Multi-Window Multi-Burn-Rate (MWMBR) SLO Alert (Fast Burn: 14.4x, 1h & 5m windows)

resource "google_monitoring_alert_policy" "k8s_service_error_rate_slo" { project = var.project_id display_name = "[K8s] ${var.cluster_name} - Service Error Rate SLO Fast Burn" combiner = "OR"

conditions { display_name = "Error Budget Fast Burn (14.4x over 1h and 5m)" condition_prometheus_query_language { query = <<-EOT ( ( sum( rate( http_requests_total{ cluster="${var.cluster_name}", namespace="${var.namespace}", status="5.." }[5m] ) ) by (service, namespace, cluster) / sum( rate( http_requests_total{ cluster="${var.cluster_name}", namespace="${var.namespace}" }[5m] ) ) by (service, namespace, cluster) ) > (1 - ${var.slo_target}) * 14.4 ) and ( ( sum( rate( http_requests_total{ cluster="${var.cluster_name}", namespace="${var.namespace}", status="5.." }[1h] ) ) by (service, namespace, cluster) / sum( rate( http_requests_total{ cluster="${var.cluster_name}", namespace="${var.namespace}" }[1h] ) ) by (service, namespace, cluster) ) > (1 - ${var.slo_target}) * 14.4 ) EOT duration = "0s" } } }


--------------------------------------------------------------------------------

resource "google_monitoring_alert_policy" "k8s_service_error_rate_slo" { project = var.project_id display_name = "[K8s] ${var.cluster_name} - Service Error Rate SLO Fast Burn" combiner = "OR"


--------------------------------------------------------------------------------

Telemetry Metrics and PromQL Examples

遥测指标与PromQL示例

For GKE metrics (

kubernetes.io/

), community open-source alerts (

awesome-prometheus-alerts

), KSM cost guardrails, and non-KSM native alternatives, you must read and follow:

metrics_and_alerts_catalog.md

For specific PromQL queries corresponding to each of the Golden Signals, you must read and follow:

promql_queries.md

For GKE cluster prerequisites, enabling Google Cloud Managed Service for Prometheus collection, configuring PodMonitoring custom scraping, and enabling control plane metrics collection (API Server, Controller Manager, Scheduler), you must read and follow:

gke_configuration_prerequisites.md

关于GKE指标（

kubernetes.io/

）、社区开源告警（

awesome-prometheus-alerts

）、KSM成本防护及非KSM原生替代方案，必须阅读并遵循：

metrics_and_alerts_catalog.md

关于对应四大黄金信号的具体PromQL查询，必须阅读并遵循：

promql_queries.md

关于GKE集群先决条件、启用Google Cloud Managed Service for Prometheus采集、配置PodMonitoring自定义抓取、启用控制平面指标采集（API Server、Controller Manager、Scheduler），必须阅读并遵循：

gke_configuration_prerequisites.md

Tooling Scripts and Validation Loop

工具脚本与验证循环

Use the

validate_config.py

script to validate change plans and Terraform configurations when working in a repository:

Pre-Edit Plan Validation: Draft a
```
changes.json
```
plan specifying the proposed policies, queries, and durations, and validate it before editing:
- Command:
```
python3 scripts/validate_config.py --plan changes.json
```
Post-Edit and Directory Validation: Scan existing or modified Terraform files in a directory to ensure no duplicates or syntax errors exist:
- Command:
```
python3 scripts/validate_config.py --directory [TARGET_TF_DIR] --cluster-var "${var.cluster_name}"
```
- Single file validation:
```
python3 scripts/validate_config.py --file [PATH_TO_TF_FILE]
```

在仓库中工作时，使用

validate_config.py

脚本验证变更计划和Terraform配置：

编辑前计划验证: 起草
```
changes.json
```
计划，指定拟议的策略、查询和时长，并在编辑前进行验证：
- 命令：
```
python3 scripts/validate_config.py --plan changes.json
```

编辑后与目录验证: 扫描目录中现有或修改后的Terraform文件，确保无重复或语法错误：

命令：

python3 scripts/validate_config.py --directory [TARGET_TF_DIR] --cluster-var "${var.cluster_name}"

单文件验证：

python3 scripts/validate_config.py --file [PATH_TO_TF_FILE]

Technical Considerations and Gotchas

技术考量与注意事项

Lookback Windows versus Duration Buffers:
- Do not add large
```
duration = "300s"
```
  buffers to alerts that already use aggregated lookback windows like
```
increase(...[15m])
```
  or multi-window SLO rates.
- The
```
[15m]
```
  window in
```
increase(...[15m]) > 3
```
  already smooths spikes. Adding
```
duration = "300s"
```
  increases MTTD by forcing the restart count to remain above 3 for an extra 5 continuous minutes, delaying alerts by up to 20 minutes total.
- Use
```
duration = "0s"
```
  or
```
"60s"
```
  when using lookback window functions. Reserve
```
duration = "300s"
```
  for raw instantaneous gauge conditions, such as
```
kube_node_status_condition == 0
```
  .
Memory Saturation Only for Cluster Alerting:
- Do not configure CPU saturation alerts for cluster or workload monitoring. CPU is compressible (throttled by the CFS scheduler), while memory is uncompressible (triggers OOMKills).
- Configure Memory Saturation using
```
container_memory_working_set_bytes
```
  /
```
container_spec_memory_limit_bytes
```
  .
Missing Resource Limits Blind Spot (Mandatory Explanation): Saturation alerts that compare usage to limits (such as
```
container_spec_memory_limit_bytes
```
) will fail to resolve or return
```
NaN
```
if workloads do not have explicit Memory limits configured in their Kubernetes manifests.
- Mandatory Instruction: Whenever you generate, discuss, or recommend any memory saturation alert comparing usage against limits (including non-KSM cAdvisor alternatives using
```
container_spec_memory_limit_bytes
```
  ), you must explicitly explain and warn the user in your response that container memory limits must be explicitly configured in the Kubernetes pod specs or manifests (
```
resources.limits.memory
```
  ) for the saturation query to resolve (and not return
```
NaN
```
  or fail to resolve).
Linear Disk Predictions (
predict_linear
): When forecasting volume exhaustion using
```
predict_linear(kubelet_volume_stats_available_bytes[6h:5m], 4 * 24 * 3600) < 0
```
, explain that
```
predict_linear
```
uses linear regression over the recent lookback window (for example, 6 hours) to project when available disk will drop below 0 (for example, within 4 days). Identify
```
kubelet_volume_stats_available_bytes
```
as a Tier 1 native kubelet metric with zero KSM surcharge.
API Server Error and Client Metrics:
- ```
apiserver_request_total
```
  and
```
rest_client_requests_total
```
  are Tier 1 Control Plane metrics with zero KSM cost surcharge. Explain that
```
apiserver_request_total
```
  monitors 5xx HTTP error rates across API server endpoints, while
```
rest_client_requests_total
```
  monitors 4xx and 5xx requests sent by REST clients communicating with the API server.
Traffic Disappearance Gotcha (
absent()
/
default 0
):
- When traffic drops completely to zero, Prometheus and GMP stop emitting the
```
http_requests_total
```
  time series.
- ```
sum(rate(...[5m])) == 0
```
  evaluates to an empty vector, preventing the alert from triggering.
- Always use
```
sum(rate(...[5m])) default 0 == 0
```
  or
```
absent(...) == 1
```
  to reliably detect total traffic loss.
CrashLooping versus Normal Restarts: A container restarting occasionally might be normal, for example job completion or a minor rolling update. Alert on frequent restarts (such as more than 3 restarts in 15 minutes with
```
duration = "0s"
```
) using
```
kube_pod_container_status_restarts_total
```
rather than a single restart to avoid noise.
Node Upgrades: During GKE cluster upgrades, nodes are drained and restarted, which can trigger "Node NotReady" alerts. Warn the user that these alerts might fire during maintenance windows, or suggest configuring maintenance windows if supported.

回溯窗口与时长缓冲:
- 请勿为已使用聚合回溯窗口（如
```
increase(...[15m])
```
  或多窗口SLO比率）的告警添加大的
```
duration = "300s"
```
  缓冲。
- ```
increase(...[15m]) > 3
```
  中的
```
[15m]
```
  窗口已平滑峰值。添加
```
duration = "300s"
```
  会通过强制重启计数在额外5分钟内持续高于3来增加MTTD，将告警延迟最多20分钟。
- 使用回溯窗口函数时，设置
```
duration = "0s"
```
  或
```
"60s"
```
  。仅对原始瞬时gauge条件保留
```
duration = "300s"
```
  ，例如
```
kube_node_status_condition == 0
```
  。
集群告警仅监控内存饱和:
- 请勿为集群或工作负载监控配置CPU饱和告警。CPU是可压缩资源（由CFS调度器限制），而内存是不可压缩资源（触发OOMKills）。
- 使用
```
container_memory_working_set_bytes
```
  /
```
container_spec_memory_limit_bytes
```
  配置内存饱和告警。
缺失资源限制的盲点（强制说明）: 将使用量与限制进行比较的饱和告警（例如
```
container_spec_memory_limit_bytes
```
），如果工作负载在Kubernetes清单中未配置明确的内存限制，将无法解析或返回
```
NaN
```
。
- 强制说明: 无论何时生成、讨论或推荐任何将使用量与限制进行比较的内存饱和告警（包括使用
```
container_spec_memory_limit_bytes
```
  的非KSM cAdvisor替代方案），你必须在响应中明确向用户解释并警告，必须在Kubernetes Pod规格或清单（
```
resources.limits.memory
```
  ）中明确配置容器内存限制，饱和查询才能解析（不会返回
```
NaN
```
  或解析失败）。
线性磁盘预测（
predict_linear
）: 使用
```
predict_linear(kubelet_volume_stats_available_bytes[6h:5m], 4 * 24 * 3600) < 0
```
预测卷耗尽时，需说明
```
predict_linear
```
使用近期回溯窗口（例如6小时）的线性回归来预测可用磁盘何时降至0以下（例如4天内）。需指出
```
kubelet_volume_stats_available_bytes
```
是Tier 1原生kubelet指标，无KSM附加费。
API Server错误与客户端指标:
- ```
apiserver_request_total
```
  和
```
rest_client_requests_total
```
  是Tier 1控制平面指标，无KSM成本附加费。需说明
```
apiserver_request_total
```
  监控API Server端点的5xx HTTP错误率，而
```
rest_client_requests_total
```
  监控与API Server通信的REST客户端发送的4xx和5xx请求。
流量消失的注意事项（
absent()
/
default 0
）:
- 当流量完全降至零时，Prometheus和GMP停止发送
```
http_requests_total
```
  时间序列。
- ```
sum(rate(...[5m])) == 0
```
  计算为空向量，导致告警无法触发。
- 始终使用
```
sum(rate(...[5m])) default 0 == 0
```
  或
```
absent(...) == 1
```
  来可靠检测流量完全丢失。
循环崩溃与正常重启: 容器偶尔重启可能是正常情况，例如作业完成或小幅滚动更新。使用
```
kube_pod_container_status_restarts_total
```
告警频繁重启（例如15分钟内超过3次重启，
```
duration = "0s"
```
），而非单次重启，以避免噪音。
节点升级: 在GKE集群升级期间，节点会被排空并重启，这可能触发“Node NotReady”告警。需警告用户这些告警可能在维护窗口期间触发，或建议在支持的情况下配置维护窗口。

gke-alert-configuration

Original

Translation

GKE Alert Configuration

GKE告警配置

Critical Rules

关键规则

Alerting Policy Structure in Terraform

Terraform中的告警策略结构

variables.tf

variables.tf

alerts.tf

alerts.tf

Example: Multi-Window Multi-Burn-Rate (MWMBR) SLO Alert (Fast Burn: 14.4x, 1h & 5m windows)

Example: Multi-Window Multi-Burn-Rate (MWMBR) SLO Alert (Fast Burn: 14.4x, 1h & 5m windows)

Telemetry Metrics and PromQL Examples

遥测指标与PromQL示例

Tooling Scripts and Validation Loop

工具脚本与验证循环

Technical Considerations and Gotchas

技术考量与注意事项

Additional Resources

额外资源