cloud-run-alert-configuration

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Cloud Run Alert Configuration

Cloud Run 告警配置

Production-grade observability for Google Cloud Run using Terraform and PromQL (Cloud Monitoring). Grounded in SRE practices, this skill focuses strictly on actionable user impact and scaling bounds.

使用Terraform和PromQL(Cloud Monitoring)为Google Cloud Run提供生产级可观测性。本技能基于SRE实践,严格聚焦于可触发处置的用户影响与扩缩容边界。

CRITICAL RULES

关键规则

  • Gcloud SDK Prerequisites: For this and any other
    gcloud
    -related tasks in this skill (such as resource discovery, parameter inspection, or metric scope centralization), ensure that the Google Cloud SDK (
    gcloud
    ) is installed, authenticated, and configured with the target project (e.g. via
    gcloud auth print-access-token
    and
    gcloud config get-value project
    ). If
    gcloud
    is missing or unconfigured, instruct the user to configure the SDK or fall back to parsing local workspace
    .tf
    files.
  • Autonomous Discovery (Config First, CLI Second): Never prompt for names, regions, or ceilings if discoverable.
    • Config First: Prioritize parsing local
      .tf
      files in the workspace. Look for
      google_cloud_run_v2_service
      ,
      google_cloud_run_service
      ,
      google_cloud_run_v2_job
      ,
      max_instance_count
      , and Knative
      maxScale
      annotations (
      autoscaling.knative.dev/maxScale
      or
      run.googleapis.com/maxScale
      ).
    • CLI Second (gcloud Fallback): If not discoverable via configuration, verify the
      gcloud
      SDK is present and configured with a valid project (
      gcloud config get-value project
      ). Then execute
      gcloud run services list --format="json"
      ,
      gcloud run jobs list --format="json"
      , or
      gcloud monitoring metrics-scopes list
      .
  • Workload Routing: Always classify the workload target and follow its specific reference guide:
    • For HTTP Services follow services.md
    • For Cloud Run Jobs follow jobs.md
    • For Worker Pools follow worker_pools.md
  • Explicit Defaults & User Overrides:
    • Always use explicit defaults for all constants specified in the target workload's reference file (SLO targets, latency thresholds, SLAs, saturation ceilings).
    • State the defaults being applied in the final summary output and clearly notify the user that any default constant can be customized or overridden via Terraform variables or prompt input.
  • Metric Scope Centralization: Run
    gcloud beta monitoring metrics-scopes list projects/[PROJECT_ID]
    . If a scoping project (
    locations/global/metricsScopes/[SCOPING_PROJECT_ID]
    ) exists, set
    project = "[SCOPING_PROJECT_ID]"
    in Terraform resources.
  • PromQL
    duration
    (Retest Window) Rules
    :
    • Lookbacks $\le$ 25h: Set
      duration = "300s"
      (5m buffer) to absorb transient blips and scale-up lag (except immediate job failure alerts which use
      duration = "0s"
      ).
    • Lookbacks $> 25$h (e.g. 3d/7d Slow Burn): Omit
      duration
      entirely
      (or set to
      0s
      ). Cloud Monitoring rejects PromQL queries with
      duration
      set on lookbacks >25h (
      INVALID_ARGUMENT
      ).
  • Terraform Standards: Output clean
    .tf
    configurations using
    google_monitoring_alert_policy
    and
    condition_prometheus_query_language
    . Include
    alert_strategy { auto_close = "604800s" }
    and parameterize
    notification_channels = var.notification_channels
    .
  • Gcloud SDK 前置要求:对于本技能中涉及
    gcloud
    的所有任务(例如资源发现、参数检查或指标范围集中化),需确保Google Cloud SDK(
    gcloud
    )已安装、完成身份验证,并配置了目标项目(例如通过
    gcloud auth print-access-token
    gcloud config get-value project
    命令验证)。如果
    gcloud
    缺失或未配置,应指导用户配置SDK,或回退为解析本地工作区的
    .tf
    文件。
  • 自动发现(配置优先,CLI次之):如果可自动发现资源名称、区域或上限,切勿向用户询问。
    • 配置优先:优先解析工作区中的本地
      .tf
      文件。查找
      google_cloud_run_v2_service
      google_cloud_run_service
      google_cloud_run_v2_job
      max_instance_count
      以及Knative的
      maxScale
      注解(
      autoscaling.knative.dev/maxScale
      run.googleapis.com/maxScale
      )。
    • CLI次之(gcloud回退方案):如果无法通过配置发现资源,先验证
      gcloud
      SDK已安装且配置了有效项目(
      gcloud config get-value project
      ),然后执行
      gcloud run services list --format="json"
      gcloud run jobs list --format="json"
      gcloud monitoring metrics-scopes list
      命令。
  • 工作负载路由:始终先对目标工作负载进行分类,然后遵循对应的参考指南:
    • HTTP服务请参考services.md
    • Cloud Run Jobs请参考jobs.md
    • 工作池请参考worker_pools.md
  • 显式默认值与用户覆盖
    • 目标工作负载参考文件中指定的所有常量(SLO目标、延迟阈值、SLA、饱和度上限)均需使用显式默认值。
    • 在最终的摘要输出中说明所使用的默认值,并明确告知用户所有默认常量都可以通过Terraform变量或提示输入进行自定义或覆盖。
  • 指标范围集中化:运行
    gcloud beta monitoring metrics-scopes list projects/[PROJECT_ID]
    命令。如果存在范围项目(
    locations/global/metricsScopes/[SCOPING_PROJECT_ID]
    ),则在Terraform资源中设置
    project = "[SCOPING_PROJECT_ID]"
  • PromQL
    duration
    (重测窗口)规则
    • 回溯周期 $\le$ 25小时:设置
      duration = "300s"
      (5分钟缓冲)以吸收瞬时波动和扩容延迟(即时作业失败告警除外,此类告警使用
      duration = "0s"
      )。
    • 回溯周期 $> 25$小时(例如3天/7天慢燃告警):完全省略
      duration
      参数
      (或设置为
      0s
      )。Cloud Monitoring会拒绝回溯周期超过25小时且设置了
      duration
      的PromQL查询(返回
      INVALID_ARGUMENT
      错误)。
  • Terraform规范:使用
    google_monitoring_alert_policy
    condition_prometheus_query_language
    输出整洁的
    .tf
    配置。需包含
    alert_strategy { auto_close = "604800s" }
    ,并将
    notification_channels = var.notification_channels
    参数化。

WORKFLOW STEPS

工作流程步骤

1. Discovery & Target Identification (Config First, CLI Second)

1. 发现与目标识别(配置优先,CLI次之)

  • Config First: Scan workspace
    .tf
    files for
    google_cloud_run_v2_service
    ,
    google_cloud_run_service
    ,
    google_cloud_run_v2_job
    , and worker pool resources.
  • CLI Second: If not found in config, verify
    gcloud
    is installed and has a valid project configured (
    gcloud config get-value project
    ), then run
    gcloud
    discovery commands.
  • Group targets by workload type: HTTP Services, Jobs, or Worker Pools.
  • Identify the scoping project using
    gcloud monitoring metrics-scopes
    .
  • 配置优先:扫描工作区的
    .tf
    文件,查找
    google_cloud_run_v2_service
    google_cloud_run_service
    google_cloud_run_v2_job
    以及工作池资源。
  • CLI次之:如果在配置中未找到,验证
    gcloud
    已安装且配置了有效项目(
    gcloud config get-value project
    ),然后运行
    gcloud
    发现命令。
  • 按工作负载类型对目标进行分组:HTTP服务、作业或工作池。
  • 使用
    gcloud monitoring metrics-scopes
    识别范围项目。

2. Configure Alerts

2. 配置告警

  • Route to the corresponding guide to generate the alert policies:
    • HTTP Services: Open services.md. Apply the comprehensive alerting suite covering availability SLOs (5xx), request latency (P95/P99), client errors (4xx), container instance saturation, container CPU/memory utilization, traffic anomalies, and billable instance time.
    • Batch Jobs: Open jobs.md. Apply immediate job execution failure alerts.
    • Worker Pools: Open worker_pools.md. Apply the 4-policy standard suite (Task Success SLO Fast/Slow Burn, Backlog ETD, Message Age SLA).
  • 跳转至对应指南生成告警策略:
    • HTTP服务:打开services.md。应用全面的告警套件,覆盖可用性SLO(5xx错误)、请求延迟(P95/P99)、客户端错误(4xx)、容器实例饱和度、容器CPU/内存利用率、流量异常以及计费实例时长。
    • 批处理作业:打开jobs.md。应用即时作业执行失败告警。
    • 工作池:打开worker_pools.md。应用4项策略的标准套件(任务成功率SLO快燃/慢燃告警、积压ETD、消息时长SLA)。

3. Terraform Generation & Review

3. Terraform生成与审核

  • Write the HCL configuration to
    .tf
    files with explicitly parameterized defaults.
  • State the applied defaults and remind the user of their ability to override any constant.
  • Provide a clear plain-English breakdown of the PromQL logic and triggering thresholds.

  • 将HCL配置写入
    .tf
    文件,所有默认值均显式参数化。
  • 说明所应用的默认值,并提醒用户可覆盖任意常量。
  • 用清晰易懂的语言解释PromQL逻辑和触发阈值。

Additional Resources

更多资源