agent-platform-alert-configuration

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Agent Platform Alert Configuration

Agent Platform告警配置

This skill provides dynamic threshold alerting configurations for Google Cloud / Vertex AI Reasoning Engines (Agent Platform container deployments) using extended 1-week lookback retention baselines. Standard static thresholds (e.g., "latency > 2s") cause excessive alert noise for AI agents. Dynamic PromQL baselines solve this.

本技能基于1周回溯保留基线，为Google Cloud / Vertex AI推理引擎（Agent Platform容器部署）提供动态阈值告警配置。标准静态阈值（如“延迟>2秒”）会为AI代理带来过多告警噪声，而动态PromQL基线可解决此问题。

Safety & Confirmation Tiers (CRITICAL)

安全与确认层级（CRITICAL）

Before executing any commands or writing configurations on behalf of the user, you MUST adhere to the following safety tiers based on the action requested:

Tier R: Read-only (
check_telemetry.py
)
- Rule: No confirmation needed. You may execute these scripts immediately to inspect the telemetry status of the Reasoning Engine.
Tier B: Billing & Resource Creation (
create_online_monitor.py
/ provisioning)
- Rule: Explicit User Confirmation Required. These actions incur additional billing charges and create cloud resources. The agent MUST ask the user directly for approval before proceeding.

在代表用户执行任何命令或编写配置之前，你必须根据请求的操作遵守以下安全层级：

层级R：只读（
check_telemetry.py
）
- 规则: 无需确认。你可立即执行这些脚本，检查推理引擎的遥测状态。
层级B：计费与资源创建（
create_online_monitor.py
/ 资源配置）
- 规则: 必须获得用户明确确认。这些操作会产生额外计费并创建云资源。代理在执行前必须直接向用户请求批准。

CRITICAL RULES

关键规则

Always configure both Reliability and Quality alerting policies for the target agent (6 policies in total):
- For Reliability Monitoring: You MUST configure exactly three alerting policies:
  1. Latency (anomaly monitoring)
  2. Error Rate - Fast Burn SLO (1-Hour Window)
  3. Error Rate - Slow Burn SLO (3-Day Window)
- For Quality Monitoring: You MUST configure exactly three alerting policies:
  1. Final Response Quality
  2. Tool Use Quality
  3. Hallucination
Online Monitor Provisioning & Cost Warning: Quality alerting policies rely on metrics exported by Online Monitors. You MUST ensure the Online Monitor is provisioned for the agent and telemetry is enabled:
- Ask for Approval: Both Online Monitors and Telemetry incur separate billing charges. Before provisioning them, you MUST warn the user about these extra costs. If not pre-approved in the prompt, you MUST ask a direct question in your response requesting confirmation/approval to proceed (e.g., "Please confirm if you approve the extra billing costs for the Online Monitor and Telemetry to proceed.").
- Verify Telemetry First: Before generating any alerting policy plan or provisioning Online Monitors, you MUST always verify the telemetry status of the Reasoning Engine first using the
```
check_telemetry.py
```
  script as detailed in Verify Telemetry Status below.
- Follow the Guide: Follow the step-by-step instructions in the Online Monitor & Telemetry Provisioning section below.
Brand New Agents (No Traffic History): When setting up alerts for a brand new agent, you MUST explicitly ask the user what traffic pattern they expect (Steady, Seasonal, or Bursty) in your response. If immediate setup is requested, ask the question but proceed using the default Steady/Consistent (Short-Window Z-Score) pattern. Follow no_historical_traffic_data.md.
PromQL for Reliability (No MQL or Threshold Filters): For the 3 reliability metrics, you MUST use
```
condition_prometheus_query_language
```
with PromQL. Do NOT use MQL or standard
```
condition_threshold
```
.
Standard Threshold Filters for Agent Quality: For the 3 agent quality metrics, you MUST use standard
```
condition_threshold
```
filters matching the monitored resource type
```
aiplatform.googleapis.com/OnlineEvaluator
```
and metric type
```
aiplatform.googleapis.com/online_evaluator/scores
```
. Do NOT use PromQL.
Install Terraform if Necessary: You should use terraform to deploy and must install terraform if you can't find a valid install.
Terraform Only: Write the generated observability configuration ONLY as Terraform (
```
.tf
```
) files (e.g.,
```
alerts.tf
```
,
```
variables.tf
```
).
Dynamic Multi-Resource Alerting (No Single-Resource Pinning): You MUST NOT hardcode specific agent IDs or resource name filters (e.g.,
```
{reasoning_engine_id="[AGENT_ID]"}
```
or
```
metric.labels.agent_resource_name="[AGENT_NAME]"
```
) in alerting conditions unless explicitly requested. Alerting policies must be written to cover all active agents in the project dynamically:
- For Reliability Metrics using PromQL: ALWAYS use grouping aggregations (
```
by (reasoning_engine_id)
```
  ) instead of filtering to a single ID. This allows a single alert policy to dynamically track each reasoning engine instance separately.
- For Quality Metrics using Standard Threshold Filters: Omit the
```
agent_resource_name
```
  filter entirely. Configure the condition filter to only target the monitored resource type (
```
aiplatform.googleapis.com/OnlineEvaluator
```
  ) and metric type (
```
aiplatform.googleapis.com/online_evaluator/scores
```
  ) globally for the project.
Check for Pre-existing Policies: Avoid creating duplicate alert policies for a reasoning engine: scan the target directory or workspace to see if a policy already exists that targets the same metrics using aggregations grouped by
```
reasoning_engine_id
```
.
Metric Scope Discovery & Project Inference: Centralize alert policies in a Metric Scope (scoping project) to save costs. Identify if a scope is used and where policies should live by checking:
1. GCP CLI Check: Run
```
gcloud beta monitoring metrics-scopes list projects/[PROJECT_ID]
```
  . If a parent scope
```
locations/global/metricsScopes/[SCOPING_PROJECT_ID]
```
  is returned, a Metric Scope is active; deploy policies there.
2. Infrastructure as Code Scan: Search Terraform configurations for
```
google_monitoring_monitored_project
```
  resources and extract the scoping project from the
```
metrics_scope
```
  attribute.
3. Ambiguity Fallback: If unable to determine, ask the user: "Are you using a multi-project Cloud Monitoring Metric Scope? If so, what is the scoping project ID?" Deploy policies to the deduced scoping project (setting the
```
project
```
  attribute in HCL), or default to the local project.
Directory Inference: Deploy configuration files to target Terraform or SRE folders (e.g.
```
monitoring/
```
,
```
ops/
```
,
```
sre/
```
). Use tools to locate where alert policies or state pointers exist in the project, rather than blindly writing to the current working directory.
Notification Channels: By default, never configure any notification channels without user input. If the user explicitly provides a notification channel in their prompt, configure the alerts to use it. If no notification channel is provided, you MUST explicitly ask the user in your final response if they would like to configure notification channels. This is a mandatory question and you MUST NOT omit it from your response. IMPORTANT Do NOT make assumptions about notification channels. If you search the codebase for a notification channel you must ALWAYS confirm with the user before using it.
Plain English Response: You MUST include a plain English explanation for what the alerts do in your response. This must explain in plain English what the alert measures, how the algorithm works, and what a trigger indicates.
Avoid Recursive Directory Operations: You MUST NOT run recursive listing or search commands (such as
```
ls -R
```
,
```
find .
```
, or raw recursive
```
grep
```
) from the google3 workspace root, as this will hang your session. Always target specific subdirectories.
Background Task Cleanup: You MUST check the status of all background tasks that you spawn. Before completing your execution and returning your final response, you MUST terminate or kill any active or hanging background tasks (using the
```
manage_task
```
tool with action
```
kill
```
).

必须为目标代理同时配置可靠性和质量告警策略（共6个策略）：
- 可靠性监控: 必须配置恰好三个告警策略：
  1. 延迟（异常监控）
  2. 错误率 - 快速消耗SLO（1小时窗口）
  3. 错误率 - 慢速消耗SLO（3天窗口）
- 质量监控: 必须配置恰好三个告警策略：
  1. 最终响应质量
  2. 工具使用质量
  3. 幻觉
在线监控器配置与成本警告: 质量告警策略依赖于在线监控器导出的指标。你必须确保已为代理配置在线监控器并启用遥测：
- 请求批准: 在线监控器和遥测均会产生单独计费。在配置之前，你必须提醒用户这些额外成本。如果提示中未预先批准，你必须在回复中直接询问用户是否确认/批准继续（例如：“请确认你是否批准在线监控器和遥测的额外计费以继续操作。”）。
- 先验证遥测: 在生成任何告警策略计划或配置在线监控器之前，你必须始终使用
```
check_telemetry.py
```
  脚本验证推理引擎的遥测状态，详情请参考下方的验证遥测状态。
- 遵循指南: 遵循下方在线监控器与遥测配置部分的分步说明。
全新代理（无流量历史）: 为全新代理设置告警时，你必须在回复中明确询问用户预期的流量模式（稳定型、季节性或突发型）。如果用户要求立即设置，需询问该问题，但默认使用稳定/一致（短窗口Z分数）模式进行配置。遵循no_historical_traffic_data.md。
可靠性指标使用PromQL（禁止MQL或阈值过滤器）: 对于3个可靠性指标，必须使用
```
condition_prometheus_query_language
```
搭配PromQL。禁止使用MQL或标准
```
condition_threshold
```
。
代理质量指标使用标准阈值过滤器: 对于3个代理质量指标，必须使用标准
```
condition_threshold
```
过滤器，匹配受监控资源类型
```
aiplatform.googleapis.com/OnlineEvaluator
```
和指标类型
```
aiplatform.googleapis.com/online_evaluator/scores
```
。禁止使用PromQL。
必要时安装Terraform: 你应使用Terraform进行部署，如果未找到有效的安装版本，必须安装Terraform。
仅使用Terraform: 生成的可观测性配置只能以Terraform（
```
.tf
```
）文件形式编写（例如
```
alerts.tf
```
、
```
variables.tf
```
）。
动态多资源告警（禁止单资源绑定）: 除非明确要求，否则不得在告警条件中硬编码特定代理ID或资源名称过滤器（如
```
{reasoning_engine_id="[AGENT_ID]"}
```
或
```
metric.labels.agent_resource_name="[AGENT_NAME]"
```
）。告警策略必须编写为动态覆盖项目中所有活跃代理：
- 使用PromQL的可靠性指标: 始终使用分组聚合（
```
by (reasoning_engine_id)
```
  ），而非过滤到单个ID。这允许单个告警策略动态分别跟踪每个推理引擎实例。
- 使用标准阈值过滤器的质量指标: 完全省略
```
agent_resource_name
```
  过滤器。将条件过滤器配置为仅针对项目范围内的受监控资源类型（
```
aiplatform.googleapis.com/OnlineEvaluator
```
  ）和指标类型（
```
aiplatform.googleapis.com/online_evaluator/scores
```
  ）。
检查已有策略: 避免为推理引擎创建重复告警策略：扫描目标目录或工作区，查看是否已存在使用
```
reasoning_engine_id
```
分组聚合来监控相同指标的策略。
指标范围发现与项目推断: 将告警策略集中在指标范围（作用域项目）中以节省成本。通过以下方式确定是否使用了范围以及策略应部署的位置：
1. GCP CLI检查: 运行
```
gcloud beta monitoring metrics-scopes list projects/[PROJECT_ID]
```
  。如果返回父范围
```
locations/global/metricsScopes/[SCOPING_PROJECT_ID]
```
  ，则表示指标范围已激活；将策略部署到该位置。
2. 基础设施即代码扫描: 在Terraform配置中搜索
```
google_monitoring_monitored_project
```
  资源，并从
```
metrics_scope
```
  属性中提取作用域项目。
3. 歧义回退: 如果无法确定，询问用户：“你是否使用多项目Cloud Monitoring指标范围？如果是，作用域项目ID是什么？”将策略部署到推断出的作用域项目（在HCL中设置
```
project
```
  属性），或默认部署到本地项目。
目录推断: 将配置文件部署到目标Terraform或SRE文件夹（如
```
monitoring/
```
、
```
ops/
```
、
```
sre/
```
）。使用工具定位项目中告警策略或状态指针的位置，而非盲目写入当前工作目录。
通知渠道: 默认情况下，未经用户输入不得配置任何通知渠道。如果用户在提示中明确提供了通知渠道，配置告警以使用该渠道。如果未提供通知渠道，你必须在最终回复中明确询问用户是否要配置通知渠道。这是必填问题，不得省略。重要提示：请勿对通知渠道做出假设。如果你在代码库中搜索到通知渠道，必须始终在使用前与用户确认。
通俗易懂的回复: 你必须在回复中包含告警功能的通俗易懂说明。必须用简单的语言解释告警监控的内容、算法的工作原理以及触发告警的含义。
避免递归目录操作: 不得从google3工作区根目录运行递归列出或搜索命令（如
```
ls -R
```
、
```
find .
```
或原始递归
```
grep
```
），否则会导致会话挂起。始终针对特定子目录。
后台任务清理: 你必须检查所有启动的后台任务状态。在完成执行并返回最终回复之前，必须终止或杀死任何活跃或挂起的后台任务（使用
```
manage_task
```
工具并指定
```
kill
```
操作）。

Algorithm Selection & Policy Mapping Process

算法选择与策略映射流程

Alerting policies for reasoning engine agents MUST map to the correct algorithms to ensure statistical stability and prevent alert noise or blind spots based on data classes:

Latency: Follows workload traffic pattern (Steady -> Z-Score; Seasonal -> Seasonal Decomposition; Bursty -> Moving Averages).
Error Rate: ALWAYS use Multi-Window Multi-Burn Rate SLOs (or ratio-based static thresholds). Error rate is naturally sparse (normally
```
0
```
). When standard deviation is
```
0
```
, Z-score computation is mathematically unstable (division-by-zero or NaN), causing false alert storms.

To resolve the workload traffic pattern (Seasonal, Steady, or Bursty), follow the instructions corresponding to the availability of historical metrics data:

Case 1: No historical metrics data available (e.g., brand new agent): You MUST read and follow: no_historical_traffic_data.md
Case 2: Historical metrics data available (e.g., active agent with traffic): You MUST read and follow: has_historical_traffic_data.md

推理引擎代理的告警策略必须映射到正确的算法，以确保统计稳定性，并根据数据类别防止告警噪声或监控盲区：

延迟: 遵循工作负载流量模式（稳定型→Z分数；季节性→季节性分解；突发型→移动平均值）。
错误率: 始终使用多窗口多消耗率SLO（或基于比率的静态阈值）。错误率本质上是稀疏的（通常为
```
0
```
）。当标准差为
```
0
```
时，Z分数计算在数学上不稳定（除以零或NaN），会导致虚假告警风暴。

要确定工作负载流量模式（季节性、稳定型或突发型），请根据历史指标数据的可用性遵循相应说明：

情况1：无历史指标数据可用（例如全新代理）: 你必须阅读并遵循：no_historical_traffic_data.md
情况2：有历史指标数据可用（例如有流量的活跃代理）: 你必须阅读并遵循：has_historical_traffic_data.md

Telemetry Metrics and PromQL Examples

遥测指标与PromQL示例

All raw telemetry metrics for the Agent Platform are cumulative counters. Because we monitor their rates or quantiles, we can optimize the PromQL queries by using longer range windows (e.g.,

[1w]

) for historical averages instead of expensive

avg_over_time

subqueries.

Signal	Raw Metric	Type	Description
Latency	`reasoning_engine_request_latencies_bucket`	Counter	Histogram bucket of request latencies
Error Rate	`reasoning_engine_request_count`	Counter	Cumulative count of requests

For the specific PromQL queries corresponding to each algorithm, you MUST read and follow: promql_queries.md

Agent Platform的所有原始遥测指标均为累积计数器。由于我们监控它们的速率或分位数，我们可以通过使用更长的范围窗口（如

[1w]

）计算历史平均值，来优化PromQL查询，避免使用昂贵的

avg_over_time

子查询。

信号	原始指标	类型	描述
延迟	`reasoning_engine_request_latencies_bucket`	计数器	请求延迟的直方图桶
错误率	`reasoning_engine_request_count`	计数器	请求的累积计数

关于每种算法对应的具体PromQL查询，你必须阅读并遵循：promql_queries.md

Agent Quality Metrics (Online Monitor)

代理质量指标（在线监控器）

All agent quality evaluation metrics are exported by Online Monitors to the monitored resource type

aiplatform.googleapis.com/OnlineEvaluator

under the metric type

aiplatform.googleapis.com/online_evaluator/scores

所有代理质量评估指标均由在线监控器导出到受监控资源类型

aiplatform.googleapis.com/OnlineEvaluator

，属于指标类型

aiplatform.googleapis.com/online_evaluator/scores

。

Metric Details & Aligners

指标详情与对齐器

Because the scores metric is of value type

DISTRIBUTION

, standard mean-based PromQL or arithmetic

ALIGN_MEAN

aligners are unsupported. You MUST use a percentile aligner (typically

ALIGN_PERCENTILE_50

to evaluate the median score) within the

aggregations

block of your

condition_threshold

Signal	Metric Name ( `evaluation_metric_name` )	Target Threshold	Recommended Aligner
Final Response Quality	`final_response_quality_v1`	`< 0.8` (or custom)	`ALIGN_PERCENTILE_50`
Tool Use Quality	`tool_use_quality_v1`	`< 0.8` (or custom)	`ALIGN_PERCENTILE_50`
Hallucination (Groundedness)	`hallucination_v1`	`< 0.9` (or custom)	`ALIGN_PERCENTILE_50`

由于分数指标的值类型为

DISTRIBUTION

，不支持基于标准均值的PromQL或算术

ALIGN_MEAN

对齐器。你必须在

condition_threshold

的

aggregations

块中使用分位数对齐器（通常使用

ALIGN_PERCENTILE_50

评估中位数分数）。

信号	指标名称 ( `evaluation_metric_name` )	目标阈值	推荐对齐器
最终响应质量	`final_response_quality_v1`	`< 0.8` （或自定义）	`ALIGN_PERCENTILE_50`
工具使用质量	`tool_use_quality_v1`	`< 0.8` （或自定义）	`ALIGN_PERCENTILE_50`
幻觉（接地性）	`hallucination_v1`	`< 0.9` （或自定义）	`ALIGN_PERCENTILE_50`

Metric Filter Example

指标过滤器示例

When configuring a quality alert policy in Terraform, use the following filter expression structure:

filter

resource.type="aiplatform.googleapis.com/OnlineEvaluator"
AND metric.type="aiplatform.googleapis.com/online_evaluator/scores"
AND metric.labels.evaluation_metric_name="[METRIC_NAME]"

在Terraform中配置质量告警策略时，请使用以下过滤器表达式结构：

filter

resource.type="aiplatform.googleapis.com/OnlineEvaluator"
AND metric.type="aiplatform.googleapis.com/online_evaluator/scores"
AND metric.labels.evaluation_metric_name="[METRIC_NAME]"

Online Monitor & Telemetry Provisioning

在线监控器与遥测配置

Quality metrics are generated by the Online Monitor by evaluating trace data exported to Cloud Trace. If telemetry is disabled on the reasoning engine, no traces are sent, and the quality metrics will remain empty.

质量指标由在线监控器通过评估导出到Cloud Trace的跟踪数据生成。如果推理引擎上的遥测已禁用，则不会发送跟踪数据，质量指标将保持为空。

Prerequisites & Dependencies

前提条件与依赖项

Before executing any scripts in this skill (such as

check_telemetry.py

create_online_monitor.py

), you MUST install the required dependencies in your environment. Run this command first:

bash

pip install -r scripts/requirements.txt

在执行本技能中的任何脚本（如

check_telemetry.py

或

create_online_monitor.py

）之前，你必须在环境中安装所需依赖项。首先运行以下命令：

bash

pip install -r scripts/requirements.txt

Verify Telemetry Status

验证遥测状态

Before generating any alerting policies, proposing a plan, or provisioning Online Monitors, you MUST always check if the agent is ready to export traces by running the telemetry checking script:

Mandatory Command:
```
python3 scripts/check_telemetry.py --project-id "[PROJECT_ID]" --agent-resource-name "[AGENT_RESOURCE_NAME]"
```
- Note on Parameters: The
```
[AGENT_RESOURCE_NAME]
```
  parameter MUST be the full resource path format
```
projects/<project_id>/locations/<location>/reasoningEngines/<agent_id>
```
  (e.g.
```
projects/gcp-prod/locations/us-central1/reasoningEngines/556677
```
  ) and not just the agent ID itself.
- Dependency Failures: If package imports or dependency installation fails, try your best to resolve the issues (e.g., by verifying package installation) and run the script again with the specific project-id and agent-resource-name parameters. If you cannot run the script successfully due to missing dependencies, you MUST still attempt to run it first and then include the complete, pre-populated execution command in your final response as a verification plan (this applies even if the user has pre-approved provisioning).
Detailed Enablement Instructions: For details on the required environment variables, Terraform setups, and project dependencies, you MUST read and follow: telemetry_enablement.md.

在生成任何告警策略、提出计划或配置在线监控器之前，你必须始终通过运行遥测检查脚本，检查代理是否准备好导出跟踪数据：

必填命令:
```
python3 scripts/check_telemetry.py --project-id "[PROJECT_ID]" --agent-resource-name "[AGENT_RESOURCE_NAME]"
```
- 参数说明:
```
[AGENT_RESOURCE_NAME]
```
  参数必须是完整的资源路径格式
```
projects/<project_id>/locations/<location>/reasoningEngines/<agent_id>
```
  （例如
```
projects/gcp-prod/locations/us-central1/reasoningEngines/556677
```
  ），而非仅代理ID。
- 依赖项失败: 如果包导入或依赖项安装失败，请尽力解决问题（例如验证包安装），并使用特定的project-id和agent-resource-name参数重新运行脚本。如果由于缺少依赖项无法成功运行脚本，你仍必须先尝试运行，然后在最终回复中包含完整的、预填充的执行命令作为验证计划（即使用户已预先批准配置，此要求也适用）。
详细启用说明: 关于所需环境变量、Terraform设置和项目依赖项的详细信息，你必须阅读并遵循：telemetry_enablement.md。

Provision the Online Monitor

配置在线监控器

Because Online Monitors cannot be configured via Terraform, run the provisioning script to generate the required metrics:

Command:

python3 scripts/create_online_monitor.py --project-id "[PROJECT_ID]" --agent-resource-name "[AGENT_RESOURCE_NAME]" [--sampling-percentage [PERCENTAGE]]

Sampling Rate Recommendation: For production agents, configure a conservative sampling percentage (default: 10%) to control LLM evaluation costs. For details, refer to Continuous evaluation with online monitors.

由于无法通过Terraform配置在线监控器，请运行配置脚本来生成所需指标：

命令:

python3 scripts/create_online_monitor.py --project-id "[PROJECT_ID]" --agent-resource-name "[AGENT_RESOURCE_NAME]" [--sampling-percentage [PERCENTAGE]]

采样率建议: 对于生产环境代理，配置保守的采样百分比（默认：10%）以控制LLM评估成本。详情请参考使用在线监控器进行持续评估。

Formatting the Execution Plan

执行计划格式

If execution fails (e.g. due to sandbox restrictions or permissions), you MUST plan it as a concrete next step. To do this, you MUST explicitly include a section titled "Online Monitor Provisioning Plan" or "Execution Plan" in your final response, containing the exact concrete python execution command with all parameter values (such as project ID, region, and agent resource name) fully populated. Do not merely state that the user should run it.

You MUST format the plan exactly as follows:

如果执行失败（例如由于沙箱限制或权限问题），你必须将其规划为具体的下一步操作。为此，你必须在最终回复中明确包含标题为**“在线监控器配置计划”或“执行计划”**的部分，其中包含所有参数值（如项目ID、区域和代理资源名称）已完全填充的具体Python执行命令。不得仅告知用户应运行该命令。

你必须严格按照以下格式规划：

Execution Plan: Online Monitor Provisioning

执行计划：在线监控器配置

Online Monitor Provisioning Command:

bash

python3 scripts/create_online_monitor.py \
  --project-id "[PROJECT_ID]" \
  --agent-resource-name "projects/[PROJECT_ID]/locations/[LOCATION]/reasoningEngines/[AGENT_ID]" \
  --sampling-percentage [PERCENTAGE]

Verify Telemetry Command (Optional fallback):

bash

python3 scripts/check_telemetry.py \
  --project-id "[PROJECT_ID]" \
  --agent-resource-name "projects/[PROJECT_ID]/locations/[LOCATION]/reasoningEngines/[AGENT_ID]"

在线监控器配置命令：

bash

python3 scripts/create_online_monitor.py \
  --project-id "[PROJECT_ID]" \
  --agent-resource-name "projects/[PROJECT_ID]/locations/[LOCATION]/reasoningEngines/[AGENT_ID]" \
  --sampling-percentage [PERCENTAGE]

验证遥测命令（可选回退）：

bash

python3 scripts/check_telemetry.py \
  --project-id "[PROJECT_ID]" \
  --agent-resource-name "projects/[PROJECT_ID]/locations/[LOCATION]/reasoningEngines/[AGENT_ID]"

Tooling Scripts

工具脚本

Use the following scripts to resolve duplicates and validate configs before presenting or applying Terraform changes:

Duplicate Check & Merge: Checks for pre-existing alerts in the target folder to ensure changes are merged in-place rather than appended:
- Command:
```
python3 scripts/validate_config.py --directory [TARGET_TF_DIR] --engine-var "${var.reasoning_engine_id}"
```
Config Linting: Validates PromQL grammar, matching engine labels, and HCL structure:
- Command:
```
python3 scripts/validate_config.py --file [PATH_TO_TF_FILE]
```
- Self-Correction Loop: If validation fails (exits non-zero or outputs errors), you MUST read the command output, locate the line/file containing the lint error, analyze the PromQL syntax or Terraform HCL issue, apply adjustments in-place, and re-run the
```
validate_config.py --file
```
  validation. Repeat this loop until the validation script passes successfully.

在展示或应用Terraform更改之前，请使用以下脚本解决重复问题并验证配置：

重复检查与合并: 检查目标文件夹中是否存在已有告警，确保更改是就地合并而非追加：
- 命令:
```
python3 scripts/validate_config.py --directory [TARGET_TF_DIR] --engine-var "${var.reasoning_engine_id}"
```
配置检查: 验证PromQL语法、匹配的引擎标签和HCL结构：
- 命令:
```
python3 scripts/validate_config.py --file [PATH_TO_TF_FILE]
```
- 自我修正循环: 如果验证失败（非零退出或输出错误），你必须读取命令输出，定位包含检查错误的行/文件，分析PromQL语法或Terraform HCL问题，就地应用调整，然后重新运行
```
validate_config.py --file
```
  验证。重复此循环直到验证脚本成功通过。

Gotchas & Behavioral Corrections

注意事项与行为修正

Duration Buffers (Transient Glitches): To avoid alerts firing on transient spikes, use duration/retest window buffers appropriately:
- Reliability Metrics (PromQL / Cloud Monitoring):
  - For short-lookback alerts querying data under 25 hours (e.g., Short-Window Z-Score, Moving Averages, Fast Burn SLO), ALWAYS use a
```
duration = "300s"
```
    (5 minutes) buffer to filter out transient cold start/deployment spikes.
  - For long-lookback alerts querying data longer than 25 hours (e.g., Long-Window Z-Score, Seasonal Decomposition, Slow Burn SLO), duration/retest windows are disabled by the platform. You must not set a duration (omit it entirely).
- Quality Metrics (Standard Filters / Online Monitor):
  - Always use a
```
duration = "300s"
```
    (5 minutes) buffer to filter out transient scoring dips or evaluation outliers caused by temporary LLM judge congestion, or edge-case query outliers.
Dynamic Baseline Adaptation Blind Spot: Explain to users that dynamic statistical Z-score thresholds compare current rates to a moving statistical baseline. If a system degrades slowly over days, the standard baseline curve adapts to this slow drift, making standard Z-score alerts blind to persistent slow errors. Recommend a hard static threshold alert in parallel for strict SLA enforcement.
Seasonal Decomposition Double Alerting: The agent MUST ONLY configure seasonal decomposition alert policies to track spikes (e.g., latency spikes) OR drops AND MUST NOT use dual-direction checks (like absolute deviation). Explain this limitation to the user: comparing to a historical offset (e.g.,
```
offset 1w
```
) the alert policy triggers twice if tracking both directions (once for the anomaly, and once 1 week later when the anomaly becomes the baseline). To prevent this, the generated policy MUST only track either spikes (using
```
>
```
) or drops (using
```
<
```
), avoiding using
```
abs()
```
.
Raw Error Boundaries: Explain that raw error counts or absolute failed request count boundaries do not scale under changing traffic throughput. Recommend ratio-based error rate alerts instead.
Safe Threshold Modulation E2E Validation: When verifying a dynamic metric threshold policy end-to-end, do NOT attempt to force real platform errors. Instead, deploy the alert policy with standard safe bounds (Z-score multiplier > 15), then temporarily update standard deviation Z-score limits to a negative value (e.g. > -3) to trigger/verify the "Firing" state before reverting. Always get confirmation before taking this action proactively.
Expected Script Failures:
- ```
validate_config.py --directory
```
  exiting with code 1: Parse the JSON output for duplicate resource targets. Perform in-place upgrade edits, then re-check until it passes with 0.
- Script Execution Failures & Self-Correction: If the execution of utility scripts (such as
```
check_telemetry.py
```
  ,
```
create_online_monitor.py
```
  , or
```
analyze_traffic.py
```
  ) fails unexpectedly, you MUST read and inspect the stdout/stderr logs or error output. Analyze the error message (e.g., connection timeouts, invalid permissions, or missing resources) and attempt to dynamically correct parameters (such as verifying or correcting the region, project ID, or resource name format) and retry execution before escalating or falling back to manual plans.
Distribution Metric Aligner Constraint: Standard
```
ALIGN_MEAN
```
cannot be applied to
```
DELTA
```
distribution metrics like
```
online_evaluator/scores
```
. You MUST use percentile-based aligners (like
```
ALIGN_PERCENTILE_50
```
) to reduce the score distribution into a comparable numeric stream.

持续时间缓冲（瞬时波动）: 为避免告警因瞬时峰值触发，请适当使用持续时间/重新测试窗口缓冲：
- 可靠性指标（PromQL / Cloud Monitoring）:
  - 对于查询25小时以内数据的短回溯告警（如短窗口Z分数、移动平均值、快速消耗SLO），始终使用
```
duration = "300s"
```
    （5分钟）缓冲，以过滤掉瞬时冷启动/部署峰值。
  - 对于查询超过25小时数据的长回溯告警（如长窗口Z分数、季节性分解、慢速消耗SLO），平台会禁用持续时间/重新测试窗口。你必须不设置持续时间（完全省略）。
- 质量指标（标准过滤器 / 在线监控器）:
  - 始终使用
```
duration = "300s"
```
    （5分钟）缓冲，以过滤掉因临时LLM评估器拥堵或边缘查询异常导致的瞬时分数下降或评估异常值。
动态基线适配盲区: 向用户说明，动态统计Z分数阈值会将当前速率与移动统计基线进行比较。如果系统在数天内缓慢退化，标准基线曲线会适应这种缓慢漂移，导致标准Z分数告警无法检测到持续的缓慢错误。建议并行配置硬静态阈值告警以严格执行SLA。
季节性分解重复告警: 代理必须仅配置季节性分解告警策略来跟踪峰值（如延迟峰值）或下降，不得使用双向检查（如绝对偏差）。向用户解释此限制：与历史偏移（如
```
offset 1w
```
）比较时，如果跟踪双向，告警策略会触发两次（一次是异常发生时，另一次是1周后异常成为基线时）。为防止这种情况，生成的策略必须仅跟踪峰值（使用
```
>
```
）或下降（使用
```
<
```
），避免使用
```
abs()
```
。
原始错误边界: 向用户说明，原始错误计数或绝对失败请求计数边界在流量吞吐量变化时无法扩展。建议使用基于比率的错误率告警。
安全阈值调制端到端验证: 在端到端验证动态指标阈值策略时，请勿尝试强制触发真实平台错误。相反，使用标准安全边界（Z分数乘数>15）部署告警策略，然后临时将标准差Z分数限制更新为负值（如>-3）以触发/验证“触发”状态，之后再恢复。主动执行此操作前必须获得确认。
预期脚本失败:
- ```
validate_config.py --directory
```
  以代码1退出：解析JSON输出以查找重复资源目标。执行就地升级编辑，然后重新检查直到以0退出。
- 脚本执行失败与自我修正: 如果实用脚本（如
```
check_telemetry.py
```
  、
```
create_online_monitor.py
```
  或
```
analyze_traffic.py
```
  ）意外执行失败，你必须读取并检查stdout/stderr日志或错误输出。分析错误消息（如连接超时、无效权限或资源缺失），尝试动态修正参数（如验证或修正区域、项目ID或资源名称格式），然后重试执行，再升级或回退到手动计划。
分布指标对齐器约束: 标准
```
ALIGN_MEAN
```
无法应用于
```
DELTA
```
分布指标（如
```
online_evaluator/scores
```
）。你必须使用基于分位数的对齐器（如
```
ALIGN_PERCENTILE_50
```
）将分数分布转换为可比较的数值流。

agent-platform-alert-configuration

Original

Translation

Agent Platform Alert Configuration

Agent Platform告警配置

Safety & Confirmation Tiers (CRITICAL)

安全与确认层级（CRITICAL）

CRITICAL RULES

关键规则

Algorithm Selection & Policy Mapping Process

算法选择与策略映射流程

Telemetry Metrics and PromQL Examples

遥测指标与PromQL示例

Agent Quality Metrics (Online Monitor)

代理质量指标（在线监控器）

Metric Details & Aligners

指标详情与对齐器

Metric Filter Example

指标过滤器示例

Online Monitor & Telemetry Provisioning

在线监控器与遥测配置

Prerequisites & Dependencies

前提条件与依赖项

Verify Telemetry Status

验证遥测状态

Provision the Online Monitor

配置在线监控器

Formatting the Execution Plan

执行计划格式

Execution Plan: Online Monitor Provisioning

执行计划：在线监控器配置

Tooling Scripts

工具脚本

Gotchas & Behavioral Corrections

注意事项与行为修正

Supporting Links

支持链接