gke-ai-troubleshooting-handle-disruption-gpu-tpu

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Handle Disruption on GPUs and TPUs Troubleshooting

GPU和TPU的中断问题处理与故障排查

🔍 Diagnostic Workflow

🔍 诊断流程

Step 0: Context Acquisition

步骤0：上下文收集

Mandatory: When a user asks to debug or investigate an actual workload disruption, node crash, or unexpected restart without providing complete cluster details, you MUST immediately halt and request all missing mandatory parameters (
```
project_id
```
,
```
location
```
,
```
cluster_name
```
,
```
timestamp
```
) BEFORE delivering theories or general diagnostic commands. Only skip context acquisition if the user explicitly requests a generic reusable runbook or provides a complete static telemetry/log dump for offline analysis.

Optional:

node_name

workload_name

workload_namespace

nodepool_name

必填项：当用户请求调试或调查实际工作负载中断、节点崩溃或意外重启但未提供完整集群详情时，必须立即暂停并请求所有缺失的必填参数（
```
project_id
```
、
```
location
```
、
```
cluster_name
```
、
```
timestamp
```
），之后再给出推测或通用诊断命令。仅当用户明确请求通用可复用运行手册，或提供了完整的静态遥测/日志转储用于离线分析时，才可跳过上下文收集步骤。

可选项：

node_name

、

workload_name

、

workload_namespace

、

nodepool_name

。

Step 1: [Low Risk] Check for Upcoming Scheduled Maintenance

步骤1：[低风险] 检查即将到来的计划内维护

Action: Propose running
```
kubectl
```
to check if nodes have the scheduled maintenance label indicating an upcoming disruption.

Example Command:

bash

kubectl get nodes -l cloud.google.com/scheduled-maintenance-time -L cloud.google.com/scheduled-maintenance-time

Interpretation: The
```
SCHEDULED-MAINTENANCE-TIME
```
column shows the Unix epoch time when the VM is scheduled for maintenance. If this label exists, a disruption is guaranteed to occur.

操作：建议运行
```
kubectl
```
命令检查节点是否带有表示即将发生中断的计划内维护标签。

示例命令：

bash

kubectl get nodes -l cloud.google.com/scheduled-maintenance-time -L cloud.google.com/scheduled-maintenance-time

解读：
```
SCHEDULED-MAINTENANCE-TIME
```
列显示VM计划进行维护的Unix时间戳。如果存在此标签，则中断必然会发生。

Step 2: [Low Risk] Investigation via Cloud Monitoring (PromQL)

步骤2：[低风险] 通过Cloud Monitoring（PromQL）进行调查

Action: Call any available monitoring tool or provide PromQL for manual verification.
Mandatory Monitoring Rule: Whenever recommending follow-up monitoring or interruption tracking over time, you MUST explicitly present a PromQL query using the metric
```
kubernetes_io:node_interruption_count
```
filtered by
```
interruption_reason="HW/SW Maintenance"
```
. Do not suggest general Cloud Monitoring dashboards or Metrics Explorer without providing this specific PromQL metric expression.

Example Query:

promql

# Fetch host maintenance events for nodes
sum by (interruption_type,interruption_reason)( sum_over_time( kubernetes_io:node_interruption_count{monitored_resource="k8s_node", interruption_reason="HW/SW Maintenance"}[${__interval}]))

promql

# See the interruption count aggregated by node pool
sum by (node_pool_name,interruption_type,interruption_reason)( sum_over_time( kubernetes_io:node_pool_interruption_count{monitored_resource="k8s_node_pool", interruption_reason="HW/SW Maintenance", node_pool_name="{nodepool_name}" }[${__interval}]))

Interpretation: If
```
kubernetes_io:node_interruption_count
```
shows values > 0 for
```
interruption_reason="HW/SW Maintenance"
```
, it indicates the underlying Compute Engine VM was interrupted due to scheduled host maintenance.

操作：调用可用的监控工具或提供PromQL供手动验证。
监控规则必填项：每当建议后续监控或长期中断跟踪时，必须明确提供一个使用
```
kubernetes_io:node_interruption_count
```
指标并按
```
interruption_reason="HW/SW Maintenance"
```
过滤的PromQL查询。不得仅建议通用Cloud Monitoring仪表板或Metrics Explorer而不提供此特定PromQL指标表达式。

示例查询：

promql

# Fetch host maintenance events for nodes
sum by (interruption_type,interruption_reason)( sum_over_time( kubernetes_io:node_interruption_count{monitored_resource="k8s_node", interruption_reason="HW/SW Maintenance"}[${__interval}]))

promql

# See the interruption count aggregated by node pool
sum by (node_pool_name,interruption_type,interruption_reason)( sum_over_time( kubernetes_io:node_pool_interruption_count{monitored_resource="k8s_node_pool", interruption_reason="HW/SW Maintenance", node_pool_name="{nodepool_name}" }[${__interval}]))

解读：如果
```
kubernetes_io:node_interruption_count
```
中
```
interruption_reason="HW/SW Maintenance"
```
的数值大于0，则表明底层Compute Engine VM因计划内主机维护而中断。

Step 3: [Low Risk] Investigation via Cloud Logging & Node Taints

步骤3：[低风险] 通过Cloud Logging与Node Taints进行调查

Action: Call
```
query_logs
```
or instruct the user to filter their GKE logs for active host maintenance events, and check node taints.
Guidance: Look for occurrences in Cloud Logging where
```
cloud.google.com/active-node-maintenance
```
is set to
```
ONGOING
```
. To check if GKE has cordoned the terminating node to prevent new workloads from being scheduled, verify whether the
```
cloud.google.com/impending-node-termination:NoSchedule
```
taint is present (either in GKE event logs or directly via
```
kubectl describe node
```
).
Interpretation:
- ```
cloud.google.com/active-node-maintenance
```
  set to
```
ONGOING
```
  means workloads are actively being stopped by GKE due to host maintenance.
- ```
cloud.google.com/impending-node-termination:NoSchedule
```
  taint means GKE has cordoned the node to prevent new Pods from being scheduled on the terminating node. DO NOT recommend tolerating this taint.

操作：调用
```
query_logs
```
或指导用户筛选GKE日志以查找正在进行的主机维护事件，并检查node taints。
指导：在Cloud Logging中查找
```
cloud.google.com/active-node-maintenance
```
设置为
```
ONGOING
```
的记录。要检查GKE是否已封锁（cordoned）即将终止的节点以防止新工作负载被调度，需验证
```
cloud.google.com/impending-node-termination:NoSchedule
```
taint是否存在（可在GKE事件日志中或直接通过
```
kubectl describe node
```
查看）。
解读：
- ```
cloud.google.com/active-node-maintenance
```
  设置为
```
ONGOING
```
  意味着GKE正因主机维护主动停止工作负载。
- ```
cloud.google.com/impending-node-termination:NoSchedule
```
  taint表示GKE已封锁该节点，防止新Pods被调度到即将终止的节点上。请勿建议容忍此taint。

Step 4: Conclusion and Resolution

步骤4：结论与解决方案

Action: Provide a summary of findings to the user and suggest appropriate mitigation strategies if host maintenance events were confirmed or scheduled.
Reporting Rule: Signal Only. Report high-signal information indicating that the disruption was caused by Compute Engine host maintenance, specifically affecting the underlying GPU/TPU nodes. DO NOT dump raw logs.
Negative Findings Rule-Out: If node scheduled-maintenance labels, PromQL interruption counts, and active maintenance logs all return negative/empty results, definitively conclude that Compute Engine host maintenance did NOT cause the disruption. Direct the user to investigate application-level causes (such as OOMKill events, CUDA runtime errors, or resource limits) and do not propose host maintenance mitigations as the primary resolution.
Mandatory Workload Protection Triad: Whenever host maintenance is identified or anticipated on GPU/TPU nodes, consistently recommend all three complementary mitigations together:
1. Configure Graceful Termination: For workloads that need time to save state (e.g., ML frameworks checkpointing via Orbax), follow the guide to Enable disruption handling and set
```
spec.terminationGracePeriodSeconds
```
  (up to 60 minutes) to handle the
```
SIGTERM
```
  signal before node shutdown.
2. Enable Opportunistic Maintenance: To automatically trigger maintenance when GKE detects that GPU/TPU nodes are idle, configure Opportunistic Maintenance.
3. Configure PodDisruptionBudgets (PDBs): Ensure your workload uses a
```
PodDisruptionBudget
```
  to maintain
```
minAvailable
```
  replicas during evictions and disruptions.

操作：向用户提供调查结果摘要，并在确认或预测到主机维护事件时建议适当的缓解策略。
报告规则：仅提供关键信号。报告表明中断由Compute Engine主机维护导致的高信号信息，特别是针对底层GPU/TPU节点的影响。请勿转储原始日志。
阴性结果排除规则：如果节点计划内维护标签、PromQL中断计数和正在进行的维护日志均返回阴性/空结果，则明确得出结论：Compute Engine主机维护未导致此次中断。引导用户调查应用层面的原因（如OOMKill事件、CUDA运行时错误或资源限制），且不得将主机维护缓解策略作为主要解决方案提出。
工作负载保护三重策略必填项：每当在GPU/TPU节点上识别到或预测到主机维护时，必须一致地同时推荐以下三种互补缓解策略：
1. 配置优雅终止：对于需要时间保存状态的工作负载（如通过Orbax进行检查点保存的ML框架），遵循启用中断处理指南，设置
```
spec.terminationGracePeriodSeconds
```
  （最长60分钟）以在节点关闭前处理
```
SIGTERM
```
  信号。
2. 启用机会性维护：为在GKE检测到GPU/TPU节点空闲时自动触发维护，配置机会性维护。
3. 配置PodDisruptionBudgets (PDBs)：确保工作负载使用
```
PodDisruptionBudget
```
  以在驱逐和中断期间维持
```
minAvailable
```
  副本数。