gke-ai-troubleshooting-jobset-interruption

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

GKE JobSet Interruption Troubleshooting

GKE JobSet中断故障排查

Use this skill to systematically diagnose and resolve JobSet interruptions, restarts, and preemptions on GKE clusters hosting large-scale AI/ML workloads.

使用本技能可系统性地诊断并解决承载大规模AI/ML工作负载的GKE集群上的JobSet中断、重启和抢占问题。

⚠️ Prerequisites & Sandbox Rules

⚠️ 前提条件与沙箱规则

JobSet metrics package must be enabled in
```
kube-state-metrics
```
for your cluster.
Cloud Logging and Cloud Monitoring enabled for the Google Cloud Project.
Autonomous & Sandbox Execution Rule: If API queries, tools, or commands encounter
```
403 Permission Denied
```
, authentication errors, or network isolation, do NOT enter authentication or credential troubleshooting loops. Populate the query templates with the acquired variables (
```
{project_id}
```
,
```
{cluster_name}
```
,
```
{workload_name}
```
,
```
{start_time}
```
,
```
{end_time}
```
), inspect any locally staged telemetry or mock data files if available, and complete the diagnostic workflow and resolution recommendations autonomously.

集群的
```
kube-state-metrics
```
中必须启用JobSet指标包。
Google Cloud项目已启用Cloud Logging和Cloud Monitoring。
自主执行与沙箱规则：若API查询、工具或命令遇到
```
403 Permission Denied
```
、认证错误或网络隔离问题，请勿进入认证或凭证排查循环。将获取到的变量（
```
{project_id}
```
、
```
{cluster_name}
```
、
```
{workload_name}
```
、
```
{start_time}
```
、
```
{end_time}
```
）填充到查询模板中，检查本地可用的遥测数据或模拟数据文件（如有），自主完成诊断流程并给出解决方案建议。

🔍 Diagnostic Workflow

🔍 诊断流程

Step 0: Context Acquisition & Time Window Definition

步骤0：上下文获取与时间窗口定义

Independently gather context using tools, workspace files, environment details, or user prompt context:

Project ID (
```
{project_id}
```
)
Cluster Name (
```
{cluster_name}
```
)
Workload Name (JobSet Name) (
```
{workload_name}
```
)
Workload Namespace (
```
{namespace}
```
)
Issue Time (
```
{issue_time}
```
)

If specific variables are not explicitly provided by the user, inspect cluster resources or logs to determine them, or use the

{variable}

placeholders provided.

通过工具、工作区文件、环境详情或用户提示上下文自主收集以下信息：

项目ID（
```
{project_id}
```
）
集群名称（
```
{cluster_name}
```
）
工作负载名称（JobSet名称）（
```
{workload_name}
```
）
工作负载命名空间（
```
{namespace}
```
）
问题发生时间（
```
{issue_time}
```
）

若用户未明确提供特定变量，可检查集群资源或日志来确定，或使用提供的

{variable}

占位符。

Time Handling Rules

时间处理规则

Autonomous Time Window: If a relative time (e.g., "X minutes ago") or no exact timestamp is provided, calculate the query window based on current time or available log timestamps.
Window Calculation: If a timestamp
```
{issue_time}
```
is available (or calculated as
```
T
```
), set
```
{start_time}
```
=
```
T - 30m
```
and
```
{end_time}
```
=
```
T + 30m
```
.

自主时间窗口：若提供的是相对时间（如“X分钟前”）或无确切时间戳，则根据当前时间或可用日志时间戳计算查询窗口。
窗口计算：若有时间戳
```
{issue_time}
```
（或计算为
```
T
```
），则设置
```
{start_time}
```
=
```
T - 30m
```
，
```
{end_time}
```
=
```
T + 30m
```
。

Step 1: Identify JobSet Restarts and Attempts [Low Risk]

步骤1：识别JobSet重启与尝试次数 [低风险]

Verify if the JobSet is experiencing restart loops and determine the frequency of restarts.

验证JobSet是否存在重启循环，并确定重启频率。

Visual Chart / MQL Query - restarts

可视化图表 / MQL查询 - 重启次数

MQL Query Specification:

mql

fetch prometheus_target
| metric 'prometheus.googleapis.com/kube_jobset_restarts/gauge'
| filter resource.cluster_name == '{cluster_name}' && metric.jobset_name == '{workload_name}'
| align next_older(1m)
| every 1m
| group_by [metric.jobset_name], [val: max(value)]

MQL查询规范：

mql

fetch prometheus_target
| metric 'prometheus.googleapis.com/kube_jobset_restarts/gauge'
| filter resource.cluster_name == '{cluster_name}' && metric.jobset_name == '{workload_name}'
| align next_older(1m)
| every 1m
| group_by [metric.jobset_name], [val: max(value)]

PromQL Metric Query - restarts

PromQL指标查询 - 重启次数

PromQL Query Specification:

promql

kube_jobset_restarts{jobset_name="{workload_name}", cluster="{cluster_name}"}

Diagnostic Logic: A non-zero or increasing value for restarts indicates that the JobSet is being actively restarted by the controller due to worker failure or interruption.
Automation: Proceed to Step 2 automatically after reporting findings.

PromQL查询规范：

promql

kube_jobset_restarts{jobset_name="{workload_name}", cluster="{cluster_name}"}

诊断逻辑：重启次数为非零值或呈上升趋势，表明JobSet因工作进程故障或中断正被控制器主动重启。
自动化流程：报告结果后自动进入步骤2。

Step 2: Inspect Nodepool Interruptions [Low Risk]

步骤2：检查节点池中断事件 [低风险]

Determine if the JobSet restarts were triggered by physical nodepool-level events (such as spot preemptions, maintenance, or host terminations).

确定JobSet重启是否由物理节点池层面的事件触发（如Spot实例抢占、维护或主机终止）。

A. Metrics Query (Nodepool Interruption Counts)

A. 指标查询（节点池中断次数）

Visual Chart / MQL Query - interruptions

可视化图表 / MQL查询 - 中断次数

MQL Query Specification:

mql

fetch k8s_node_pool
| metric 'kubernetes.io/node_pool/interruption_count'
| filter cluster_name == '{cluster_name}'
| align next_older(10m)
| every 10m
| group_by [metric.interruption_type, metric.interruption_reason, metadata.system.node_pool_name], [val: sum(value)]

MQL查询规范：

mql

fetch k8s_node_pool
| metric 'kubernetes.io/node_pool/interruption_count'
| filter cluster_name == '{cluster_name}'
| align next_older(10m)
| every 10m
| group_by [metric.interruption_type, metric.interruption_reason, metadata.system.node_pool_name], [val: sum(value)]

PromQL Query - interruptions

PromQL查询 - 中断次数

PromQL Query Specification:

promql

sum by (interruption_type, interruption_reason, node_pool_name, cluster_name) (
  avg_over_time(kubernetes_io:node_pool_interruption_count{cluster_name="{cluster_name}"}[10m])
)

PromQL查询规范：

promql

sum by (interruption_type, interruption_reason, node_pool_name, cluster_name) (
  avg_over_time(kubernetes_io:node_pool_interruption_count{cluster_name="{cluster_name}"}[10m])
)

B. Log Query (Nodepool Life Events)

B. 日志查询（节点池生命周期事件）

LQL Log Filter Specification:

sql

resource.type="gke_nodepool"
AND resource.labels.cluster_name="{cluster_name}"
AND timestamp >= "{start_time}"
AND timestamp <= "{end_time}"

Diagnostic Logic:
- PreemptionEvent: Spot VMs were preempted, or node was scale-down.
- MaintenanceEvent: Node pool updated or Google scheduled maintenance.
- TerminationEvent: Serious host failures. Check
```
interruption_reason
```
  or logs for host issues.
- See Failure Signatures for examples of node termination logs and preemption events.
Automation: Proceed to Step 3 automatically.

LQL日志过滤规范：

sql

resource.type="gke_nodepool"
AND resource.labels.cluster_name="{cluster_name}"
AND timestamp >= "{start_time}"
AND timestamp <= "{end_time}"

诊断逻辑：
- PreemptionEvent：Spot VM被抢占，或节点被缩容。
- MaintenanceEvent：节点池更新或Google计划内维护。
- TerminationEvent：严重主机故障。检查
```
interruption_reason
```
  或日志获取主机问题详情。
- 可参考故障特征查看节点终止日志和抢占事件示例。
自动化流程：自动进入步骤3。

Step 3: Inspect Nodes and Underlying Host VMs [Low Risk]

步骤3：检查节点与底层主机VM [低风险]

Correlate node readiness failures with physical host VMs to see if a single faulty host repeatedly fails coordinator pods.

将节点就绪失败与物理主机VM关联，查看是否存在单个故障主机反复导致协调器Pod失败的情况。

A. Metrics Query (Node Ready Status Check)

A. 指标查询（节点就绪状态检查）

Visual Chart / MQL Query - node status

可视化图表 / MQL查询 - 节点状态

MQL Query Specification:

mql

fetch k8s_node
| metric 'kubernetes.io/node/status_condition'
| filter cluster_name == '{cluster_name}' && metric.condition == 'Ready' && metric.status == 'False'
| align next_older(1m)
| every 1m
| group_by [node_name, metadata.user.gke_nodepool], [val: max(value)]

MQL查询规范：

mql

fetch k8s_node
| metric 'kubernetes.io/node/status_condition'
| filter cluster_name == '{cluster_name}' && metric.condition == 'Ready' && metric.status == 'False'
| align next_older(1m)
| every 1m
| group_by [node_name, metadata.user.gke_nodepool], [val: max(value)]

PromQL Query - node status

PromQL查询 - 节点状态

PromQL Query Specification:

promql

sum by (status, condition, node_pool_name) (
  kubernetes_io:node_status_condition{cluster_name="{cluster_name}", condition="Ready", status="False"}
)

PromQL查询规范：

promql

sum by (status, condition, node_pool_name) (
  kubernetes_io:node_status_condition{cluster_name="{cluster_name}", condition="Ready", status="False"}
)

B. Metrics Query (Node-to-Host Metadata Topology Correlation)

B. 指标查询（节点到主机元数据拓扑关联）

MQL Query Specification:

mql

fetch k8s_node
| metric 'kubernetes.io/node/cpu/total_cores'
| filter cluster_name == '{cluster_name}'
| align next_older(1m)
| every 1m
| group_by [node_name, metadata.user.gce_topology_host, metadata.user.gke_nodepool], [val: max(value)]

MQL查询规范：

mql

fetch k8s_node
| metric 'kubernetes.io/node/cpu/total_cores'
| filter cluster_name == '{cluster_name}'
| align next_older(1m)
| every 1m
| group_by [node_name, metadata.user.gce_topology_host, metadata.user.gke_nodepool], [val: max(value)]

C. Log Query (Node Fault Logs)

C. 日志查询（节点故障日志）

LQL Log Filter Specification:

sql

resource.type="k8s_node"
AND resource.labels.cluster_name="{cluster_name}"
AND (textPayload:"host error" OR textPayload:"kernel panic" OR textPayload:"hardware failure" OR textPayload:"NodeNotReady")
AND timestamp >= "{start_time}"
AND timestamp <= "{end_time}"

Diagnostic Logic: Identify if specific nodes are unhealthy (
```
Ready=False
```
or
```
Unknown
```
) and correlate them to their GCE physical host ID via
```
metadata.user.gce_topology_host
```
. Check if the same host is repeatedly failing.
Automation: Proceed to Step 4 automatically.

LQL日志过滤规范：

sql

resource.type="k8s_node"
AND resource.labels.cluster_name="{cluster_name}"
AND (textPayload:"host error" OR textPayload:"kernel panic" OR textPayload:"hardware failure" OR textPayload:"NodeNotReady")
AND timestamp >= "{start_time}"
AND timestamp <= "{end_time}"

诊断逻辑：识别是否存在特定节点处于不健康状态（
```
Ready=False
```
或
```
Unknown
```
），并通过
```
metadata.user.gce_topology_host
```
关联其GCE物理主机ID。检查同一主机是否反复出现故障。
自动化流程：自动进入步骤4。

Step 4: Inspect Pod and Worker / Container Failures [Low Risk]

步骤4：检查Pod与工作进程/容器故障 [低风险]

Analyze pod status phases and retrieve coordinator worker logs to identify application-level crashes or network deadlocks.

Required Execution Order: You MUST analyze pod status phases (Section A) and unschedulable pod metrics (Section B) to assess overall workload health before inspecting specific worker container logs (Section C).

分析Pod状态阶段，获取协调器工作进程日志以识别应用层面崩溃或网络死锁问题。

执行顺序要求：在检查特定工作进程容器日志（C部分）之前，必须先分析Pod状态阶段（A部分）和不可调度Pod指标（B部分），以评估整体工作负载健康状况。

A. Metrics Query (Pod Lifecycle Phases)

A. 指标查询（Pod生命周期阶段）

Visual Chart / MQL Query - pod phase

可视化图表 / MQL查询 - Pod阶段

MQL Query Specification:

mql

fetch k8s_pod
| metric 'kubernetes.io/pod/status/phase'
| filter cluster_name == '{cluster_name}' && pod_name ==~ '{workload_name}.*'
| align next_older(10m)
| every 10m
| group_by [metric.phase], [val: count()]

MQL查询规范：

mql

fetch k8s_pod
| metric 'kubernetes.io/pod/status/phase'
| filter cluster_name == '{cluster_name}' && pod_name ==~ '{workload_name}.*'
| align next_older(10m)
| every 10m
| group_by [metric.phase], [val: count()]

PromQL Query - pod phase

PromQL查询 - Pod阶段

PromQL Query Specification:

promql

sum by (phase) (
  avg_over_time(kube_pod_status_phase{cluster="{cluster_name}", pod=~"{workload_name}.*"}[10m])
)

PromQL查询规范：

promql

sum by (phase) (
  avg_over_time(kube_pod_status_phase{cluster="{cluster_name}", pod=~"{workload_name}.*"}[10m])
)

B. Metrics Query (Unschedulable Pod Count)

B. 指标查询（不可调度Pod数量）

MQL Query Specification:

mql

fetch k8s_pod
| metric 'kubernetes.io/pod/status/unschedulable'
| filter cluster_name == '{cluster_name}' && pod_name ==~ '{workload_name}.*'
| align next_older(10m)
| every 10m
| group_by [pod_name], [val: max(value)]

MQL查询规范：

mql

fetch k8s_pod
| metric 'kubernetes.io/pod/status/unschedulable'
| filter cluster_name == '{cluster_name}' && pod_name ==~ '{workload_name}.*'
| align next_older(10m)
| every 10m
| group_by [pod_name], [val: max(value)]

C. Log Query (Worker Container Logs)

C. 日志查询（工作进程容器日志）

LQL Log Filter Specification:

sql

resource.type="k8s_container"
AND resource.labels.cluster_name="{cluster_name}"
AND labels."k8s-pod/jobset_sigs_k8s_io/jobset-name"="{workload_name}"
AND timestamp >= "{start_time}"
AND timestamp <= "{end_time}"

Diagnostic Logic:
1. Check the pod timeline to spot pending or unschedulable pods.
2. Use worker container logs to analyze worker 0 in slice 0 (coordinator) for NCCL timeouts, collective communication issues, or MegaScale hangs.
Automation: Proceed to Resolution.

LQL日志过滤规范：

sql

resource.type="k8s_container"
AND resource.labels.cluster_name="{cluster_name}"
AND labels."k8s-pod/jobset_sigs_k8s_io/jobset-name"="{workload_name}"
AND timestamp >= "{start_time}"
AND timestamp <= "{end_time}"

诊断逻辑：
1. 查看Pod时间线，识别处于Pending或不可调度状态的Pod。
2. 利用工作进程容器日志分析分片0中的工作进程0（协调器）是否存在NCCL超时、集体通信问题或MegaScale挂起情况。
自动化流程：进入解决方案环节。

🛠️ Resolution Workflow

🛠️ 解决方案流程

Resolution 1: Preemption & Autoscaling Optimizations [Low Risk]

方案1：抢占与自动扩缩容优化 [低风险]

If Step 2 showed high preemption counts on Spot VMs:

Action: Suggest switching critical long-running training workloads to GKE Reserved/On-Demand VMs or utilizing Compact Placement Policies to minimize defragmentation interruptions.
Justification: Eliminates spot-market preemptions and reduces training restarts.

若步骤2显示Spot VM存在高抢占次数：

操作建议：建议将关键长期运行的训练工作负载切换为GKE预留/按需VM，或使用紧凑放置策略以减少碎片导致的中断。
理由：消除Spot市场抢占问题，减少训练重启次数。

Resolution 2: Quarantine Faulty Host VMs [High Risk]

方案2：隔离故障主机VM [高风险]

If Step 3 identified a specific host ID (

gce-topology-host

) that consistently fails or triggers restarts across multiple attempts:

Action: Recommend cordoning/draining the GKE node, deleting the underlying GCE VM instance to trigger instance recreation, and opening a support ticket with Google Cloud Support specifying the physical host ID.
Justification: GKE auto-repair will recreate the VM instance on healthy physical hardware, preventing infinite restart loops.

若步骤3识别到特定主机ID（

gce-topology-host

）持续出现故障或触发多次重启：

操作建议：建议封锁/驱逐GKE节点，删除底层GCE VM实例以触发实例重建，并向Google Cloud Support提交工单，注明物理主机ID。
理由：GKE自动修复功能会在健康物理硬件上重建VM实例，避免无限重启循环。

📋 Copypaste Checklist

📋 复制粘贴检查清单

Gather context and compute

{start_time}

(

{issue_time} - 30m

) and

{end_time}

(

{issue_time} + 30m

) window.

Query JobSet restart attempts.
Check Nodepool interruptions (spot preemptions vs. hardware terminations).
Query node-to-host mapping and check node logs for physical host errors.
Inspect pod timeline status and coordinator worker container logs.
Recommend appropriate scheduling strategy (On-demand vs Spot) or host VM quarantining.

收集上下文并计算时间窗口：

{start_time}

（

{issue_time} - 30m

）和

{end_time}

（

{issue_time} + 30m

）。

查询JobSet重启尝试次数。
检查节点池中断事件（Spot抢占 vs 硬件终止）。
查询节点到主机的映射关系，检查节点日志中的物理主机错误。
检查Pod时间线状态和协调器工作进程容器日志。
推荐合适的调度策略（按需 vs Spot）或主机VM隔离方案。