gke-ai-troubleshooting-node-unresponsive-timeout

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Troubleshoot unresponsive GKE TPU and GPU nodes (

NodeStatusUnknown

)

排查无响应的 GKE TPU 和 GPU 节点 (

NodeStatusUnknown

)

When the Compute Engine host of a TPU or GPU node has a fatal hardware error, kernel panic, or non-maskable interrupt (NMI) lockup, the guest OS stops responding. The kubelet can no longer send heartbeats, so the node

Ready

condition becomes

Unknown

with

Reason: NodeStatusUnknown

(

Kubelet stopped posting node status.

). If node auto-repair is disabled on the node pool, GKE doesn't repair the node. The node can stay

NotReady

, and pods on it can stay in

Terminating

, which blocks multi-host

JobSet

workloads from recovering.

当 TPU 或 GPU 节点的 Compute Engine 宿主机发生致命硬件错误、内核 panic 或不可屏蔽中断（NMI）死锁时，客户机操作系统会停止响应。kubelet 无法再发送心跳，因此节点的

Ready

condition 会变为

Unknown

，并带有

Reason: NodeStatusUnknown

（

Kubelet stopped posting node status.

）。如果节点池未启用节点自动修复，GKE 不会修复该节点。节点可能持续处于

NotReady

状态，其上的 Pod 也可能持续处于

Terminating

状态，从而阻碍多宿主机

JobSet

工作负载恢复。

Prerequisites

前置条件

Tools: Install the Google Cloud SDK (
```
gcloud
```
) and
```
kubectl
```
.

Cloud Billing & Project Configuration: Verify an active billing account is linked (

gcloud billing projects describe {project_id}

), authenticate (

gcloud auth login

), set the target project (

gcloud config set project {project_id}

), and ensure

container.googleapis.com

compute.googleapis.com

logging.googleapis.com

, and

monitoring.googleapis.com

are enabled.

Required IAM Roles:
- Kubernetes Engine Viewer (
```
roles/container.viewer
```
  )
- Compute Viewer (
```
roles/compute.viewer
```
  )
- Logs Viewer (
```
roles/logging.viewer
```
  )
- Monitoring Viewer (
```
roles/monitoring.viewer
```
  )
- For remediation (
```
[High Risk]
```
  steps): Kubernetes Engine Cluster Admin (
```
roles/container.clusterAdmin
```
  )
Documentation:
- Troubleshoot nodes with the NotReady status in GKE (Sections: "Check the node's status and conditions", "Confirm node preemption", "Verify that the node has recovered")
- Deploy TPU workloads in GKE Standard (Sections: "Monitor health metrics for TPU nodes and node pools", "Configure auto repair for TPU slice nodes")
- Troubleshoot OOM events
- View GKE logs (Sections: "System logs")
- Viewing serial port output
- Troubleshoot Linux VM boot issues due to kernel panic
- Auto-repair nodes (Sections: "Settings for Autopilot and Standard", "Verify node auto-repair is enabled for a Standard node pool", "Get information about recent automated repair events", "Enable auto-repair for an existing Standard node pool", "Repair criteria", "Node repair process", "Node auto repair in TPU slice nodes")
- Troubleshooting VM shutdowns and reboots (Sections: "Querying Cloud Audit Logs", "Reviewing Cloud Audit Logs")

Read-only rule: Run read-only diagnostic commands only. Never drain, delete, or re-create nodes, or run any other command that changes the cluster. Give the user any fix to apply themselves.

When you recommend a fix, link the doc section that describes it.

工具：安装 Google Cloud SDK（
```
gcloud
```
）和
```
kubectl
```
。

Cloud Billing 与项目配置：确认已关联有效结算账号（

gcloud billing projects describe {project_id}

）、完成身份验证（

gcloud auth login

）、设置目标项目（

gcloud config set project {project_id}

），并确保已启用

container.googleapis.com

、

compute.googleapis.com

、

logging.googleapis.com

和

monitoring.googleapis.com

。

必需的 IAM 角色：
- Kubernetes Engine Viewer（
```
roles/container.viewer
```
  ）
- Compute Viewer（
```
roles/compute.viewer
```
  ）
- Logs Viewer（
```
roles/logging.viewer
```
  ）
- Monitoring Viewer（
```
roles/monitoring.viewer
```
  ）
- 对于修复操作（
```
[高风险]
```
  步骤）：Kubernetes Engine Cluster Admin（
```
roles/container.clusterAdmin
```
  ）
文档：
- 排查 GKE 中处于 NotReady 状态的节点（章节：“检查节点的状态和 condition”、“确认节点是否被抢占”、“验证节点是否已恢复”）
- 在 GKE Standard 中部署 TPU 工作负载（章节：“监控 TPU 节点和节点池的健康指标”、“为 TPU slice 节点配置自动修复”）
- 排查 OOM 事件
- 查看 GKE 日志（章节：“系统日志”）
- 查看串口输出
- 排查因内核 panic 导致的 Linux VM 启动问题
- 自动修复节点（章节：“Autopilot 和 Standard 的设置”、“验证 Standard 节点池是否启用节点自动修复”、“获取最近的自动修复事件信息”、“为现有 Standard 节点池启用自动修复”、“修复条件”、“节点修复流程”、“TPU slice 节点中的节点自动修复”）
- 排查 VM 关闭和重启问题（章节：“查询 Cloud Audit Logs”、“审查 Cloud Audit Logs”）

只读规则：仅运行只读诊断命令。绝不排空（drain）、删除或重新创建节点，也不要运行任何其他会更改集群的命令。应向用户提供修复方案，由用户自行应用。

推荐修复方案时，请链接到描述该方案的文档章节。

Diagnostic workflow

诊断工作流

Step 0: Collect context and set the investigation window

[Low Risk]

步骤 0：收集上下文并设置调查窗口

[低风险]

Collect the target parameters. By default, query a 60-minute window `[T - 30m, T

30m]
```
around
```
{issue_time}`:

```
{project_id}
```
: Google Cloud project ID
```
{cluster_name}
```
: GKE cluster name
```
{location}
```
: Cluster region or zone
```
{nodepool_name}
```
: Target TPU or GPU node pool name
```
{node_name}
```
: Unresponsive GKE node name (and its Compute Engine
```
{zone}
```
)
```
{issue_time}
```
: Incident timestamp in RFC3339 UTC
```
{start_time}
```
:
```
{issue_time} - 30m
```
```
{end_time}
```
:
```
{issue_time} + 30m
```

收集目标参数。默认查询

{issue_time}

前后 60 分钟的窗口

[T - 30m, T + 30m]

：

```
{project_id}
```
：Google Cloud 项目 ID
```
{cluster_name}
```
：GKE 集群名称
```
{location}
```
：集群所在区域或可用区
```
{nodepool_name}
```
：目标 TPU 或 GPU 节点池名称
```
{node_name}
```
：无响应的 GKE 节点名称及其 Compute Engine
```
{zone}
```
```
{issue_time}
```
：事件时间戳（RFC3339 UTC）
```
{start_time}
```
：
```
{issue_time} - 30m
```
```
{end_time}
```
：
```
{issue_time} + 30m
```

Step 1: Verify the

NodeStatusUnknown

heartbeat timeout

[Low Risk]

步骤 1：验证

NodeStatusUnknown

心跳超时

[低风险]

Check Kubernetes node conditions: To inspect the node status and verify whether the
```
Ready
```
condition is
```
Unknown
```
with
```
Reason: NodeStatusUnknown
```
(
```
Kubelet stopped posting node status.
```
), follow the instructions in the section Check the node's status and conditions.
Query Cloud Logging (read-only LQL): Query
```
k8s_node
```
and
```
k8s_cluster
```
logs across
```
[{start_time}, {end_time}]
```
to confirm when the control plane lost heartbeat contact with
```
{node_name}
```
:

(resource.type="k8s_node" OR resource.type="k8s_cluster")
resource.labels.cluster_name="{cluster_name}"
("{node_name}" AND ("NodeNotReady" OR "NodeStatusUnknown" OR "Kubelet stopped posting node status"))
timestamp >= "{start_time}" AND timestamp <= "{end_time}"

Query Cloud Monitoring (read-only PromQL): Correlate the duration of the
```
Unknown
```
state using the GKE system metric
```
kubernetes.io/node/status_condition
```
(
```
kubernetes_io:node_status_condition
```
, GKE
```
1.32.1-gke.1357001+
```
) documented in Monitor health metrics for TPU nodes and node pools, filtered by
```
condition="Ready"
```
and
```
status="Unknown"
```
:

promql

kubernetes_io:node_status_condition{
  monitored_resource="k8s_node",
  cluster_name="{cluster_name}",
  node_name="{node_name}",
  condition="Ready",
  status="Unknown"
}

Decision logic:
- If the node
```
Ready
```
  condition is
```
True
```
  and
```
NodeStatusUnknown
```
  is absent, rule out an unresponsive node timeout and pivot to workload-level troubleshooting (for example, Troubleshoot OOM events) rather than repairing or draining the node.
- If
```
Ready
```
  is
```
Unknown
```
  (
```
NodeStatusUnknown
```
  ), proceed to Step 2.

检查 Kubernetes 节点 condition：按照检查节点的状态和 condition章节中的说明，检查节点状态并确认
```
Ready
```
condition 是否为
```
Unknown
```
，且
```
Reason
```
为
```
NodeStatusUnknown
```
（
```
Kubelet stopped posting node status.
```
）。
查询 Cloud Logging（只读 LQL）：查询
```
[{start_time}, {end_time}]
```
范围内的
```
k8s_node
```
和
```
k8s_cluster
```
日志，确认控制平面何时与
```
{node_name}
```
失去心跳联系：

(resource.type="k8s_node" OR resource.type="k8s_cluster")
resource.labels.cluster_name="{cluster_name}"
("{node_name}" AND ("NodeNotReady" OR "NodeStatusUnknown" OR "Kubelet stopped posting node status"))
timestamp >= "{start_time}" AND timestamp <= "{end_time}"

查询 Cloud Monitoring（只读 PromQL）：使用 GKE 系统指标
```
kubernetes.io/node/status_condition
```
（
```
kubernetes_io:node_status_condition
```
，GKE
```
1.32.1-gke.1357001+
```
）关联
```
Unknown
```
状态的持续时间。该指标记录在监控 TPU 节点和节点池的健康指标中，并按
```
condition="Ready"
```
和
```
status="Unknown"
```
过滤：

promql

kubernetes_io:node_status_condition{
  monitored_resource="k8s_node",
  cluster_name="{cluster_name}",
  node_name="{node_name}",
  condition="Ready",
  status="Unknown"
}

决策逻辑：
- 如果节点的
```
Ready
```
  condition 为
```
True
```
  ，且未出现
```
NodeStatusUnknown
```
  ，则排除节点无响应超时问题，转而排查工作负载层问题（例如排查 OOM 事件），不要修复或排空节点。
- 如果
```
Ready
```
  为
```
Unknown
```
  （
```
NodeStatusUnknown
```
  ），继续执行步骤 2。

Step 2: Inspect serial port output for a kernel panic

[Low Risk]

步骤 2：检查串口输出中的内核 panic

[低风险]

The guest OS can't send logs after a fatal kernel freeze, so check the serial port output of the node's VM:

Cloud Logging: If serial port logging is enabled (i.e. VM metadata
```
serial-port-logging-enable
```
is set to
```
true
```
), GKE system logs in Cloud Logging include the node's serial port output. See the section System logs. Use Cloud Logging when the VM is stopped or has already been replaced by auto-repair, or when you need more than the most recent output.
Running VM: Follow Viewing serial port output to retrieve the serial port 1 output (
```
gcloud compute instances get-serial-port-output
```
with
```
--port=1
```
) for
```
{node_name}
```
in
```
{zone}
```
. This method returns only the most recent 1 MB of output per port.

Consult Troubleshoot Linux VM boot issues due to kernel panic to identify documented kernel panic and hardware crash patterns (such as

Fatal Machine check

hung_task: blocked tasks

, or

NMI: Not continuing

) in the serial port output.

客户机操作系统在发生致命内核冻结后无法发送日志，因此应检查节点 VM 的串口输出：

Cloud Logging：如果已启用串口日志记录（即 VM 元数据
```
serial-port-logging-enable
```
设置为
```
true
```
），Cloud Logging 中的 GKE 系统日志会包含节点的串口输出。请参阅系统日志章节。当 VM 已停止、已被自动修复替换，或者需要查看最近一次输出之外的内容时，使用 Cloud Logging。
运行中的 VM：按照查看串口输出中的说明，获取
```
{zone}
```
中
```
{node_name}
```
的串口 1 输出（使用
```
gcloud compute instances get-serial-port-output
```
并指定
```
--port=1
```
）。此方法每个端口仅返回最近的 1 MB 输出。

查阅排查因内核 panic 导致的 Linux VM 启动问题，识别串口输出中已有文档说明的内核 panic 和硬件崩溃模式，例如

Fatal Machine check

、

hung_task: blocked tasks

或

NMI: Not continuing

。

Step 3: Check node auto-repair status

[Low Risk]

步骤 3：检查节点自动修复状态

[低风险]

Find out why GKE hasn't repaired the unresponsive node:

Autopilot clusters: Autopilot always repairs nodes, and you can't turn this off (see Settings for Autopilot and Standard). Skip the configuration check and check the repair history.
Standard clusters: Follow Verify node auto-repair is enabled for a Standard node pool to check whether
```
autoRepair
```
is enabled on
```
{nodepool_name}
```
.
Repair history (both modes): Auto-repair can be enabled and the repair can still fail. Per Configure auto repair for TPU slice nodes, check the repair status, including the failure reason, in the operation history described in Get information about recent automated repair events. If the failure is caused by insufficient quota, tell the user to contact their Google Cloud account representative to increase the quota.

查明 GKE 尚未修复无响应节点的原因：

Autopilot 集群：Autopilot 始终会修复节点，且无法关闭此功能（请参阅Autopilot 和 Standard 的设置）。跳过配置检查，直接检查修复历史。
Standard 集群：按照验证 Standard 节点池是否启用节点自动修复中的说明，检查
```
{nodepool_name}
```
是否启用了
```
autoRepair
```
。
修复历史（两种模式均适用）：即使启用了自动修复，修复仍可能失败。按照为 TPU slice 节点配置自动修复中的说明，在获取最近的自动修复事件信息所描述的操作历史中检查修复状态及失败原因。如果失败原因是配额不足，应告知用户联系其 Google Cloud 客户代表以提高配额。

Step 4: Check Compute Engine system events

[Low Risk]

步骤 4：检查 Compute Engine 系统事件

[低风险]

To list the system events for

{node_name}

around

{issue_time}

, follow the instructions in the section Querying Cloud Audit Logs. Compare the

method

field with the table in the "Reviewing Cloud Audit Logs" section of the same document, and look for:

```
compute.instances.hostError
```
: a hardware or software issue on the physical host caused the VM to crash.
```
compute.instances.preempted
```
: Compute Engine preempted a Spot VM or preemptible VM. For preempted nodes, also see Confirm node preemption.

compute.instances.automaticRestart

: Compute Engine restarted the VM after a

hostError

terminateOnHostMaintenance

event.

```
compute.instances.guestTerminate
```
: the VM's operating system initiated the shutdown.

按照查询 Cloud Audit Logs章节中的说明，列出

{issue_time}

前后与

{node_name}

相关的系统事件。将

method

字段与同一文档“审查 Cloud Audit Logs”章节中的表格进行对比，并查找以下事件：

```
compute.instances.hostError
```
：物理宿主机上的硬件或软件问题导致 VM 崩溃。
```
compute.instances.preempted
```
：Compute Engine 抢占了一个 Spot VM 或可抢占 VM。对于被抢占的节点，另请参阅确认节点是否被抢占。

compute.instances.automaticRestart

：Compute Engine 在发生

hostError

或

terminateOnHostMaintenance

事件后重启了 VM。

```
compute.instances.guestTerminate
```
：VM 的操作系统主动发起关闭。

Step 5: Resolution

[High Risk]

步骤 5：解决

[高风险]

Guardrails:
Never force-delete stuck
Terminating
pods on an unresponsive node. Force deletion doesn't wait for the kubelet to confirm that the pod has stopped, so a replacement pod can start while the old one is still running. See Force Delete StatefulSet Pods.
Never delete GKE-managed Compute Engine VM instances directly (
gcloud compute instances delete
). Instead, check how long the node has reported
NodeStatusUnknown
(Step 1), check the serial console output (Step 2), and rely on node auto-repair, as described in the following steps.

Enable node auto-repair (Standard clusters only):
- Autopilot clusters always auto-repair nodes, so skip this step.
- Before the user enables auto-repair on a multi-host TPU slice node pool, tell them that GKE re-creates the entire node pool when a node in it needs repair (see step 2).
- If
```
autoRepair
```
  is disabled on
```
{nodepool_name}
```
  , link the user to Enable auto-repair for an existing Standard node pool and Configure auto repair for TPU slice nodes so they can apply the change themselves.
Let GKE repair the node:
- Explain that GKE repairs a node that reports
```
NotReady
```
  or no status for the documented time threshold by draining and re-creating it, and link the "Repair criteria" and "Node repair process" sections of Auto-repair nodes. GKE waits one hour for the drain to complete. If the drain doesn't complete, GKE shuts the node down and creates a new node. Tell the user to expect this one-hour drain wait before GKE re-creates the node.
- For multi-host TPU slice node pools, note per Node auto repair in TPU slice nodes that the entire node pool is re-created.
Verify node recovery:
- After the repair completes, follow the instructions in the section Verify that the node has recovered to verify that the repaired node returns to
```
Ready
```
  status, then confirm that the
```
JobSet
```
  pods are running again.

防护规则：
绝不强制删除无响应节点上卡在
Terminating
状态的 Pod。强制删除不会等待 kubelet 确认 Pod 已停止，因此替代 Pod 可能在旧 Pod 仍在运行时启动。请参阅强制删除 StatefulSet Pod。
绝不直接删除由 GKE 管理的 Compute Engine VM 实例（
gcloud compute instances delete
）。应改为检查节点报告
NodeStatusUnknown
的时长（步骤 1）、检查串口控制台输出（步骤 2），并依赖节点自动修复，具体方式如下。

启用节点自动修复（仅适用于 Standard 集群）：
- Autopilot 集群始终会自动修复节点，因此跳过此步骤。
- 在用户为多宿主机 TPU slice 节点池启用自动修复之前，应告知用户：当其中某个节点需要修复时，GKE 会重新创建整个节点池（请参阅步骤 2）。
- 如果
```
{nodepool_name}
```
  未启用
```
autoRepair
```
  ，请向用户提供为现有 Standard 节点池启用自动修复和为 TPU slice 节点配置自动修复的文档链接，以便其自行应用更改。
让 GKE 修复节点：
- 说明 GKE 会在节点报告
```
NotReady
```
  或无状态持续达到文档规定的时间阈值后，通过排空并重新创建节点来进行修复，并链接到自动修复节点中的“修复条件”和“节点修复流程”章节。 GKE 会等待一小时以完成节点排空。如果排空未完成，GKE 将关闭该节点并创建新节点。应提醒用户，GKE 重新创建节点前可能会经历这一小时的排空等待时间。
- 对于多宿主机 TPU slice 节点池，请根据 TPU slice 节点中的节点自动修复说明，整个节点池都会被重新创建。
验证节点恢复情况：
- 修复完成后，按照验证节点是否已恢复章节中的说明，验证已修复节点是否恢复为
```
Ready
```
  状态，然后确认
```
JobSet
```
  Pod 已重新运行。

gke-ai-troubleshooting-node-unresponsive-timeout

Original

Translation

Troubleshoot unresponsive GKE TPU and GPU nodes (
`NodeStatusUnknown`
)

排查无响应的 GKE TPU 和 GPU 节点 (
`NodeStatusUnknown`
)

Prerequisites

前置条件

Diagnostic workflow

诊断工作流

Step 0: Collect context and set the investigation window
`[Low Risk]`

步骤 0：收集上下文并设置调查窗口
`[低风险]`

Step 1: Verify the
`NodeStatusUnknown`
heartbeat timeout
`[Low Risk]`

步骤 1：验证
`NodeStatusUnknown`
心跳超时
`[低风险]`

Step 2: Inspect serial port output for a kernel panic
`[Low Risk]`

步骤 2：检查串口输出中的内核 panic
`[低风险]`

Step 3: Check node auto-repair status
`[Low Risk]`

步骤 3：检查节点自动修复状态
`[低风险]`

Step 4: Check Compute Engine system events
`[Low Risk]`

步骤 4：检查 Compute Engine 系统事件
`[低风险]`

Step 5: Resolution
`[High Risk]`

步骤 5：解决
`[高风险]`

gke-ai-troubleshooting-node-unresponsive-timeout

Original

Translation

Troubleshoot unresponsive GKE TPU and GPU nodes (NodeStatusUnknown)

排查无响应的 GKE TPU 和 GPU 节点 (NodeStatusUnknown)

Prerequisites

前置条件

Diagnostic workflow

诊断工作流

Step 0: Collect context and set the investigation window [Low Risk]

步骤 0：收集上下文并设置调查窗口 [低风险]

Step 1: Verify the NodeStatusUnknown heartbeat timeout [Low Risk]

步骤 1：验证 NodeStatusUnknown 心跳超时 [低风险]

Step 2: Inspect serial port output for a kernel panic [Low Risk]

步骤 2：检查串口输出中的内核 panic [低风险]

Step 3: Check node auto-repair status [Low Risk]

步骤 3：检查节点自动修复状态 [低风险]

Step 4: Check Compute Engine system events [Low Risk]

步骤 4：检查 Compute Engine 系统事件 [低风险]

Step 5: Resolution [High Risk]

步骤 5：解决 [高风险]

Troubleshoot unresponsive GKE TPU and GPU nodes (
`NodeStatusUnknown`
)

排查无响应的 GKE TPU 和 GPU 节点 (
`NodeStatusUnknown`
)

Step 0: Collect context and set the investigation window
`[Low Risk]`

步骤 0：收集上下文并设置调查窗口
`[低风险]`

Step 1: Verify the
`NodeStatusUnknown`
heartbeat timeout
`[Low Risk]`

步骤 1：验证
`NodeStatusUnknown`
心跳超时
`[低风险]`

Step 2: Inspect serial port output for a kernel panic
`[Low Risk]`

步骤 2：检查串口输出中的内核 panic
`[低风险]`

Step 3: Check node auto-repair status
`[Low Risk]`

步骤 3：检查节点自动修复状态
`[低风险]`

Step 4: Check Compute Engine system events
`[Low Risk]`

步骤 4：检查 Compute Engine 系统事件
`[低风险]`

Step 5: Resolution
`[High Risk]`

步骤 5：解决
`[高风险]`