gke-ai-troubleshooting-node-unresponsive-timeout

Troubleshoot unresponsive GKE TPU and GPU nodes (

NodeStatusUnknown

)

When the Compute Engine host of a TPU or GPU node has a fatal hardware error, kernel panic, or non-maskable interrupt (NMI) lockup, the guest OS stops responding. The kubelet can no longer send heartbeats, so the node

Ready

condition becomes

Unknown

with

Reason: NodeStatusUnknown

(

Kubelet stopped posting node status.

). If node auto-repair is disabled on the node pool, GKE doesn't repair the node. The node can stay

NotReady

, and pods on it can stay in

Terminating

, which blocks multi-host

JobSet

workloads from recovering.

Prerequisites

Tools: Install the Google Cloud SDK (
```
gcloud
```
) and
```
kubectl
```
.

Cloud Billing & Project Configuration: Verify an active billing account is linked (

gcloud billing projects describe {project_id}

), authenticate (

gcloud auth login

), set the target project (

gcloud config set project {project_id}

), and ensure

container.googleapis.com

compute.googleapis.com

logging.googleapis.com

, and

monitoring.googleapis.com

are enabled.

Required IAM Roles:
- Kubernetes Engine Viewer (
```
roles/container.viewer
```
  )
- Compute Viewer (
```
roles/compute.viewer
```
  )
- Logs Viewer (
```
roles/logging.viewer
```
  )
- Monitoring Viewer (
```
roles/monitoring.viewer
```
  )
- For remediation (
```
[High Risk]
```
  steps): Kubernetes Engine Cluster Admin (
```
roles/container.clusterAdmin
```
  )
Documentation:
- Troubleshoot nodes with the NotReady status in GKE (Sections: "Check the node's status and conditions", "Confirm node preemption", "Verify that the node has recovered")
- Deploy TPU workloads in GKE Standard (Sections: "Monitor health metrics for TPU nodes and node pools", "Configure auto repair for TPU slice nodes")
- Troubleshoot OOM events
- View GKE logs (Sections: "System logs")
- Viewing serial port output
- Troubleshoot Linux VM boot issues due to kernel panic
- Auto-repair nodes (Sections: "Settings for Autopilot and Standard", "Verify node auto-repair is enabled for a Standard node pool", "Get information about recent automated repair events", "Enable auto-repair for an existing Standard node pool", "Repair criteria", "Node repair process", "Node auto repair in TPU slice nodes")
- Troubleshooting VM shutdowns and reboots (Sections: "Querying Cloud Audit Logs", "Reviewing Cloud Audit Logs")

Read-only rule: Run read-only diagnostic commands only. Never drain, delete, or re-create nodes, or run any other command that changes the cluster. Give the user any fix to apply themselves.

When you recommend a fix, link the doc section that describes it.

Diagnostic workflow

Step 0: Collect context and set the investigation window

[Low Risk]

Collect the target parameters. By default, query a 60-minute window `[T - 30m, T

30m]
```
around
```
{issue_time}`:

```
{project_id}
```
: Google Cloud project ID
```
{cluster_name}
```
: GKE cluster name
```
{location}
```
: Cluster region or zone
```
{nodepool_name}
```
: Target TPU or GPU node pool name
```
{node_name}
```
: Unresponsive GKE node name (and its Compute Engine
```
{zone}
```
)
```
{issue_time}
```
: Incident timestamp in RFC3339 UTC
```
{start_time}
```
:
```
{issue_time} - 30m
```
```
{end_time}
```
:
```
{issue_time} + 30m
```

Step 1: Verify the

NodeStatusUnknown

heartbeat timeout

[Low Risk]

Check Kubernetes node conditions: To inspect the node status and verify whether the
```
Ready
```
condition is
```
Unknown
```
with
```
Reason: NodeStatusUnknown
```
(
```
Kubelet stopped posting node status.
```
), follow the instructions in the section Check the node's status and conditions.
Query Cloud Logging (read-only LQL): Query
```
k8s_node
```
and
```
k8s_cluster
```
logs across
```
[{start_time}, {end_time}]
```
to confirm when the control plane lost heartbeat contact with
```
{node_name}
```
:

(resource.type="k8s_node" OR resource.type="k8s_cluster")
resource.labels.cluster_name="{cluster_name}"
("{node_name}" AND ("NodeNotReady" OR "NodeStatusUnknown" OR "Kubelet stopped posting node status"))
timestamp >= "{start_time}" AND timestamp <= "{end_time}"

Query Cloud Monitoring (read-only PromQL): Correlate the duration of the
```
Unknown
```
state using the GKE system metric
```
kubernetes.io/node/status_condition
```
(
```
kubernetes_io:node_status_condition
```
, GKE
```
1.32.1-gke.1357001+
```
) documented in Monitor health metrics for TPU nodes and node pools, filtered by
```
condition="Ready"
```
and
```
status="Unknown"
```
:

promql

kubernetes_io:node_status_condition{
  monitored_resource="k8s_node",
  cluster_name="{cluster_name}",
  node_name="{node_name}",
  condition="Ready",
  status="Unknown"
}

Decision logic:
- If the node
```
Ready
```
  condition is
```
True
```
  and
```
NodeStatusUnknown
```
  is absent, rule out an unresponsive node timeout and pivot to workload-level troubleshooting (for example, Troubleshoot OOM events) rather than repairing or draining the node.
- If
```
Ready
```
  is
```
Unknown
```
  (
```
NodeStatusUnknown
```
  ), proceed to Step 2.

Step 2: Inspect serial port output for a kernel panic

[Low Risk]

The guest OS can't send logs after a fatal kernel freeze, so check the serial port output of the node's VM:

Cloud Logging: If serial port logging is enabled (i.e. VM metadata
```
serial-port-logging-enable
```
is set to
```
true
```
), GKE system logs in Cloud Logging include the node's serial port output. See the section System logs. Use Cloud Logging when the VM is stopped or has already been replaced by auto-repair, or when you need more than the most recent output.
Running VM: Follow Viewing serial port output to retrieve the serial port 1 output (
```
gcloud compute instances get-serial-port-output
```
with
```
--port=1
```
) for
```
{node_name}
```
in
```
{zone}
```
. This method returns only the most recent 1 MB of output per port.

Consult Troubleshoot Linux VM boot issues due to kernel panic to identify documented kernel panic and hardware crash patterns (such as

Fatal Machine check

hung_task: blocked tasks

, or

NMI: Not continuing

) in the serial port output.

Step 3: Check node auto-repair status

[Low Risk]

Find out why GKE hasn't repaired the unresponsive node:

Autopilot clusters: Autopilot always repairs nodes, and you can't turn this off (see Settings for Autopilot and Standard). Skip the configuration check and check the repair history.
Standard clusters: Follow Verify node auto-repair is enabled for a Standard node pool to check whether
```
autoRepair
```
is enabled on
```
{nodepool_name}
```
.
Repair history (both modes): Auto-repair can be enabled and the repair can still fail. Per Configure auto repair for TPU slice nodes, check the repair status, including the failure reason, in the operation history described in Get information about recent automated repair events. If the failure is caused by insufficient quota, tell the user to contact their Google Cloud account representative to increase the quota.

Step 4: Check Compute Engine system events

[Low Risk]

To list the system events for

{node_name}

around

{issue_time}

, follow the instructions in the section Querying Cloud Audit Logs. Compare the

method

field with the table in the "Reviewing Cloud Audit Logs" section of the same document, and look for:

```
compute.instances.hostError
```
: a hardware or software issue on the physical host caused the VM to crash.
```
compute.instances.preempted
```
: Compute Engine preempted a Spot VM or preemptible VM. For preempted nodes, also see Confirm node preemption.

compute.instances.automaticRestart

: Compute Engine restarted the VM after a

hostError

terminateOnHostMaintenance

event.

```
compute.instances.guestTerminate
```
: the VM's operating system initiated the shutdown.

Step 5: Resolution

[High Risk]

Guardrails:
Never force-delete stuck
Terminating
pods on an unresponsive node. Force deletion doesn't wait for the kubelet to confirm that the pod has stopped, so a replacement pod can start while the old one is still running. See Force Delete StatefulSet Pods.
Never delete GKE-managed Compute Engine VM instances directly (
gcloud compute instances delete
). Instead, check how long the node has reported
NodeStatusUnknown
(Step 1), check the serial console output (Step 2), and rely on node auto-repair, as described in the following steps.

Enable node auto-repair (Standard clusters only):
- Autopilot clusters always auto-repair nodes, so skip this step.
- Before the user enables auto-repair on a multi-host TPU slice node pool, tell them that GKE re-creates the entire node pool when a node in it needs repair (see step 2).
- If
```
autoRepair
```
  is disabled on
```
{nodepool_name}
```
  , link the user to Enable auto-repair for an existing Standard node pool and Configure auto repair for TPU slice nodes so they can apply the change themselves.
Let GKE repair the node:
- Explain that GKE repairs a node that reports
```
NotReady
```
  or no status for the documented time threshold by draining and re-creating it, and link the "Repair criteria" and "Node repair process" sections of Auto-repair nodes. GKE waits one hour for the drain to complete. If the drain doesn't complete, GKE shuts the node down and creates a new node. Tell the user to expect this one-hour drain wait before GKE re-creates the node.
- For multi-host TPU slice node pools, note per Node auto repair in TPU slice nodes that the entire node pool is re-created.
Verify node recovery:
- After the repair completes, follow the instructions in the section Verify that the node has recovered to verify that the repaired node returns to
```
Ready
```
  status, then confirm that the
```
JobSet
```
  pods are running again.

gke-ai-troubleshooting-node-unresponsive-timeout

NPX Install

Tags

SKILL.md Content

Troubleshoot unresponsive GKE TPU and GPU nodes (
`NodeStatusUnknown`
)

Prerequisites

Diagnostic workflow

Step 0: Collect context and set the investigation window
`[Low Risk]`

Step 1: Verify the
`NodeStatusUnknown`
heartbeat timeout
`[Low Risk]`

Step 2: Inspect serial port output for a kernel panic
`[Low Risk]`

Step 3: Check node auto-repair status
`[Low Risk]`

Step 4: Check Compute Engine system events
`[Low Risk]`

Step 5: Resolution
`[High Risk]`

gke-ai-troubleshooting-node-unresponsive-timeout

NPX Install

Tags

SKILL.md Content

Troubleshoot unresponsive GKE TPU and GPU nodes (NodeStatusUnknown)

Prerequisites

Diagnostic workflow

Step 0: Collect context and set the investigation window [Low Risk]

Step 1: Verify the NodeStatusUnknown heartbeat timeout [Low Risk]

Step 2: Inspect serial port output for a kernel panic [Low Risk]

Step 3: Check node auto-repair status [Low Risk]

Step 4: Check Compute Engine system events [Low Risk]

Step 5: Resolution [High Risk]

Troubleshoot unresponsive GKE TPU and GPU nodes (
`NodeStatusUnknown`
)

Step 0: Collect context and set the investigation window
`[Low Risk]`

Step 1: Verify the
`NodeStatusUnknown`
heartbeat timeout
`[Low Risk]`

Step 2: Inspect serial port output for a kernel panic
`[Low Risk]`

Step 3: Check node auto-repair status
`[Low Risk]`

Step 4: Check Compute Engine system events
`[Low Risk]`

Step 5: Resolution
`[High Risk]`