gke-ai-troubleshooting-tpu-vm-repair-stuck

Troubleshoot GKE TPU VMs stuck in Compute Engine repair (

REPAIRING

)

This skill checks the VM status, the host events, and the GKE repair status, and routes recovery through GKE instead of direct Compute Engine changes.

Prerequisites

Tools:
- For installing and initializing
```
gcloud
```
  (which covers authentication and project selection), follow Before you begin.
- For installing
```
kubectl
```
  and required plugins, follow Install kubectl and Install required plugins.
Cloud Billing & Project Configuration: Ensure the project has an active Cloud Billing account. Follow Before you begin to initialize
```
gcloud
```
and enable the Google Kubernetes Engine API (
```
container.googleapis.com
```
), ensure the Compute Engine API (
```
compute.googleapis.com
```
) and Cloud Logging API (
```
logging.googleapis.com
```
) are enabled, and follow Store cluster information for kubectl to configure
```
kubectl
```
cluster credentials.
Required IAM Roles:
- Kubernetes Engine Viewer (
```
roles/container.viewer
```
  )
- Compute Viewer (
```
roles/compute.viewer
```
  )
- Logs Viewer (
```
roles/logging.viewer
```
  )
- For the user who enables node auto-repair in Step 5, item 1: Kubernetes Engine Cluster Admin (
```
roles/container.clusterAdmin
```
  )
Documentation:
- View information about MIGs and managed instances (Sections: "List MIGs", "Check the status of managed instances")
- Compute Engine instance lifecycle (Sections: "
```
REPAIRING
```
  state")
- Troubleshooting VM shutdowns and reboots (Sections: "Querying Cloud Audit Logs", "Reviewing Cloud Audit Logs")
- Configure cluster access for kubectl (Sections: "Install kubectl", "Install required plugins", "Store cluster information for kubectl")
- Add and manage node pools (Sections: "View node pools in a Standard cluster", "Deploy a Pod to a specific node pool")
- Troubleshoot NodeNotReady state (Sections: "Check the node's status and conditions", "Confirm node preemption")
- About TPUs in GKE (Sections: "Multi-host TPU slice node pools")
- Use dynamic slicing with a custom scheduler (Sections: "Requirements", "Verify the status of the nodes and the partitions", "Monitor the status of the Slice custom resource", "Hardware maintenance and failures")
- About GKE dynamic slicing (Sections: "Key characteristics of dynamic slicing configurations")
- Deploy TPUs in All Capacity mode in GKE (Sections: "Manage node failures", "Node failure and recovery")
- Report and repair faulty TPU hosts in All Capacity mode (Sections: "Understand the faulty host report process", "Review report faulty host operations")
- Auto-repair nodes (Sections: "Settings for Autopilot and Standard", "Repair criteria", "Node repair process", "Verify node auto-repair is enabled for a Standard node pool", "Enable auto-repair for an existing Standard node pool", "Get information about recent automated repair events", "Node auto repair in TPU slice nodes")
- Deploy TPU workloads in GKE Standard (Sections: "Before you begin", "Configure auto repair for TPU slice nodes")
- Get support (Sections: "Before you contact support", "Recommendations for efficient support")
- REST Resource: projects.locations.clusters.nodePools (Sections: "Resource: NodePool")

Read-only rule: Run read-only diagnostic commands only. Never drain, delete, or re-create nodes, or run any other command that changes the cluster. Give the user any fix to apply themselves.

When you recommend a fix, link the doc section that describes it.

Diagnostic workflow

Step 0: Collect context

[Low Risk]

Collect the target parameters:

```
{project_id}
```
: Google Cloud project ID
```
{cluster_name}
```
: GKE cluster name
```
{location}
```
: Cluster region or zone
```
{cluster_mode}
```
: Cluster mode (
```
Autopilot
```
or
```
Standard
```
)
```
{tpu_version}
```
: TPU version (such as Ironwood
```
tpu7x
```
, TPU v6e, v5p, v5e, or v4) and whether the node pool is a single-host or multi-host TPU slice node pool
```
{nodepool_name}
```
: TPU node pool name
```
{group_name}
```
and
```
{zone}
```
: Name and zone of the managed instance group (MIG) that backs the node pool:
- In a Standard cluster, follow View node pools in a Standard cluster to describe
```
{nodepool_name}
```
  and read its
```
instanceGroupUrls
```
  field (see Resource: NodePool).
- In an Autopilot cluster (or if
```
{nodepool_name}
```
  isn't known), follow List MIGs to list the managed instance groups in
```
{project_id}
```
  and find
```
{group_name}
```
  and
```
{zone}
```
  .
```
{instance_name}
```
: Affected TPU VM name, if known. If the user doesn't know
```
{instance_name}
```
, Step 1 finds the instances in
```
REPAIRING
```
.
```
{issue_time}
```
: Incident timestamp in RFC3339 UTC, if known

Step 1: Check the VM status in the managed instance group

[Low Risk]

To list the instances in

{group_name}

with their status (

STATUS

instanceStatus

) and current action (

ACTION

currentAction

), follow the instructions in Check the status of managed instances.

Analysis:
- STATUS
  (
  instanceStatus
  ) is
  REPAIRING
  : Compute Engine is repairing the VM because it encountered an internal error or its host server is unavailable due to maintenance. The VM can't be used during the repair. See
  REPAIRING
  state
  . Record
```
{instance_name}
```
  and proceed to Step 2.
- No instance is in
  REPAIRING
  (every instance reports
```
RUNNING
```
  with
```
ACTION
```
```
NONE
```
  , or an instance reports another status or action such as
```
ACTION
```
```
RECREATING
```
  , which means "The instance is being replaced" per Check the status of managed instances): Check each node's status and conditions following Check the node's status and conditions and hand off to the matching sibling skill:
  - If a node's
    Ready
    condition is
    Unknown
    (
    NodeStatusUnknown
    ), hand off to
    gke-ai-troubleshooting-node-unresponsive-timeout
    .
  - If a node isn't
    Ready
    for another reason, hand off to
    gke-node-notready
    .
  - If every node is
    Ready
    (per Repair criteria, "A node reporting a Ready status is considered healthy") but the workload's Pods are still
    Pending
    , hand off to
    gke-workload-troubleshooting
    .

Step 2: Check Compute Engine system events

[Low Risk]

To list the system events for

{instance_name}

around

{issue_time}

, follow the instructions in the section Querying Cloud Audit Logs. Compare the

method

field with the table in the Reviewing Cloud Audit Logs section of the same document:

compute.instances.hostError
,
compute.instances.terminateOnHostMaintenance
, or
compute.instances.automaticRestart
: Record the matching system event log entries and timestamps, and continue to Steps 3 and 4.

compute.instances.preempted
: Compute Engine preempted a Spot VM or preemptible VM. If the affected workload is a
```
JobSet
```
, hand off to
```
gke-ai-troubleshooting-jobset-interruption
```
. Otherwise, follow Confirm node preemption.
No method from the table, or another method: Per the note in Reviewing Cloud Audit Logs, if a VM rebooted and none of the listed methods appears in the query results, the reboot likely happened because the VM's operating system initiated it. If the VM is still in
```
REPAIRING
```
in Step 1 (or a method such as
```
compute.instances.reportHostAsFaulty
```
appears), continue to Steps 3 and 4.

Step 3: Check the node status and slice impact

[Low Risk]

Follow Check the node's status and conditions to check the status and conditions of the nodes in the cluster and match

{instance_name}

using the node's

ProviderID

field. To filter nodes to

{nodepool_name}

, use the

cloud.google.com/gke-nodepool

node label documented in Deploy a Pod to a specific node pool.

If the node for
```
{instance_name}
```
reports
```
Ready
```
(per Repair criteria, "A node reporting a Ready status is considered healthy") and the VM is no longer in
```
REPAIRING
```
, proceed to Step 5, item 4.
Otherwise, determine how the unhealthy node affects the node pool based on the node pool configuration:
- Single-host TPU slice node pool: Per Configure auto repair for TPU slice nodes, "In a single-host TPU slice node pool, only the unhealthy TPU node is auto-repaired."
- Dynamic slicing node pool: To check whether
```
{nodepool_name}
```
  uses dynamic slicing, see Requirements: dynamic slicing requires a Standard cluster, Ironwood (
```
tpu7x
```
  ), and All Capacity mode reservations. For a Standard cluster with Ironwood (
```
tpu7x
```
  ), check whether the node has
```
cloud.google.com/gke-accelerator-topology-mode: PROVISION_ONLY
```
  as shown in step 2 of Verify the status of the nodes and the partitions (per Key characteristics of dynamic slicing configurations, "dynamic slicing uses incremental provisioning"). Per Hardware maintenance and failures, "If you trigger customer-initiated maintenance or if a hardware failover occurs, only the targeted nodes are affected rather than the entire node pool." For an active dynamic sub-slice, the
```
Slice
```
  custom resource transitions to
```
FAILED
```
  and the scheduler attempts to re-create the slice on other available healthy nodes. Check the
```
Slice
```
  status following Monitor the status of the Slice custom resource; if the
```
Slice
```
  custom resource is in
```
FAILED
```
  or another non-active state, hand off
```
Slice
```
  troubleshooting to
```
gke-ai-troubleshooting-tpu-dynamic-slices-monitoring
```
  alongside checking the VM repair in Step 4.
- TPU All Capacity mode node pool: Per Manage node failures and Node failure and recovery, in TPU All Capacity mode, if a VM fails due to issues like hardware failure, Google Cloud "attempts to recover the VM on the same host (in-place repair)", failed nodes have a status of
```
NotReady
```
  , and the node pool's status doesn't change to
```
ERROR
```
  .
- Multi-host TPU slice node pool (without dynamic slicing or All Capacity mode): Per the section Multi-host TPU slice node pools, the VMs in a multi-host TPU slice node pool are treated as a single atomic unit. When a node in the slice requires repairing, GKE shuts down all VMs in the TPU slice, forcing eviction of all the Kubernetes Pods in the workload, and the Pods can be scheduled again only after all VMs in the slice are up and running.

Step 4: Check node auto-repair status and faulty host reports

[Low Risk]

Find out whether GKE or Compute Engine is repairing the node:

Check whether node auto-repair is enabled:
- Autopilot clusters: Autopilot always repairs nodes, and you can't turn this off (see Settings for Autopilot and Standard).
- Standard clusters: Follow Verify node auto-repair is enabled for a Standard node pool to check whether
```
autoRepair
```
  is enabled on
```
{nodepool_name}
```
  .
Check the GKE repair operation history: Per Configure auto repair for TPU slice nodes, check the repair status, including any failure reason, in the operation history described in Get information about recent automated repair events:
- If a repair operation failed because of insufficient quota, tell the user to contact their Google Cloud account representative to increase the quota.
- If
```
autoRepair
```
  is enabled and no repair operation has appeared yet, check how long the node has been unhealthy (from
```
{issue_time}
```
  or the
```
Ready
```
  condition's
```
LastTransitionTime
```
  in Step 3). Per Repair criteria, GKE initiates a repair when a node reports a
```
NotReady
```
  status, or doesn't report any status at all, over a given time threshold of approximately 10 minutes. If the node has reported
```
NotReady
```
  (or no status) for less than approximately 10 minutes, check the operation history again after that threshold has passed. If it has been longer than approximately 10 minutes and there is still no repair operation (and no All Capacity mode faulty host report in item 3), escalate in Step 5, item 3.
For TPU All Capacity mode node pools, check for a faulty host report: Per Understand the faulty host report process, after a user reports a faulty host, the TPU VM is stopped, its state changes to
```
REPAIRING
```
, and "The host repair process can take between 3 and 14 days, or longer." Before escalating a VM in
```
REPAIRING
```
in an All Capacity mode reservation, follow Review report faulty host operations to check for a
```
compute.instances.reportHostAsFaulty
```
operation on
```
{instance_name}
```
.

Step 5: Resolution

[High Risk]

Guardrails:
Never change GKE-managed instance groups or VMs directly through Compute Engine, for example with
gcloud compute instances delete
or
gcloud compute instance-groups managed
commands that change the group. Manage the node pool through GKE.
Never run
kubectl delete node
on nodes in the node pool. Per Settings for Autopilot and Standard, if you use
kubectl
to delete an unhealthy node when auto-repair is disabled, the underlying VM might become orphaned from the cluster, and you would continue to be billed for it.

Enable node auto-repair (Standard clusters only):
- Autopilot clusters always auto-repair nodes, so skip this step.
- Before the user enables auto-repair on a multi-host TPU slice node pool that doesn't use dynamic slicing or All Capacity mode, tell them that GKE re-creates the entire node pool when a node in it needs repair (see item 2 of this step).
- If
```
autoRepair
```
  is disabled on
```
{nodepool_name}
```
  , link the user (who needs the
```
Kubernetes Engine Cluster Admin
```
  role,
```
roles/container.clusterAdmin
```
  ) to Enable auto-repair for an existing Standard node pool and Configure auto repair for TPU slice nodes so they can apply the change themselves.
Let GKE or All Capacity mode repair the node or slice:
- Single-host TPU slice node pool: Per Configure auto repair for TPU slice nodes, "In a single-host TPU slice node pool, only the unhealthy TPU node is auto-repaired." Per Node repair process, for a single node repair, GKE drains the node and waits one hour for the drain to complete; if the drain doesn't complete, GKE shuts the node down and creates a new node.
- Multi-host TPU slice node pool (without dynamic slicing or All Capacity mode): Per Node auto repair in TPU slice nodes and Configure auto repair for TPU slice nodes, if a TPU slice node in a multi-host TPU slice node pool is unhealthy and requires auto repair, the entire node pool is re-created.
- Dynamic slicing node pool: Per Hardware maintenance and failures, only the targeted nodes are affected rather than the entire node pool, and for dynamic sub-slicing the scheduler attempts to re-create failed slices on other available healthy nodes. If
```
Slice
```
  custom resources fail to form or remain
```
FAILED
```
  , hand off to
```
gke-ai-troubleshooting-tpu-dynamic-slices-monitoring
```
  .
- TPU All Capacity mode with a faulty host report: If Step 4, item 3 found a
```
compute.instances.reportHostAsFaulty
```
  operation for
```
{instance_name}
```
  (tracked via Review report faulty host operations), do not escalate the VM as stuck. Tell the user that per Understand the faulty host report process, the VM stops during the repair and restarts on the same host after the repair is complete, and "The host repair process can take between 3 and 14 days, or longer."
Contact Cloud Customer Care if the repair doesn't start or complete:
- If the VM stays in
```
REPAIRING
```
  (and isn't waiting on a 3-to-14-day All Capacity mode faulty host repair), if the node has been
```
NotReady
```
  (or reported no status) for longer than the ~10-minute threshold in Repair criteria with
```
autoRepair
```
  enabled and no repair operation in Step 4, or if the operation history shows a failed repair that isn't caused by quota, direct the user to Before you contact support and Recommendations for efficient support to contact Cloud Customer Care. In addition to the details listed in Recommendations for efficient support, include the node pool name, instance name, the system event log entries from Step 2, and the repair operation name from Step 4.
Verify recovery:
- Repeat Step 1 and Step 3. When recovery is complete, every instance in
```
{group_name}
```
  reports
```
RUNNING
```
  with action
```
NONE
```
  , and every node in
```
{nodepool_name}
```
  is
```
Ready
```
  .
- If the instances report
```
RUNNING
```
  with action
```
NONE
```
  (or another status or action) and a node still isn't
```
Ready
```
  , follow the handoffs in Step 1 (
```
gke-ai-troubleshooting-node-unresponsive-timeout
```
  or
```
gke-node-notready
```
  ) or escalate to Cloud Customer Care in item 3 of this step.

gke-ai-troubleshooting-tpu-vm-repair-stuck

NPX Install

Tags

SKILL.md Content

Troubleshoot GKE TPU VMs stuck in Compute Engine repair (
`REPAIRING`
)

Prerequisites

Diagnostic workflow

Step 0: Collect context
`[Low Risk]`

Step 1: Check the VM status in the managed instance group
`[Low Risk]`

Step 2: Check Compute Engine system events
`[Low Risk]`

Step 3: Check the node status and slice impact
`[Low Risk]`

Step 4: Check node auto-repair status and faulty host reports
`[Low Risk]`

Step 5: Resolution
`[High Risk]`

gke-ai-troubleshooting-tpu-vm-repair-stuck

NPX Install

Tags

SKILL.md Content

Troubleshoot GKE TPU VMs stuck in Compute Engine repair (REPAIRING)

Prerequisites

Diagnostic workflow

Step 0: Collect context [Low Risk]

Step 1: Check the VM status in the managed instance group [Low Risk]

Step 2: Check Compute Engine system events [Low Risk]

Step 3: Check the node status and slice impact [Low Risk]

Step 4: Check node auto-repair status and faulty host reports [Low Risk]

Step 5: Resolution [High Risk]

Troubleshoot GKE TPU VMs stuck in Compute Engine repair (
`REPAIRING`
)

Step 0: Collect context
`[Low Risk]`

Step 1: Check the VM status in the managed instance group
`[Low Risk]`

Step 2: Check Compute Engine system events
`[Low Risk]`

Step 3: Check the node status and slice impact
`[Low Risk]`

Step 4: Check node auto-repair status and faulty host reports
`[Low Risk]`

Step 5: Resolution
`[High Risk]`