gke-tpu-dynamic-slices-monitoring

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

GKE TPU Dynamic Slices Monitoring & Management

GKE TPU动态切片监控与管理

Monitors the status of TPU Slice custom resources, troubleshoots provisioning failures, validates workload manifests on dynamic slices, and performs cleanups.

监控TPU切片自定义资源的状态，排查配置失败问题，验证动态切片上的工作负载清单，并执行清理操作。

Prerequisites

前提条件

Cloud Logging enabled for the project.
```
kubectl
```
and
```
gcloud
```
CLIs configured to access the GKE cluster.

项目已启用Cloud Logging。
```
kubectl
```
和
```
gcloud
```
CLI已配置为可访问GKE集群。

Diagnostic Workflow

诊断流程

Step 0: Context Acquisition & Time Window Definition

步骤0：上下文获取与时间窗口定义

Gather project, cluster, and slice context using cluster tools or the following parameters:

Project ID:
```
{project_id}
```
(e.g.,
```
my-gcp-project
```
)
Cluster Name:
```
{cluster_name}
```
(e.g.,
```
tpu-cluster
```
)
Region/Zone:
```
{location}
```
(e.g.,
```
us-central1-a
```
)
Slice Name:
```
{slice_name}
```
(e.g.,
```
test-slice
```
)
Issue Time:
```
{timestamp}
```
(Optional; default to the last 30 minutes window
```
[T - 30m]
```
to
```
[T + 30m]
```
)

使用集群工具或以下参数收集项目、集群和切片的上下文信息：

项目ID：
```
{project_id}
```
（例如：
```
my-gcp-project
```
）
集群名称：
```
{cluster_name}
```
（例如：
```
tpu-cluster
```
）
区域/可用区：
```
{location}
```
（例如：
```
us-central1-a
```
）
切片名称：
```
{slice_name}
```
（例如：
```
test-slice
```
）
问题时间：
```
{timestamp}
```
（可选；默认使用过去30分钟的时间窗口
```
[T - 30m]
```
至
```
[T + 30m]
```
）

Step 1: Describe the Slice Custom Resource [Low Risk]

步骤1：描述切片自定义资源 [低风险]

When asked to inspect, troubleshoot, or check a slice status, immediately execute

kubectl describe slice {slice_name}

using available cluster tools to perform the inspection. Parse the resulting

Status.Conditions

output against the condition table below to diagnose the exact state and provide concrete recommendations.

Command:
bash
```
kubectl describe slice {slice_name}
```

当需要检查、排查或查看切片状态时，立即使用集群工具执行

kubectl describe slice {slice_name}

命令进行检查。将得到的

Status.Conditions

输出与下方的状态表进行比对，以诊断具体状态并提供明确建议。

命令：
bash
```
kubectl describe slice {slice_name}
```

State & Reason Analysis

状态与原因分析

Analyze the

Status.Conditions

(especially

Type: Ready

and its

Reason

and

Status

Lifecycle State / Reason	Meaning	Recommended Action
`SliceNotCreated`	GKE Slice Controller	Wait a few minutes and
: : is initializing the : re-check slice status. :
: : slice and performing : :
: : resource checks. : :
`SliceCreationFailed`	Prerequisites	Verify selected nodes
: : validation failed : exist, are unallocated, :
: : (e.g., selected nodes : and topology matches :
: : don't exist, nodes are : partition count. :
: : already used by : :
: : another slice, or the : :
: : topology doesn't match : :
: : the number of : :
: : partitions). : :
`ACTIVATING`	GKE is actively	Monitor node
: : forming and : provisioning. :
: : provisioning the TPU : :
: : slice. : :
`ACTIVE`	The TPU slice is	Proceed to deploy or
: : successfully formed : check workloads. :
: : and ready to host : :
: : workloads. : :
`ACTIVE_DEGRADED`	The slice is usable,	Monitor workload logs
: : but one or more : for interconnect or :
: : sub-blocks are : device errors. Check :
: : degraded. : faulty node VMs. :
`FAILED`	GKE failed to form the	Ensure all selected
: : TPU slice (e.g., : nodes belong to the :
: : selected nodes are not : same reservation block. :
: : part of the same : :
: : reservation block). : :
`DEACTIVATING`	The slice is	Wait for dismantling to
: : dismantling (triggered : finish, or patch :
: : by user deletion or a : finalizers if stuck. :
: : critical systemic : :
: : failure). : :
`INCOMPLETE`	The terminal phase	No action required; the
: : before the Slice CR is : resource will be :
: : deleted from the : removed shortly. :
: : cluster. : :

分析

Status.Conditions

（尤其是

Type: Ready

及其

Reason

和

Status

）：

生命周期状态/原因	含义	建议操作
`SliceNotCreated`	GKE切片控制器正在初始化切片并执行资源检查。	等待几分钟后重新检查切片状态。
`SliceCreationFailed`	前提条件验证失败（例如：所选节点不存在、节点已被其他切片占用，或拓扑结构与分区数量不匹配）。	验证所选节点存在且未被分配，并且拓扑结构与分区数量匹配。
`ACTIVATING`	GKE正在主动构建并配置TPU切片。	监控节点配置进度。
`ACTIVE`	TPU切片已成功构建，可用于承载工作负载。	继续部署或检查工作负载。
`ACTIVE_DEGRADED`	切片可正常使用，但一个或多个子块已降级。	监控工作负载日志以排查互连或设备错误，检查故障节点VM。
`FAILED`	GKE无法构建TPU切片（例如：所选节点不属于同一个预留块）。	确保所有所选节点属于同一个预留和预留块。
`DEACTIVATING`	切片正在拆除（由用户删除或系统性严重故障触发）。	等待拆除完成，若卡住则修补终结器。
`INCOMPLETE`	Slice CR被从集群中删除前的终端阶段。	无需操作；资源将很快被移除。

Provisioning Failure Troubleshooting Checklist

配置失败排查清单

When investigating slice creation or provisioning failures (

SliceCreationFailed

FAILED

), perform the following verification steps:

Node Existence & Allocation Check: Verify that the selected TPU nodes exist in the cluster and are not already allocated to another slice (
```
kubectl get nodes -l cloud.google.com/gke-tpu-slice
```
,
```
kubectl get slice -A
```
).
Topology Alignment: Confirm that the partition count matches the requested topology dimensions (e.g. topology
```
2x2
```
requires 4 nodes).
Reservation Block Alignment Check: Confirm that all selected TPU nodes belong to the same reservation and reservation block.

当排查切片创建或配置失败（

SliceCreationFailed

或

FAILED

）时，执行以下验证步骤：

节点存在性与分配检查：验证所选TPU节点存在于集群中且未被分配给其他切片（执行
```
kubectl get nodes -l cloud.google.com/gke-tpu-slice
```
、
```
kubectl get slice -A
```
）。
拓扑结构匹配：确认分区数量与请求的拓扑维度匹配（例如：拓扑
```
2x2
```
需要4个节点）。
预留块匹配检查：确认所有所选TPU节点属于同一个预留和预留块。

Step 2: Verify Workload Specification [Low Risk]

步骤2：验证工作负载规格 [低风险]

Ensure workload manifests are configured correctly to target the dynamic slice.

确保工作负载清单已正确配置以指向动态切片。

1. Single-Slice Workload Requirements

1. 单切片工作负载要求

Check that the Pod template contains the following annotations and selectors:

Annotations:

cloud.google.com/gke-tpu-slice-topology: "{topology}"

(e.g.,

"4x4x4"

)

NodeSelector:

cloud.google.com/gke-tpu-topology: "{topology}"

(e.g.,

"4x4x4"

)

cloud.google.com/gke-tpu-accelerator: "{accelerator_type}"

(e.g.,

"tpu7x"

)

cloud.google.com/gke-tpu-slice: "{slice_name}"

(e.g.,

"test-slice"

)

检查Pod模板是否包含以下注解和选择器：

注解：

cloud.google.com/gke-tpu-slice-topology: "{topology}"

（例如：

"4x4x4"

）

节点选择器：

cloud.google.com/gke-tpu-topology: "{topology}"

（例如：

"4x4x4"

）

cloud.google.com/gke-tpu-accelerator: "{accelerator_type}"

（例如：

"tpu7x"

）

cloud.google.com/gke-tpu-slice: "{slice_name}"

（例如：

"test-slice"

）

2. Multi-Slice (JobSet) Workload Requirements

2. 多切片（JobSet）工作负载要求

If deploying a multi-slice JobSet, verify:

JobSet Annotation:

alpha.jobset.sigs.k8s.io/exclusive-topology: cloud.google.com/gke-tpu-slice

Pod Template Annotations:

cloud.google.com/gke-tpu-slice-topology: "{topology}"

Pod Template NodeSelector:

cloud.google.com/gke-tpu-topology: "{topology}"

cloud.google.com/gke-tpu-accelerator: "{accelerator_type}"

Note: Do NOT manually specify
cloud.google.com/gke-tpu-slice
in the nodeSelector; JobSet handles slice assignment automatically.

若部署多切片JobSet，请验证：

JobSet注解：

alpha.jobset.sigs.k8s.io/exclusive-topology: cloud.google.com/gke-tpu-slice

Pod模板注解：

cloud.google.com/gke-tpu-slice-topology: "{topology}"

Pod模板节点选择器：

cloud.google.com/gke-tpu-topology: "{topology}"

cloud.google.com/gke-tpu-accelerator: "{accelerator_type}"

注意：请勿在节点选择器中手动指定
cloud.google.com/gke-tpu-slice
；JobSet会自动处理切片分配。

Resolution & Management Workflow

解决与管理流程

Resolution 1: Force Delete a Stuck Slice [High Risk]

解决方案1：强制删除卡住的切片 [高风险]

If a slice is stuck in

DEACTIVATING

or deletion hangs indefinitely due to stuck finalizers:

Identify Cause: Explain that finalizers on the slice resource (
```
metadata.finalizers
```
) are preventing Kubernetes from completing resource deletion.

Propose Resolution: Propose removing finalizers from the metadata path (

/metadata/finalizers

) using a JSON patch operation:

bash

kubectl patch slice {slice_name} --type json -p='[{"op": "remove", "path": "/metadata/finalizers"}]'

Provide Warning: Explicitly warn the user that removing finalizers bypasses standard controller dismantling and may leave underlying VM, network, or accelerator resources uncleaned or orphaned.
CRITICAL SAFETY MANDATE: The response MUST explicitly ask the user for confirmation (e.g. "Removing finalizers on
/metadata/finalizers
via JSON patch is a high-risk operation that may leave orphaned resources. Do you confirm you want to apply this patch to slice
{slice_name}
?") and pause for user confirmation before applying or executing the patch.

若切片卡在

DEACTIVATING

状态或因终结器卡住导致删除操作无限期挂起：

确定原因：说明切片资源上的终结器（
```
metadata.finalizers
```
）阻止Kubernetes完成资源删除。

提出解决方案：建议使用JSON补丁操作移除metadata路径（

/metadata/finalizers

）下的终结器：

bash

kubectl patch slice {slice_name} --type json -p='[{"op": "remove", "path": "/metadata/finalizers"}]'

提供警告：明确警告用户，移除终结器会绕过标准控制器拆除流程，可能导致底层VM、网络或加速器资源未被清理或成为孤儿资源。
关键安全要求：回复中必须明确请求用户确认（例如："通过JSON补丁移除
/metadata/finalizers
下的终结器是高风险操作，可能会留下孤儿资源。您确认要对切片
{slice_name}
应用此补丁吗？"），并在应用或执行补丁前等待用户确认。

Resolution 2: Disable and Clean Up Slice Controller [High Risk]

解决方案2：禁用并清理切片控制器 [高风险]

If dynamic slicing needs to be disabled:

Check for existing Slices:
bash
```
kubectl get slice -A
```
Ensure all slices are deleted before disabling the controller.

Disable Slice Controller via gcloud:

bash

gcloud container clusters update {cluster_name} \
    --location={location} \
    --no-enable-slice-controller

Delete the Slice CRD:

bash

kubectl delete crd slices.accelerator.gke.io

Clean up Node Labels: Remove GKE TPU Slice labels from all nodes in the cluster:

bash

kubectl label nodes --all cloud.google.com/gke-tpu-slice- cloud.google.com/gke-tpu-slice-topology-

Safety Rule: Propose the exact commands and confirm before executing disabling or destructive cleanup steps.

若需要禁用动态切片功能：

检查现有切片：
bash
```
kubectl get slice -A
```
确保在禁用控制器前已删除所有切片。

通过gcloud禁用切片控制器：

bash

gcloud container clusters update {cluster_name} \
    --location={location} \
    --no-enable-slice-controller

删除Slice CRD：

bash

kubectl delete crd slices.accelerator.gke.io

清理节点标签：移除集群中所有节点的GKE TPU切片标签：

bash

kubectl label nodes --all cloud.google.com/gke-tpu-slice- cloud.google.com/gke-tpu-slice-topology-

安全规则：提供准确的命令，并在执行禁用或破坏性清理步骤前请求用户确认。