gke-ai-troubleshooting-tpu-partition-overlap

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Troubleshoot TPU slice creation failures caused by partition overlap

排查由分区重叠导致的 TPU slice 创建失败

A dynamic slice fails to form with the status reason

SliceCreationFailed

when, among other documented causes, any of its nodes are already used by another slice. This skill confirms the failure, rules out the other documented causes, finds the slice that uses the same nodes, and routes the fix through GKE.

当 dynamic slice 的任一节点已被另一个 slice 使用时，除其他已有记录的原因外，它也可能因状态原因为

SliceCreationFailed

而无法形成。此 skill 会确认故障、排除其他已有记录的原因、找出使用相同节点的 slice，并通过 GKE 处理修复方案。

Prerequisites

前置条件

Tools:
- For installing and initializing
```
gcloud
```
  (which covers authentication and project selection), follow Before you begin.
- For installing
```
kubectl
```
  and required plugins, follow Install kubectl and Install required plugins.
Cloud Billing & Project Configuration: Ensure the project has an active Cloud Billing account. Follow Before you begin to initialize
```
gcloud
```
and enable the Google Kubernetes Engine API (
```
container.googleapis.com
```
), ensure the Cloud Monitoring API (
```
monitoring.googleapis.com
```
) is enabled, and follow Store cluster information for kubectl to configure
```
kubectl
```
cluster credentials.
Required IAM Roles:
- Kubernetes Engine Viewer (
```
roles/container.viewer
```
  )
- Monitoring Viewer (
```
roles/monitoring.viewer
```
  )
Documentation:
- Use dynamic slicing with a custom scheduler (Sections: "Before you begin", "Target your node pool to belong to a block or a sub-block", "Verify the status of the nodes and the partitions", "Create a Slice custom resource", "Monitor the status of the Slice custom resource", "Delete the slice")
- Configure cluster access for kubectl (Sections: "Install kubectl", "Install required plugins", "Store cluster information for kubectl")
- Slice (Sections: "Slice status", "Slice specification")
- Schedule dynamic slices with Kueue and TAS (Sections: "Target your node pool to belong to a block or a sub-block", "Create a dynamic slice with Kueue and TAS", "Define the partition health selection", "Optimize workload scheduling with slice reuse", "Enable Slice reuse", "Enforce slice deletion per workload", "Monitor the status of the slice", "Monitor the metrics of the slice", "Clean up")
- GKE system metrics (Sections: "Kubernetes metrics")
- Monitored resource types (Sections: "k8s_node_pool")
- About GKE dynamic slicing (Sections: "Dynamic slicing configurations")
- Get support (Sections: "Before you contact support", "Recommendations for efficient support")

Read-only rule: Run read-only diagnostic commands only. Never drain, delete, or re-create nodes, or run any other command that changes the cluster. Give the user any fix to apply themselves.

When you recommend a fix, link the doc section that describes it.

工具：
- 如需安装和初始化
```
gcloud
```
  （涵盖身份验证和项目选择），请参阅开始之前。
- 如需安装
```
kubectl
```
  和所需插件，请参阅安装 kubectl 和安装所需插件。
Cloud Billing 与项目配置：确保项目拥有有效的 Cloud Billing 账户。请参阅开始之前初始化
```
gcloud
```
并启用 Google Kubernetes Engine API（
```
container.googleapis.com
```
），确保已启用 Cloud Monitoring API（
```
monitoring.googleapis.com
```
），并参阅为 kubectl 存储集群信息配置
```
kubectl
```
集群凭据。
必需的 IAM 角色：
- Kubernetes Engine Viewer（
```
roles/container.viewer
```
  ）
- Monitoring Viewer（
```
roles/monitoring.viewer
```
  ）
文档：
- 使用 custom scheduler 进行 dynamic slicing （章节：“开始之前”、“将节点池定位到 block 或 sub-block”、“验证节点和分区的状态”、“创建 Slice custom resource”、“监控 Slice custom resource 的状态”、“删除 slice”）
- 配置 kubectl 的集群访问权限（章节：“安装 kubectl”、“安装所需插件”、“存储集群信息”）
- Slice （章节：“Slice 状态”、“Slice 规格”）
- 使用 Kueue 和 TAS 调度 dynamic slice （章节：“将节点池定位到 block 或 sub-block”、“使用 Kueue 和 TAS 创建 dynamic slice”、“定义分区健康状况选择”、“使用 slice 复用优化工作负载调度”、“启用 slice 复用”、“按工作负载强制删除 slice”、“监控 slice 状态”、“监控 slice 指标”、“清理”）
- GKE 系统指标（章节：“Kubernetes 指标”）
- 受监控的资源类型（章节：“k8s_node_pool”）
- 关于 GKE dynamic slicing （章节：“Dynamic slicing 配置”）
- 获取支持（章节：“联系支持之前”、“高效获取支持的建议”）

只读规则：仅运行只读诊断命令。绝不 drain、删除或重新创建节点，也不要运行任何其他会更改集群的命令。将所有修复方案提供给用户自行应用。

建议修复方案时，请链接说明该方案的文档章节。

Diagnostic workflow

诊断工作流

Step 0: Collect context

[Low Risk]

第 0 步：收集上下文

[低风险]

Collect the target parameters:

```
{project_id}
```
: Google Cloud project ID
```
{cluster_name}
```
: GKE cluster name
```
{location}
```
: Cluster region or zone
```
{slice_name}
```
: Name of the Slice custom resource that failed to form
```
{node_pool_name}
```
: Name of the TPU node pool (or node pools) backing the partitions in
```
{slice_name}
```
, if known. If the user doesn't know
```
{node_pool_name}
```
, after Step 1 run the Cloud Monitoring query in Step 3 for the partition IDs in
```
{partition_ids}
```
and read the
```
node_pool_name
```
label on the k8s_node_pool monitored resource.
```
{issue_time}
```
: Incident timestamp in RFC3339 UTC, if known. If the user doesn't know
```
{issue_time}
```
, use the
```
Last Transition Time
```
(
```
lastTransitionTime
```
, "A timestamp of the most recent status change" in Slice status) of the
```
Ready
```
condition from Step 1. By default, query Cloud Monitoring over a 60-minute window around
```
{issue_time}
```
.

收集以下目标参数：

```
{project_id}
```
：Google Cloud 项目 ID
```
{cluster_name}
```
：GKE 集群名称
```
{location}
```
：集群区域或可用区
```
{slice_name}
```
：无法形成的 Slice custom resource 名称
```
{node_pool_name}
```
：为
```
{slice_name}
```
中分区提供支持的 TPU 节点池名称（如果已知有多个节点池，也可填写多个名称）。如果用户不知道
```
{node_pool_name}
```
，请在第 1 步之后执行第 3 步中的 Cloud Monitoring 查询，获取
```
{partition_ids}
```
中的分区 ID，并读取 k8s_node_pool 受监控资源上的
```
node_pool_name
```
标签。
```
{issue_time}
```
：已知时填写 RFC3339 UTC 格式的故障时间戳。如果用户不知道
```
{issue_time}
```
，请使用第 1 步中
```
Ready
```
condition 的
```
Last Transition Time
```
（
```
lastTransitionTime
```
，Slice 状态中说明为“最近一次状态变更的时间戳”）。默认以
```
{issue_time}
```
为中心查询前后 60 分钟的 Cloud Monitoring 时间窗口。

Step 1: Confirm the

SliceCreationFailed

status

[Low Risk]

第 1 步：确认

SliceCreationFailed

状态

[低风险]

To describe

{slice_name}

, follow the instructions in the section Monitor the status of the Slice custom resource.

Status

Conditions

, check that the

Reason

of the

Ready

condition is

SliceCreationFailed

, and record its

Last Transition Time

(to use as

{issue_time}

if needed).

Record the IDs in

Spec

Partition Ids

{partition_ids}

, the

Topology

, and any

Labels

and

Annotations

{slice_name}

If the
```
Ready
```
reason isn't
```
SliceCreationFailed
```
, this skill doesn't apply. See Slice status for what each reason means, and hand off to
```
gke-ai-troubleshooting-tpu-dynamic-slices-monitoring
```
.

要获取

{slice_name}

的详细信息，请按照监控 Slice custom resource 的状态章节中的说明操作。

在
```
Status
```
>
```
Conditions
```
中，检查
```
Ready
```
condition 的
```
Reason
```
是否为
```
SliceCreationFailed
```
，并记录其
```
Last Transition Time
```
（如需使用，可将其作为
```
{issue_time}
```
）。

记录

Spec

Partition Ids

中的 ID（记为

{partition_ids}

）、

Topology

，以及

{slice_name}

上的所有

Labels

和

Annotations

。

如果
```
Ready
```
reason 不是
```
SliceCreationFailed
```
，则此 skill 不适用。请参阅 Slice 状态了解每种 reason 的含义，并移交给
```
gke-ai-troubleshooting-tpu-dynamic-slices-monitoring
```
。

Step 2: Rule out the other documented causes

[Low Risk]

第 2 步：排除其他已有记录的原因

[低风险]

Per Create a Slice custom resource, slice creation also fails with

SliceCreationFailed

when:

The selected nodes don't exist.
The topology doesn't match the number of partitions. A dynamic sub-slice lists exactly one partition ID. A dynamic super-slice lists
```
(A × B × C) / 64
```
partition IDs.

If either of those causes applies, hand off to

gke-ai-troubleshooting-tpu-dynamic-slices-monitoring

Otherwise, follow Verify the status of the nodes and the partitions on

{node_pool_name}

Verify that the nodes exist and carry each ID in
```
{partition_ids}
```
.
On the nodes that carry
```
{partition_ids}
```
, record every partition ID of every shape (
```
4x4x4
```
,
```
2x4x4
```
,
```
2x2x4
```
,
```
2x2x2
```
, and
```
2x2x1
```
) from the
```
cloud.google.com/gke-tpu-partition-.*-id
```
labels. Per Dynamic slicing configurations, dynamic sub-slicing partitions a
```
4x4x4
```
sub-block into smaller topologies on the same physical node pool, so partitions of different shapes share the same nodes.
Read each partition's
```
cloud.google.com/gke-tpu-partition-.*-state
```
label. Per Verify the status of the nodes and the partitions and Define the partition health selection, count
```
HEALTHY
```
partitions and
```
DEGRADED
```
```
4x4x4
```
partitions ("still usable for dynamic slice formation"; "Smaller topologies don't have a degraded state") as available. Partitions in
```
UNHEALTHY
```
("can't be used to form a slice"),
```
UNSET
```
, or
```
INCOMPLETE
```
aren't available. If a requested partition is in
```
UNHEALTHY
```
,
```
UNSET
```
, or
```
INCOMPLETE
```
, suggest an available partition whose nodes no other slice uses (Step 4, item 2), hand off to
```
gke-ai-troubleshooting-tpu-dynamic-slices-monitoring
```
, or escalate to Cloud Customer Care (Step 4, item 4).

根据创建 Slice custom resource，在以下情况下，slice 创建也会因

SliceCreationFailed

而失败：

所选节点不存在。
拓扑与分区数量不匹配。dynamic sub-slice 恰好列出一个分区 ID；dynamic super-slice 列出
```
(A × B × C) / 64
```
个分区 ID。

如果上述任一原因适用，请移交给

gke-ai-troubleshooting-tpu-dynamic-slices-monitoring

。

否则，请按照验证节点和分区的状态对

{node_pool_name}

执行以下检查：

验证节点存在，并且每个节点都包含
```
{partition_ids}
```
中的一个 ID。
在包含
```
{partition_ids}
```
的节点上，记录每种 shape（
```
4x4x4
```
、
```
2x4x4
```
、
```
2x2x4
```
、
```
2x2x2
```
和
```
2x2x1
```
）的所有分区 ID。这些 ID 来自
```
cloud.google.com/gke-tpu-partition-.*-id
```
标签。根据 Dynamic slicing 配置，dynamic sub-slicing 会在同一物理节点池中把一个
```
4x4x4
```
sub-block 划分为更小的拓扑，因此不同 shape 的分区会共享相同节点。
读取每个分区的
```
cloud.google.com/gke-tpu-partition-.*-state
```
标签。根据验证节点和分区的状态和定义分区健康状况选择，将
```
HEALTHY
```
分区和
```
DEGRADED
```
的
```
4x4x4
```
分区（“仍可用于形成 dynamic slice”；“较小的拓扑没有 degraded 状态”）计为可用分区。处于
```
UNHEALTHY
```
（“不能用于形成 slice”）、
```
UNSET
```
或
```
INCOMPLETE
```
状态的分区不可用。如果请求的分区处于
```
UNHEALTHY
```
、
```
UNSET
```
或
```
INCOMPLETE
```
状态，请建议使用节点未被任何其他 slice 占用的可用分区（第 4 步第 2 项）、移交给
```
gke-ai-troubleshooting-tpu-dynamic-slices-monitoring
```
，或升级至 Cloud Customer Care（第 4 步第 4 项）。

Step 3: Find the slice that uses the same nodes

[Low Risk]

第 3 步：找出使用相同节点的 slice

[低风险]

Per Create a Slice custom resource, slice creation fails with

SliceCreationFailed

when any nodes on the custom resource are used by another slice. Because partitions of different shapes on the same sub-block share nodes (see Dynamic slicing configurations), another slice conflicts with

{slice_name}

if it lists any of the partition IDs (of any shape) recorded on

{slice_name}

's nodes in Step 2.

List the
```
Slice
```
custom resources in the cluster (as shown in Delete the slice) and describe each one following Monitor the status of the Slice custom resource to read
```
Spec
```
>
```
Partition Ids
```
,
```
Metadata
```
>
```
Labels
```
, and
```
Status
```
>
```
Conditions
```
(
```
Ready
```
reason):
- Treat any slice other than
```
{slice_name}
```
  whose
```
Spec
```
  >
```
Partition Ids
```
  includes any partition ID recorded on
```
{slice_name}
```
  's nodes in Step 2 (whether the exact ID in
```
{partition_ids}
```
  or another shape's partition ID on the same nodes) as conflicting. Record its name, its
```
Ready
```
  reason (such as
```
ACTIVE
```
  ), and whether Kueue manages it (Step 4, item 1).
To check partition-to-slice associations in Cloud Monitoring (including around
```
{issue_time}
```
), query the
```
kubernetes.io/accelerator/partition/state
```
metric listed in Monitor the metrics of the slice. Per Kubernetes metrics, this metric "is 1 if the partition state is detected, 0 otherwise", and its labels include
```
partition_id
```
,
```
slice_id
```
("ID of the associated slice"), and
```
state
```
("Health of the partition (e.g., HEALTHY, UNHEALTHY)"). Run the query for each
```
{partition_id}
```
recorded on
```
{slice_name}
```
's nodes in Step 2:

promql

kubernetes_io:accelerator_partition_state{
  monitored_resource="k8s_node_pool",
  cluster_name="{cluster_name}",
  partition_id="{partition_id}"
}

Read only time series whose value is
```
1
```
(where
```
state="HEALTHY"
```
means the partition is healthy and functional, per Verify the status of the nodes and the partitions).
Because Kubernetes metrics defines
```
slice_id
```
only as "ID of the associated slice", always confirm the conflicting slice from the
```
Slice
```
custom resources in the cluster rather than from
```
slice_id
```
alone: list the
```
Slice
```
custom resources as shown in Delete the slice, describe each one following Monitor the status of the Slice custom resource, and check which slice's
```
Spec
```
>
```
Partition Ids
```
includes the partition ID (or any other partition ID of any shape on the same nodes from Step 2).

Next steps based on Step 3 results:
- Another slice uses the nodes now: Proceed to Step 4.
- No other slice uses the nodes now, but the metric (value
  1
  ) showed an associated
  slice_id
  around
  {issue_time}
  : The conflict may already be over. Proceed to Step 4, item 1 (to re-create or retry
```
{slice_name}
```
  ) and then Step 4, item 4.
- Neither the current
  Slice
  custom resources nor the metric around
  {issue_time}
  shows another slice: Node overlap isn't confirmed. Hand off to
```
gke-ai-troubleshooting-tpu-dynamic-slices-monitoring
```
  or escalate to Cloud Customer Care (Step 4, item 4).

根据创建 Slice custom resource，当 custom resource 上的任一节点已被另一个 slice 使用时，slice 创建会因

SliceCreationFailed

而失败。由于同一 sub-block 上不同 shape 的分区会共享节点（参阅 Dynamic slicing 配置），如果另一个 slice 列出了第 2 步在

{slice_name}

节点上记录的任何分区 ID（无论其 shape），则该 slice 与

{slice_name}

冲突。

按照删除 slice 中所示的方式列出集群中的
```
Slice
```
custom resource，并按照监控 Slice custom resource 的状态查看每个 resource，以读取
```
Spec
```
>
```
Partition Ids
```
、
```
Metadata
```
>
```
Labels
```
和
```
Status
```
>
```
Conditions
```
（
```
Ready
```
reason）：
- 如果除
```
{slice_name}
```
  以外的某个 slice，其
```
Spec
```
  >
```
Partition Ids
```
  包含第 2 步在
```
{slice_name}
```
  节点上记录的任何分区 ID（无论是
```
{partition_ids}
```
  中的精确 ID，还是同一节点上另一种 shape 的分区 ID），则将其视为冲突 slice。记录其名称、
```
Ready
```
  reason（例如
```
ACTIVE
```
  ），以及它是否由 Kueue 管理（第 4 步第 1 项）。
要在 Cloud Monitoring 中检查分区与 slice 的关联（包括
```
{issue_time}
```
前后的情况），请查询监控 slice 指标中列出的
```
kubernetes.io/accelerator/partition/state
```
指标。根据 Kubernetes 指标，该指标“在检测到分区状态时为 1，否则为 0”，其标签包括
```
partition_id
```
、
```
slice_id
```
（“关联 slice 的 ID”）和
```
state
```
（“分区的健康状况，例如 HEALTHY、UNHEALTHY”）。对第 2 步在
```
{slice_name}
```
节点上记录的每个
```
{partition_id}
```
运行以下查询：

promql

kubernetes_io:accelerator_partition_state{
  monitored_resource="k8s_node_pool",
  cluster_name="{cluster_name}",
  partition_id="{partition_id}"
}

仅读取值为
```
1
```
的时间序列，其中
```
state="HEALTHY"
```
表示分区健康且可正常工作，具体参见验证节点和分区的状态。
由于 Kubernetes 指标仅将
```
slice_id
```
定义为“关联 slice 的 ID”，因此务必通过集群中的
```
Slice
```
custom resource 确认冲突 slice，而不能只依赖
```
slice_id
```
：按照删除 slice 中所示的方式列出
```
Slice
```
custom resource，按照监控 Slice custom resource 的状态查看每个 resource，并检查哪个 slice 的
```
Spec
```
>
```
Partition Ids
```
包含该分区 ID，或包含第 2 步中同一节点上另一种 shape 的任何分区 ID。

根据第 3 步结果采取下一步操作：
- 当前有另一个 slice 使用这些节点：继续执行第 4 步。
- 当前没有其他 slice 使用这些节点，但指标在
  {issue_time}
  前后显示了一个关联的
  slice_id
  （值为
  1
  ）：冲突可能已经结束。执行第 4 步第 1 项（重新创建或重试
```
{slice_name}
```
  ），然后执行第 4 步第 4 项。
- 当前
  Slice
  custom resource 和
  {issue_time}
  前后的指标均未显示其他 slice：无法确认存在节点重叠。请移交给
```
gke-ai-troubleshooting-tpu-dynamic-slices-monitoring
```
  ，或升级至 Cloud Customer Care（第 4 步第 4 项）。

Step 4: Resolution

[High Risk]

第 4 步：解决

[高风险]

Guardrails:
Never delete or change the other slice, or stop its workload, to free the nodes. Deleting a
Slice
custom resource starts dismantling that slice (
DEACTIVATING
, see Monitor the status of the Slice custom resource). The owner of that slice decides when to release it.
Never manually modify or delete
Slice
custom resources created by Kueue. Per Target your node pool to belong to a block or a sub-block, "Avoid manually modifying Slice custom resources created by Kueue."
Never change GKE-managed instance groups or VMs directly through Compute Engine, and never delete node pools to free a partition. Manage node pools and slices through GKE.

First, check whether Kueue manages

{slice_name}

or the conflicting slice:

Per Monitor the status of the slice, when a slice is created or actively used by a workload, the slice controller attaches the
```
kueue.gke.io/tpu-slice-used-by-namespace
```
and
```
kueue.gke.io/tpu-slice-used-by-workload
```
labels.
Per Optimize workload scheduling with slice reuse and Enable Slice reuse, when slice reuse is enabled, "Kueue marks the slice as idle at the end of a job instead of deleting it", and "Reusable slices are named dynamically with an
```
s-<topology>
```
prefix, decoupling them from specific workload names."

Then apply the matching option:

Wait for the conflicting slice to release the nodes and retry:
- If Kueue manages the conflicting slice (it has the
```
kueue.gke.io/...
```
  labels or an
```
s-<topology>
```
  name prefix): Do not suggest deleting the
```
Slice
```
  custom resource directly. Point the workload owner to Clean up (deleting the
```
JobSet
```
  triggers Kueue to delete the associated
```
Slice
```
  custom resources) and, if slice reuse keeps an idle
```
s-<topology>
```
  slice allocated after a job ends, to Enforce slice deletion per workload (
```
kueue.gke.io/tpu-enforce-slice-deletion: "true"
```
  ).
- If a custom scheduler or user created the conflicting slice directly: Its owner can delete it when finished, as described in Delete the slice.
- How
  {slice_name}
  retries: Check whether
```
{slice_name}
```
  already has
```
slice.gke.io/retry-on-failure: "true"
```
  in
```
Metadata
```
  >
```
Annotations
```
  . Per Create a Slice custom resource, when a
```
Slice
```
  custom resource is created with this annotation, "If the slice isn't created because of the
```
SliceCreationFailed
```
  status reason, the controller will retry until the slice is successfully formed." Because the documentation only shows configuring
```
slice.gke.io/retry-on-failure: "true"
```
  in the
```
Slice
```
  manifest when creating a
```
Slice
```
  custom resource (and doesn't state that adding the annotation to an existing slice already in
```
SliceCreationFailed
```
  starts a retry), if
```
{slice_name}
```
  wasn't created by Kueue and lacks the annotation, tell the user to delete
```
{slice_name}
```
  following Delete the slice and re-create it with
```
slice.gke.io/retry-on-failure: "true"
```
  in its manifest following Create a Slice custom resource.
Use available partitions whose nodes no other slice uses: If the user manages
```
Slice
```
custom resources directly (not through Kueue) and can't wait, direct them to Verify the status of the nodes and the partitions and Create a Slice custom resource so they can pick available partitions (
```
HEALTHY
```
partitions, or
```
DEGRADED
```
```
4x4x4
```
partitions) whose nodes aren't used by any other slice (across all partition shapes on those nodes) and create the
```
Slice
```
custom resource with them.
Schedule dynamic slices with Kueue and TAS instead of creating
Slice
custom resources directly: Per Target your node pool to belong to a block or a sub-block and Create a dynamic slice with Kueue and TAS, instead of creating
```
Slice
```
custom resources directly, teams can schedule GKE workloads with Kueue and Topology Aware Scheduling (TAS), where "Kueue automatically handles the creation and deletion of Slice custom resources" and admits a workload after its slice becomes active.
Verify or escalate: Repeat Step 1 following Monitor the status of the Slice custom resource:
- ACTIVE
  : The slice is fully formed, healthy, and ready to execute workloads.
- ACTIVE_DEGRADED
  : The slice is formed, and workloads can run, but performance might be impacted.
- Still
  SliceCreationFailed
  after no other slice uses the nodes, or
  FAILED
  : Hand off to
```
gke-ai-troubleshooting-tpu-dynamic-slices-monitoring
```
  , or advise the user to contact Cloud Customer Care following Before you contact support and Recommendations for efficient support.

防护规则：
绝不删除或修改其他 slice，也绝不停止其工作负载以释放节点。删除
Slice
custom resource 会开始拆除该 slice（
DEACTIVATING
，参阅监控 Slice custom resource 的状态）。何时释放该 slice 应由其所有者决定。
绝不手动修改或删除由 Kueue 创建的
Slice
custom resource。根据将节点池定位到 block 或 sub-block，“请避免手动修改由 Kueue 创建的 Slice custom resource”。
绝不通过 Compute Engine 直接修改或删除 GKE 管理的 instance group 或 VM，也绝不通过删除节点池来释放分区。请通过 GKE 管理节点池和 slice。

首先，检查

{slice_name}

或冲突 slice 是否由 Kueue 管理：

根据监控 slice 状态，当 slice 被创建或正被工作负载主动使用时，slice controller 会附加
```
kueue.gke.io/tpu-slice-used-by-namespace
```
和
```
kueue.gke.io/tpu-slice-used-by-workload
```
标签。
根据使用 slice 复用优化工作负载调度和启用 slice 复用，启用 slice 复用了之后，“Kueue 会在 Job 结束时将 slice 标记为空闲，而不是将其删除”，并且“可复用 slice 会使用
```
s-<topology>
```
前缀动态命名，从而不与特定的工作负载名称绑定”。

然后应用匹配的解决方案：

等待冲突 slice 释放节点，然后重试：
- 如果 Kueue 管理冲突 slice（它具有
```
kueue.gke.io/...
```
  标签或使用
```
s-<topology>
```
  名称前缀）：不要建议直接删除
```
Slice
```
  custom resource。请将工作负载所有者引导至清理（删除
```
JobSet
```
  会触发 Kueue 删除关联的
```
Slice
```
  custom resource）；如果启用 slice 复用后，Job 结束时仍保留已分配的空闲
```
s-<topology>
```
  slice，还应引导其参阅按工作负载强制删除 slice（
```
kueue.gke.io/tpu-enforce-slice-deletion: "true"
```
  ）。
- 如果冲突 slice 由 custom scheduler 或用户直接创建：其所有者可以在使用完毕后将其删除，详见删除 slice。
- {slice_name}
  的重试方式：检查
```
{slice_name}
```
  的
```
Metadata
```
  >
```
Annotations
```
  中是否已经存在
```
slice.gke.io/retry-on-failure: "true"
```
  。根据创建 Slice custom resource，使用此 annotation 创建
```
Slice
```
  custom resource 时，“如果 slice 因
```
SliceCreationFailed
```
  状态原因而未能创建，controller 将持续重试，直到 slice 成功形成”。由于文档仅展示了在创建
```
Slice
```
  custom resource 时于 manifest 中配置
```
slice.gke.io/retry-on-failure: "true"
```
  ，并未说明向已经处于
```
SliceCreationFailed
```
  状态的现有 slice 添加该 annotation 会触发重试，因此，如果
```
{slice_name}
```
  不是由 Kueue 创建且缺少该 annotation，请告知用户按照删除 slice 删除
```
{slice_name}
```
  ，然后按照创建 Slice custom resource 重新创建，并在 manifest 中加入
```
slice.gke.io/retry-on-failure: "true"
```
  。
使用节点未被任何其他 slice 占用的可用分区：如果用户直接管理
```
Slice
```
custom resource（不通过 Kueue）且无法等待，请引导其参阅验证节点和分区的状态和创建 Slice custom resource，以选择可用分区（
```
HEALTHY
```
分区或
```
DEGRADED
```
的
```
4x4x4
```
分区），并确保这些分区的节点未被任何其他 slice 使用（需检查这些节点上所有 shape 的分区），然后使用它们创建
```
Slice
```
custom resource。
使用 Kueue 和 TAS 调度 dynamic slice，而不是直接创建
Slice
custom resource：根据将节点池定位到 block 或 sub-block 和使用 Kueue 和 TAS 创建 dynamic slice，团队可以使用 Kueue 和 Topology Aware Scheduling（TAS）来调度 GKE 工作负载，而不必直接创建
```
Slice
```
custom resource；这种方式下，“Kueue 自动处理 Slice custom resource 的创建和删除”，并在 slice 变为 active 后接纳工作负载。
验证或升级处理：按照监控 Slice custom resource 的状态重复执行第 1 步：
- ACTIVE
  ：slice 已完全形成且状态健康，可以执行工作负载。
- ACTIVE_DEGRADED
  ：slice 已形成且可以运行工作负载，但性能可能受到影响。
- 在节点不再被其他 slice 使用后仍为
  SliceCreationFailed
  ，或状态为
  FAILED
  ：移交给
```
gke-ai-troubleshooting-tpu-dynamic-slices-monitoring
```
  ，或建议用户按照联系支持之前和高效获取支持的建议联系 Cloud Customer Care。

gke-ai-troubleshooting-tpu-partition-overlap

Original

Translation

Troubleshoot TPU slice creation failures caused by partition overlap

排查由分区重叠导致的 TPU slice 创建失败

Prerequisites

前置条件

Diagnostic workflow

诊断工作流

Step 0: Collect context
`[Low Risk]`

第 0 步：收集上下文
`[低风险]`

Step 1: Confirm the
`SliceCreationFailed`
status
`[Low Risk]`

第 1 步：确认
`SliceCreationFailed`
状态
`[低风险]`

Step 2: Rule out the other documented causes
`[Low Risk]`

第 2 步：排除其他已有记录的原因
`[低风险]`

Step 3: Find the slice that uses the same nodes
`[Low Risk]`

第 3 步：找出使用相同节点的 slice
`[低风险]`

Step 4: Resolution
`[High Risk]`

第 4 步：解决
`[高风险]`