Troubleshoot TPU slice creation failures caused by partition overlap

A dynamic slice fails to form with the status reason

SliceCreationFailed

when, among other documented causes, any of its nodes are already used by another slice. This skill confirms the failure, rules out the other documented causes, finds the slice that uses the same nodes, and routes the fix through GKE.

Prerequisites

Tools:
- For installing and initializing
```
gcloud
```
  (which covers authentication and project selection), follow Before you begin.
- For installing
```
kubectl
```
  and required plugins, follow Install kubectl and Install required plugins.
Cloud Billing & Project Configuration: Ensure the project has an active Cloud Billing account. Follow Before you begin to initialize
```
gcloud
```
and enable the Google Kubernetes Engine API (
```
container.googleapis.com
```
), ensure the Cloud Monitoring API (
```
monitoring.googleapis.com
```
) is enabled, and follow Store cluster information for kubectl to configure
```
kubectl
```
cluster credentials.
Required IAM Roles:
- Kubernetes Engine Viewer (
```
roles/container.viewer
```
  )
- Monitoring Viewer (
```
roles/monitoring.viewer
```
  )
Documentation:
- Use dynamic slicing with a custom scheduler (Sections: "Before you begin", "Target your node pool to belong to a block or a sub-block", "Verify the status of the nodes and the partitions", "Create a Slice custom resource", "Monitor the status of the Slice custom resource", "Delete the slice")
- Configure cluster access for kubectl (Sections: "Install kubectl", "Install required plugins", "Store cluster information for kubectl")
- Slice (Sections: "Slice status", "Slice specification")
- Schedule dynamic slices with Kueue and TAS (Sections: "Target your node pool to belong to a block or a sub-block", "Create a dynamic slice with Kueue and TAS", "Define the partition health selection", "Optimize workload scheduling with slice reuse", "Enable Slice reuse", "Enforce slice deletion per workload", "Monitor the status of the slice", "Monitor the metrics of the slice", "Clean up")
- GKE system metrics (Sections: "Kubernetes metrics")
- Monitored resource types (Sections: "k8s_node_pool")
- About GKE dynamic slicing (Sections: "Dynamic slicing configurations")
- Get support (Sections: "Before you contact support", "Recommendations for efficient support")

Read-only rule: Run read-only diagnostic commands only. Never drain, delete, or re-create nodes, or run any other command that changes the cluster. Give the user any fix to apply themselves.

When you recommend a fix, link the doc section that describes it.

Diagnostic workflow

Step 0: Collect context

[Low Risk]

Collect the target parameters:

```
{project_id}
```
: Google Cloud project ID
```
{cluster_name}
```
: GKE cluster name
```
{location}
```
: Cluster region or zone
```
{slice_name}
```
: Name of the Slice custom resource that failed to form
```
{node_pool_name}
```
: Name of the TPU node pool (or node pools) backing the partitions in
```
{slice_name}
```
, if known. If the user doesn't know
```
{node_pool_name}
```
, after Step 1 run the Cloud Monitoring query in Step 3 for the partition IDs in
```
{partition_ids}
```
and read the
```
node_pool_name
```
label on the k8s_node_pool monitored resource.
```
{issue_time}
```
: Incident timestamp in RFC3339 UTC, if known. If the user doesn't know
```
{issue_time}
```
, use the
```
Last Transition Time
```
(
```
lastTransitionTime
```
, "A timestamp of the most recent status change" in Slice status) of the
```
Ready
```
condition from Step 1. By default, query Cloud Monitoring over a 60-minute window around
```
{issue_time}
```
.

Step 1: Confirm the

SliceCreationFailed

status

[Low Risk]

To describe

{slice_name}

, follow the instructions in the section Monitor the status of the Slice custom resource.

Status

Conditions

, check that the

Reason

of the

Ready

condition is

SliceCreationFailed

, and record its

Last Transition Time

(to use as

{issue_time}

if needed).

Record the IDs in

Spec

Partition Ids

{partition_ids}

, the

Topology

, and any

Labels

and

Annotations

{slice_name}

If the
```
Ready
```
reason isn't
```
SliceCreationFailed
```
, this skill doesn't apply. See Slice status for what each reason means, and hand off to
```
gke-ai-troubleshooting-tpu-dynamic-slices-monitoring
```
.

Step 2: Rule out the other documented causes

[Low Risk]

Per Create a Slice custom resource, slice creation also fails with

SliceCreationFailed

when:

The selected nodes don't exist.
The topology doesn't match the number of partitions. A dynamic sub-slice lists exactly one partition ID. A dynamic super-slice lists
```
(A × B × C) / 64
```
partition IDs.

If either of those causes applies, hand off to

gke-ai-troubleshooting-tpu-dynamic-slices-monitoring

Otherwise, follow Verify the status of the nodes and the partitions on

{node_pool_name}

Verify that the nodes exist and carry each ID in
```
{partition_ids}
```
.
On the nodes that carry
```
{partition_ids}
```
, record every partition ID of every shape (
```
4x4x4
```
,
```
2x4x4
```
,
```
2x2x4
```
,
```
2x2x2
```
, and
```
2x2x1
```
) from the
```
cloud.google.com/gke-tpu-partition-.*-id
```
labels. Per Dynamic slicing configurations, dynamic sub-slicing partitions a
```
4x4x4
```
sub-block into smaller topologies on the same physical node pool, so partitions of different shapes share the same nodes.
Read each partition's
```
cloud.google.com/gke-tpu-partition-.*-state
```
label. Per Verify the status of the nodes and the partitions and Define the partition health selection, count
```
HEALTHY
```
partitions and
```
DEGRADED
```
```
4x4x4
```
partitions ("still usable for dynamic slice formation"; "Smaller topologies don't have a degraded state") as available. Partitions in
```
UNHEALTHY
```
("can't be used to form a slice"),
```
UNSET
```
, or
```
INCOMPLETE
```
aren't available. If a requested partition is in
```
UNHEALTHY
```
,
```
UNSET
```
, or
```
INCOMPLETE
```
, suggest an available partition whose nodes no other slice uses (Step 4, item 2), hand off to
```
gke-ai-troubleshooting-tpu-dynamic-slices-monitoring
```
, or escalate to Cloud Customer Care (Step 4, item 4).

Step 3: Find the slice that uses the same nodes

[Low Risk]

Per Create a Slice custom resource, slice creation fails with

SliceCreationFailed

when any nodes on the custom resource are used by another slice. Because partitions of different shapes on the same sub-block share nodes (see Dynamic slicing configurations), another slice conflicts with

{slice_name}

if it lists any of the partition IDs (of any shape) recorded on

{slice_name}

's nodes in Step 2.

List the
```
Slice
```
custom resources in the cluster (as shown in Delete the slice) and describe each one following Monitor the status of the Slice custom resource to read
```
Spec
```
>
```
Partition Ids
```
,
```
Metadata
```
>
```
Labels
```
, and
```
Status
```
>
```
Conditions
```
(
```
Ready
```
reason):
- Treat any slice other than
```
{slice_name}
```
  whose
```
Spec
```
  >
```
Partition Ids
```
  includes any partition ID recorded on
```
{slice_name}
```
  's nodes in Step 2 (whether the exact ID in
```
{partition_ids}
```
  or another shape's partition ID on the same nodes) as conflicting. Record its name, its
```
Ready
```
  reason (such as
```
ACTIVE
```
  ), and whether Kueue manages it (Step 4, item 1).
To check partition-to-slice associations in Cloud Monitoring (including around
```
{issue_time}
```
), query the
```
kubernetes.io/accelerator/partition/state
```
metric listed in Monitor the metrics of the slice. Per Kubernetes metrics, this metric "is 1 if the partition state is detected, 0 otherwise", and its labels include
```
partition_id
```
,
```
slice_id
```
("ID of the associated slice"), and
```
state
```
("Health of the partition (e.g., HEALTHY, UNHEALTHY)"). Run the query for each
```
{partition_id}
```
recorded on
```
{slice_name}
```
's nodes in Step 2:

promql

kubernetes_io:accelerator_partition_state{
  monitored_resource="k8s_node_pool",
  cluster_name="{cluster_name}",
  partition_id="{partition_id}"
}

Read only time series whose value is
```
1
```
(where
```
state="HEALTHY"
```
means the partition is healthy and functional, per Verify the status of the nodes and the partitions).
Because Kubernetes metrics defines
```
slice_id
```
only as "ID of the associated slice", always confirm the conflicting slice from the
```
Slice
```
custom resources in the cluster rather than from
```
slice_id
```
alone: list the
```
Slice
```
custom resources as shown in Delete the slice, describe each one following Monitor the status of the Slice custom resource, and check which slice's
```
Spec
```
>
```
Partition Ids
```
includes the partition ID (or any other partition ID of any shape on the same nodes from Step 2).

Next steps based on Step 3 results:
- Another slice uses the nodes now: Proceed to Step 4.
- No other slice uses the nodes now, but the metric (value
  1
  ) showed an associated
  slice_id
  around
  {issue_time}
  : The conflict may already be over. Proceed to Step 4, item 1 (to re-create or retry
```
{slice_name}
```
  ) and then Step 4, item 4.
- Neither the current
  Slice
  custom resources nor the metric around
  {issue_time}
  shows another slice: Node overlap isn't confirmed. Hand off to
```
gke-ai-troubleshooting-tpu-dynamic-slices-monitoring
```
  or escalate to Cloud Customer Care (Step 4, item 4).

Step 4: Resolution

[High Risk]

Guardrails:
Never delete or change the other slice, or stop its workload, to free the nodes. Deleting a
Slice
custom resource starts dismantling that slice (
DEACTIVATING
, see Monitor the status of the Slice custom resource). The owner of that slice decides when to release it.
Never manually modify or delete
Slice
custom resources created by Kueue. Per Target your node pool to belong to a block or a sub-block, "Avoid manually modifying Slice custom resources created by Kueue."
Never change GKE-managed instance groups or VMs directly through Compute Engine, and never delete node pools to free a partition. Manage node pools and slices through GKE.

First, check whether Kueue manages

{slice_name}

or the conflicting slice:

Per Monitor the status of the slice, when a slice is created or actively used by a workload, the slice controller attaches the
```
kueue.gke.io/tpu-slice-used-by-namespace
```
and
```
kueue.gke.io/tpu-slice-used-by-workload
```
labels.
Per Optimize workload scheduling with slice reuse and Enable Slice reuse, when slice reuse is enabled, "Kueue marks the slice as idle at the end of a job instead of deleting it", and "Reusable slices are named dynamically with an
```
s-<topology>
```
prefix, decoupling them from specific workload names."

Then apply the matching option:

Wait for the conflicting slice to release the nodes and retry:
- If Kueue manages the conflicting slice (it has the
```
kueue.gke.io/...
```
  labels or an
```
s-<topology>
```
  name prefix): Do not suggest deleting the
```
Slice
```
  custom resource directly. Point the workload owner to Clean up (deleting the
```
JobSet
```
  triggers Kueue to delete the associated
```
Slice
```
  custom resources) and, if slice reuse keeps an idle
```
s-<topology>
```
  slice allocated after a job ends, to Enforce slice deletion per workload (
```
kueue.gke.io/tpu-enforce-slice-deletion: "true"
```
  ).
- If a custom scheduler or user created the conflicting slice directly: Its owner can delete it when finished, as described in Delete the slice.
- How
  {slice_name}
  retries: Check whether
```
{slice_name}
```
  already has
```
slice.gke.io/retry-on-failure: "true"
```
  in
```
Metadata
```
  >
```
Annotations
```
  . Per Create a Slice custom resource, when a
```
Slice
```
  custom resource is created with this annotation, "If the slice isn't created because of the
```
SliceCreationFailed
```
  status reason, the controller will retry until the slice is successfully formed." Because the documentation only shows configuring
```
slice.gke.io/retry-on-failure: "true"
```
  in the
```
Slice
```
  manifest when creating a
```
Slice
```
  custom resource (and doesn't state that adding the annotation to an existing slice already in
```
SliceCreationFailed
```
  starts a retry), if
```
{slice_name}
```
  wasn't created by Kueue and lacks the annotation, tell the user to delete
```
{slice_name}
```
  following Delete the slice and re-create it with
```
slice.gke.io/retry-on-failure: "true"
```
  in its manifest following Create a Slice custom resource.
Use available partitions whose nodes no other slice uses: If the user manages
```
Slice
```
custom resources directly (not through Kueue) and can't wait, direct them to Verify the status of the nodes and the partitions and Create a Slice custom resource so they can pick available partitions (
```
HEALTHY
```
partitions, or
```
DEGRADED
```
```
4x4x4
```
partitions) whose nodes aren't used by any other slice (across all partition shapes on those nodes) and create the
```
Slice
```
custom resource with them.
Schedule dynamic slices with Kueue and TAS instead of creating
Slice
custom resources directly: Per Target your node pool to belong to a block or a sub-block and Create a dynamic slice with Kueue and TAS, instead of creating
```
Slice
```
custom resources directly, teams can schedule GKE workloads with Kueue and Topology Aware Scheduling (TAS), where "Kueue automatically handles the creation and deletion of Slice custom resources" and admits a workload after its slice becomes active.
Verify or escalate: Repeat Step 1 following Monitor the status of the Slice custom resource:
- ACTIVE
  : The slice is fully formed, healthy, and ready to execute workloads.
- ACTIVE_DEGRADED
  : The slice is formed, and workloads can run, but performance might be impacted.
- Still
  SliceCreationFailed
  after no other slice uses the nodes, or
  FAILED
  : Hand off to
```
gke-ai-troubleshooting-tpu-dynamic-slices-monitoring
```
  , or advise the user to contact Cloud Customer Care following Before you contact support and Recommendations for efficient support.

gke-ai-troubleshooting-tpu-partition-overlap

NPX Install

Tags

SKILL.md Content

Troubleshoot TPU slice creation failures caused by partition overlap

Prerequisites

Diagnostic workflow

Step 0: Collect context
`[Low Risk]`

Step 1: Confirm the
`SliceCreationFailed`
status
`[Low Risk]`

Step 2: Rule out the other documented causes
`[Low Risk]`

Step 3: Find the slice that uses the same nodes
`[Low Risk]`

Step 4: Resolution
`[High Risk]`