gke-ai-troubleshooting-tpu-vbar-oom

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

TPU Connection Failure and VBAR OOM Troubleshooting

TPU连接失败与VBAR OOM故障排查

Use this skill to systematically diagnose and prevent

vbar_control_agent

segfaults and Out-Of-Memory (OOM) errors on TPU v6e nodes.

使用本技能可系统性诊断并预防TPU v6e节点上的

vbar_control_agent

段错误和内存不足（OOM）错误。

⚠️ Prerequisites

⚠️ 前置条件

Cloud Logging must be enabled for the project.
Access to the project and cluster via
```
gcloud
```
or equivalent tool.

项目必须已启用Cloud Logging。
可通过
```
gcloud
```
或等效工具访问项目及集群。

🔍 Diagnostic Workflow

🔍 诊断流程

Step 0: Context Acquisition & Time Window Definition

步骤0：上下文获取与时间窗口定义

Independently gather required context using available GCP/GKE tools or use the provided

{variable}

placeholders:

```
{project_id}
```
: The GCP Project ID (e.g.,
```
customer-ai-project-123
```
).
```
{cluster_name}
```
: The GKE Cluster Name (e.g.,
```
tpu-cluster-prod
```
).
```
{node_name}
```
: The Node Name or Instance ID (e.g.,
```
tpu-node-1
```
).
```
{workload_name}
```
: The Workload Name / JobSet Name (e.g.,
```
my-training-job-456
```
).
```
{namespace}
```
: The Workload Namespace.
```
{issue_time}
```
: The timestamp of the issue (e.g.,
```
2026-04-14T20:00:00Z
```
).

使用可用的GCP/GKE工具独立收集所需上下文，或使用提供的

{variable}

占位符：

```
{project_id}
```
：GCP项目ID（例如：
```
customer-ai-project-123
```
）。
```
{cluster_name}
```
：GKE集群名称（例如：
```
tpu-cluster-prod
```
）。
```
{node_name}
```
：节点名称或实例ID（例如：
```
tpu-node-1
```
）。
```
{workload_name}
```
：工作负载名称/JobSet名称（例如：
```
my-training-job-456
```
）。
```
{namespace}
```
：工作负载命名空间。
```
{issue_time}
```
：问题发生的时间戳（例如：
```
2026-04-14T20:00:00Z
```
）。

Time Handling & Execution Rules

时间处理与执行规则

Window Calculation: If an issue timestamp

{issue_time}

is provided, calculate the query time window as

[{issue_time} - 30m]

[{issue_time} + 30m]

Let
```
{start_time}
```
=
```
{issue_time} - 30m
```
Let
```
{end_time}
```
=
```
{issue_time} + 30m
```

Informational vs. Live Execution: If the user request is informational or query-formulation (e.g. "How can I check...", "How do I determine..."), or if live GCP project resources are not actively targetable, directly output the calculated time window, log names, and Cloud Logging filter templates without attempting live log execution commands.

窗口计算：若提供了问题时间戳

{issue_time}

，则将查询时间窗口计算为

[{issue_time} - 30m]

至

[{issue_time} + 30m]

。

令
```
{start_time}
```
=
```
{issue_time} - 30m
```
令
```
{end_time}
```
=
```
{issue_time} + 30m
```

信息查询与实时执行：若用户请求为信息查询或查询公式制定（例如“我如何检查...”, “我如何确定...”），或无法直接访问实时GCP项目资源，则直接输出计算后的时间窗口、日志名称和Cloud Logging过滤模板，无需尝试执行实时日志命令。

Step 1: Check for

vbar_control_agent

OOMs

步骤1：检查

vbar_control_agent

OOM情况

Look for specific

out of memory

messages from

vbar_control_agent

in serial console logs (

serialconsole.googleapis.com%2fserial_port_1_output

Tool to use:
```
query_logs
```
(for live diagnostics)
Filter Templates:

Serial Console Logs (OOMs):

sql

logName="projects/{project_id}/logs/serialconsole.googleapis.com%2fserial_port_1_output"
AND labels."compute.googleapis.com/resource_name"="{node_name}"
AND SEARCH(text_payload, "Memory cgroup out of memory: Killed process .* (vbar_control_ag)")
AND timestamp >= "{start_time}"
AND timestamp <= "{end_time}"

Logic: Presence of

Memory cgroup out of memory

messages related to

vbar_control_agent

. Stack traces pointing to

libtpu::tpunetd::VBARControlHelper::MetricsReadFromVBAR

are a strong indicator.

Automation: Proceed to next step automatically after reporting findings.
Reference: See
```
references/failure_signatures.md
```
for example log patterns.

在串行控制台日志（

serialconsole.googleapis.com%2fserial_port_1_output

）中查找来自

vbar_control_agent

的特定“out of memory”消息。

使用工具：
```
query_logs
```
（用于实时诊断）
过滤模板：

串行控制台日志（OOM相关）：

sql

logName="projects/{project_id}/logs/serialconsole.googleapis.com%2fserial_port_1_output"
AND labels."compute.googleapis.com/resource_name"="{node_name}"
AND SEARCH(text_payload, "Memory cgroup out of memory: Killed process .* (vbar_control_ag)")
AND timestamp >= "{start_time}"
AND timestamp <= "{end_time}"

逻辑：存在与
```
vbar_control_agent
```
相关的“Memory cgroup out of memory”消息。指向
```
libtpu::tpunetd::VBARControlHelper::MetricsReadFromVBAR
```
的堆栈跟踪是明确的指示信号。
自动化：报告结果后自动进入下一步。
参考：请参阅
```
references/failure_signatures.md
```
查看示例日志模式。

Step 2: Investigate

tpu-device-plugin

Metrics Fetch Failures [Low Risk]

步骤2：排查

tpu-device-plugin

指标获取失败 [低风险]

Check if

tpu-device-plugin

is reporting metric fetch failures.

Tool to use:
```
query_logs
```
Filter Template:

sql

resource.type="k8s_container"
AND resource.labels.project_id="{project_id}"
AND resource.labels.cluster_name="{cluster_name}"
AND resource.labels.container_name="tpu-device-plugin"
AND severity=ERROR
AND textPayload:"metrics fetch failed for .* deviceID and .* device path with error: checksum didn't match with the metrics data. Corrupt data found"
AND timestamp >= "{start_time}"
AND timestamp <= "{end_time}"

Logic: Errors indicating "metrics fetch failed" with "checksum didn't match" suggest vBAR memory corruption.
Automation: Proceed to next step automatically after reporting findings.

检查

tpu-device-plugin

是否报告指标获取失败。

使用工具：
```
query_logs
```
过滤模板：

sql

resource.type="k8s_container"
AND resource.labels.project_id="{project_id}"
AND resource.labels.cluster_name="{cluster_name}"
AND resource.labels.container_name="tpu-device-plugin"
AND severity=ERROR
AND textPayload:"metrics fetch failed for .* deviceID and .* device path with error: checksum didn't match with the metrics data. Corrupt data found"
AND timestamp >= "{start_time}"
AND timestamp <= "{end_time}"

逻辑：包含“metrics fetch failed”和“checksum didn't match”的错误表明vBAR内存已损坏。
自动化：报告结果后自动进入下一步。

Step 3: Check for Custom Metrics Collection Usage [Low Risk]

步骤3：检查自定义指标采集使用情况 [低风险]

Inspect cluster configurations, workloads, or container specs to determine if custom TPU metrics collection mechanisms are deployed.

Action: Check if custom scripts or agents (e.g., using
```
libtpu.sdk.tpumonitoring
```
) are deployed that frequently query
```
GetHostMetrics
```
from
```
vBAR Control Agent
```
.

Verification Commands:

Kubectl Search (Inspect workload env/specs):

bash

kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{"\t"}{.spec.containers[*].image}{"\n"}{end}'

Log Search Filter (
query_logs
):

sql

resource.type="k8s_container"
AND resource.labels.project_id="{project_id}"
AND resource.labels.cluster_name="{cluster_name}"
AND textPayload:"libtpu.sdk.tpumonitoring"
AND timestamp >= "{start_time}"
AND timestamp <= "{end_time}"

Logic: Confirmation of custom metrics collection helps confirm the race condition hypothesis.

检查集群配置、工作负载或容器规格，确定是否部署了自定义TPU指标采集机制。

操作：检查是否部署了自定义脚本或代理（例如使用
```
libtpu.sdk.tpumonitoring
```
）频繁从
```
vBAR Control Agent
```
查询
```
GetHostMetrics
```
。

验证命令：

Kubectl搜索（检查工作负载环境/规格）：

bash

kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{"\t"}{.spec.containers[*].image}{"\n"}{end}'

日志搜索过滤（
query_logs
）：

sql

resource.type="k8s_container"
AND resource.labels.project_id="{project_id}"
AND resource.labels.cluster_name="{cluster_name}"
AND textPayload:"libtpu.sdk.tpumonitoring"
AND timestamp >= "{start_time}"
AND timestamp <= "{end_time}"

逻辑：确认存在自定义指标采集有助于验证竞态条件假设。

🛠️ Resolution Workflow

🛠️ 解决流程

Resolution 1: Temporarily Disable Custom Metrics Collection [High Risk]

解决方案1：临时禁用自定义指标采集 [高风险]

If a custom metrics collection agent is identified, recommend disabling it.

Action: Recommend disabling the custom metrics collector.
Justification: Prevents reads from vBAR during device resets, stopping crashes and OOMs.

若识别到自定义指标采集代理，建议禁用它。

操作：建议禁用自定义指标采集器。
理由：避免在设备重置期间读取vBAR，从而阻止崩溃和OOM错误。

Resolution 2: Await

vbar_control_agent

Resiliency Update [Low Risk]

解决方案2：等待

vbar_control_agent

弹性更新 [低风险]

Advise that a permanent fix will be available in a future GKE version.

Action: Recommend upgrading GKE when the fix is available.
Justification: The updated agent will be resilient to memory corruption and gracefully handle reads from unbound vBARs.

告知用户永久修复将在未来的GKE版本中提供。

操作：建议在修复可用时升级GKE。
理由：更新后的代理将能抵御内存损坏，并优雅处理未绑定vBAR的读取操作。

📋 copypaste checklist

📋 复制粘贴检查清单

Acquire context and compute
```
[{start_time}, {end_time}]
```
window.
Check for
```
vbar_control_agent
```
segfaults and OOMs using
```
query_logs
```
.
Investigate
```
tpu-device-plugin
```
failures using
```
query_logs
```
.
Inspect for custom metrics collection usage.
Advise disabling custom metrics collection if applicable.
Advise awaiting resiliency update.

获取上下文并计算
```
[{start_time}, {end_time}]
```
时间窗口。
使用
```
query_logs
```
检查
```
vbar_control_agent
```
段错误和OOM情况。
使用
```
query_logs
```
排查
```
tpu-device-plugin
```
故障。
检查自定义指标采集使用情况。
若适用，建议禁用自定义指标采集。
建议等待弹性更新。

gke-ai-troubleshooting-tpu-vbar-oom

Original

Translation

TPU Connection Failure and VBAR OOM Troubleshooting

TPU连接失败与VBAR OOM故障排查

⚠️ Prerequisites

⚠️ 前置条件

🔍 Diagnostic Workflow

🔍 诊断流程

Step 0: Context Acquisition & Time Window Definition

步骤0：上下文获取与时间窗口定义

Time Handling & Execution Rules

时间处理与执行规则

Step 1: Check for
`vbar_control_agent`
OOMs

步骤1：检查
`vbar_control_agent`
OOM情况

Step 2: Investigate
`tpu-device-plugin`
Metrics Fetch Failures [Low Risk]

步骤2：排查
`tpu-device-plugin`
指标获取失败 [低风险]

Step 3: Check for Custom Metrics Collection Usage [Low Risk]

步骤3：检查自定义指标采集使用情况 [低风险]

🛠️ Resolution Workflow

🛠️ 解决流程

Resolution 1: Temporarily Disable Custom Metrics Collection [High Risk]

解决方案1：临时禁用自定义指标采集 [高风险]

Resolution 2: Await
`vbar_control_agent`
Resiliency Update [Low Risk]

解决方案2：等待
`vbar_control_agent`
弹性更新 [低风险]

📋 copypaste checklist

📋 复制粘贴检查清单

gke-ai-troubleshooting-tpu-vbar-oom

Original

Translation

TPU Connection Failure and VBAR OOM Troubleshooting

TPU连接失败与VBAR OOM故障排查

⚠️ Prerequisites

⚠️ 前置条件

🔍 Diagnostic Workflow

🔍 诊断流程

Step 0: Context Acquisition & Time Window Definition

步骤0：上下文获取与时间窗口定义

Time Handling & Execution Rules

时间处理与执行规则

Step 1: Check for vbar_control_agent OOMs

步骤1：检查vbar_control_agent OOM情况

Step 2: Investigate tpu-device-plugin Metrics Fetch Failures [Low Risk]

步骤2：排查tpu-device-plugin指标获取失败 [低风险]

Step 3: Check for Custom Metrics Collection Usage [Low Risk]

步骤3：检查自定义指标采集使用情况 [低风险]

🛠️ Resolution Workflow

🛠️ 解决流程

Resolution 1: Temporarily Disable Custom Metrics Collection [High Risk]

解决方案1：临时禁用自定义指标采集 [高风险]

Resolution 2: Await vbar_control_agent Resiliency Update [Low Risk]

解决方案2：等待vbar_control_agent弹性更新 [低风险]

📋 copypaste checklist

📋 复制粘贴检查清单

Step 1: Check for
`vbar_control_agent`
OOMs

步骤1：检查
`vbar_control_agent`
OOM情况

Step 2: Investigate
`tpu-device-plugin`
Metrics Fetch Failures [Low Risk]

步骤2：排查
`tpu-device-plugin`
指标获取失败 [低风险]

Resolution 2: Await
`vbar_control_agent`
Resiliency Update [Low Risk]

解决方案2：等待
`vbar_control_agent`
弹性更新 [低风险]