gke-workload-troubleshooting

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

GKE Workload Troubleshooting Skill

GKE工作负载故障排查Skill

Use this skill to systematically diagnose and resolve failures in application workloads deployed in GKE clusters. This skill operates non-interactively and enforces a read-only diagnostics boundary before proposing manifest or config corrections.

使用本Skill可系统性诊断并解决部署在GKE集群中的应用工作负载故障。本Skill以非交互式运行，在提出清单或配置修正方案前，会严格遵循只读诊断边界。

🔍 Diagnostic Workflow

🔍 诊断流程

Step 0: Non-Interactive Context Discovery & Time Window Definition

步骤0：非交互式上下文发现与时间窗口定义

Parameter Extraction: Extract required context (
```
project_id
```
,
```
cluster_name
```
,
```
cluster_location
```
,
```
workload_name
```
,
```
workload_namespace
```
) non-interactively from the user prompt, active
```
SETTINGS.md
```
, or active environment defaults:
- Default
```
workload_namespace
```
  to
```
default
```
  if omitted.
- Infer missing cluster parameters from active environment (
```
kubectl config current-context
```
  or
```
gcloud config get-value project
```
  ).
- Prioritize non-interactive context discovery from prompts and environment defaults to ensure autonomous execution flow.
Cluster Credentials & Fallback Mode:
- Attempt credential fetch:
```
gcloud container clusters get-credentials {cluster_name} --region/--zone {cluster_location}
```
- Fallback / Dry-Run Mode: If the cluster is unreachable, non-existent, or live command execution fails (such as in sandboxed evaluations, dry-run mode, or offline analysis):
  - Limit retry attempts to avoid resource exhaustion and context overflow in unreachable cluster scenarios.
  - Immediately present the exact sequence of
```
kubectl
```
    diagnostic commands for the human operator to run.
  - Synthesize the root cause analysis and output the proposed GitOps manifest fix based on the reported symptoms.
Time Handling & Fallbacks:
- Determine Issue Timestamp ({issue_time}):
  - Specific Time Provided: If the user provides a specific timestamp, use it as
```
{issue_time}
```
    .
  - Relative Time Provided (e.g., "5 minutes ago"): Dynamically calculate the corresponding UTC timestamp based on current system time, and use it as
```
{issue_time}
```
    .
  - No Time Provided (Default): Use current system time as
```
{issue_time}
```
    .
- Window Calculation: Center a 1-hour query window around
```
{issue_time}
```
  (
```
start_time
```
  =
```
{issue_time} - 30m
```
  ,
```
end_time
```
  =
```
{issue_time} + 30m
```
  ).

参数提取：从用户提示、活跃的
```
SETTINGS.md
```
或活跃环境默认值中，非交互式提取所需上下文（
```
project_id
```
、
```
cluster_name
```
、
```
cluster_location
```
、
```
workload_name
```
、
```
workload_namespace
```
）：
- 若未指定
```
workload_namespace
```
  ，默认设为
```
default
```
  。
- 从活跃环境（
```
kubectl config current-context
```
  或
```
gcloud config get-value project
```
  ）推断缺失的集群参数。
- 优先从提示和环境默认值中进行非交互式上下文发现，以确保执行流程的自主性。
集群凭据与回退模式：
- 尝试获取凭据：
```
gcloud container clusters get-credentials {cluster_name} --region/--zone {cluster_location}
```
- 回退/干运行模式：若集群无法访问、不存在或实时命令执行失败（例如在沙箱评估、干运行模式或离线分析场景中）：
  - 限制重试次数，避免在集群无法访问的场景下耗尽资源或导致上下文溢出。
  - 立即向人工操作员展示完整的
```
kubectl
```
    诊断命令序列，供其运行。
  - 综合分析根本原因，并基于报告的症状输出建议的GitOps清单修复方案。
时间处理与回退机制：
- 确定问题时间戳({issue_time})：
  - 提供了具体时间：若用户提供了具体时间戳，将其作为
```
{issue_time}
```
    。
  - 提供了相对时间（如“5分钟前”）：基于当前系统时间动态计算对应的UTC时间戳，并将其作为
```
{issue_time}
```
    。
  - 未提供时间（默认）：将当前系统时间作为
```
{issue_time}
```
    。
- 窗口计算：以
```
{issue_time}
```
  为中心设置1小时的查询窗口（
```
start_time
```
  =
```
{issue_time} - 30m
```
  ，
```
end_time
```
  =
```
{issue_time} + 30m
```
  ）。

Step 1: Analyze Pod Status and Conditions

步骤1：分析Pod状态与条件

Inspect the workload's active pod states and controller status.

Diagnostic Commands:

bash

undefined

检查工作负载的活跃Pod状态和控制器状态。

诊断命令：

bash

undefined

1. Inspect the deployment's actual selector labels:

1. 检查Deployment的实际选择器标签：

kubectl get deployment {workload_name} -n {workload_namespace} -o jsonpath='{.spec.selector.matchLabels}'

2. Query the pods using the returned labels, for example:

2. 使用返回的标签查询Pod，例如：

kubectl get pods -l {selector_labels} -n {workload_namespace} kubectl get deploy/{workload_name} -n {workload_namespace} -o yaml

undefined

kubectl get pods -l {selector_labels} -n {workload_namespace} kubectl get deploy/{workload_name} -n {workload_namespace} -o yaml

undefined

Diagnostic Decision Tree:

诊断决策树：

Phase: Pending:
- The Pod cannot schedule on any node. Proceed directly to Step 2 (Query Namespace Events).
State: CrashLoopBackOff / Error:
- Container is booting but exiting repeatedly. Check the terminated status using:
bash
```
kubectl get pod {pod_name} -n {workload_namespace} -o jsonpath='{.status.containerStatuses[*].lastState.terminated}'
```
- ExitCode: 137 (OOMKilled): Memory limit reached. Proceed to Step 3 (Inspect Logs) and inspect container startup command to differentiate between an application-level memory leak/loop vs an infrastructure capacity limit mismatch, then proceed to Step 5 to propose fixes.
- ExitCode: 1 or other non-zero codes: The application code crashed. Proceed directly to Step 3 (Inspect Logs).
State: ContainerCreating:
- The container is blocked during volume mount, networking setup, or image pulling. Proceed directly to Step 2 (Query Namespace Events).

阶段：Pending：
- Pod无法调度到任何节点。直接进入步骤2（查询命名空间事件）。
状态：CrashLoopBackOff / Error：
- 容器正在启动但反复退出。使用以下命令检查终止状态：
bash
```
kubectl get pod {pod_name} -n {workload_namespace} -o jsonpath='{.status.containerStatuses[*].lastState.terminated}'
```
- 退出码：137（OOMKilled）：达到内存限制。进入步骤3（检查日志），并检查容器启动命令以区分应用级内存泄漏/循环与基础设施容量限制不匹配的情况，然后进入步骤5提出修复方案。
- 退出码：1或其他非零码：应用代码崩溃。直接进入步骤3（检查日志）。
状态：ContainerCreating：
- 容器在卷挂载、网络设置或镜像拉取过程中受阻。直接进入步骤2（查询命名空间事件）。

Step 2: Query Namespace Events

步骤2：查询命名空间事件

Look for infrastructure, volume, image, or scheduling alerts in GKE.

Diagnostic Command:

bash

kubectl get events -n {workload_namespace} --sort-by='.metadata.creationTimestamp'

在GKE中查找基础设施、存储卷、镜像或调度相关的告警。

诊断命令：

bash

kubectl get events -n {workload_namespace} --sort-by='.metadata.creationTimestamp'

Or query Cloud Logging for historical GKE events within the time window:

或在时间窗口内查询Cloud Logging中的历史GKE事件：

gcloud logging read "resource.type="k8s_cluster" AND logName="projects/{project_id}/logs/events" AND jsonPayload.involvedObject.namespace="{workload_namespace}"" --start-time="{start_time}" --end-time="{end_time}" --project="{project_id}"


*Note: Retrieve the sorted events list and manually inspect the event timestamps
(CreationTimestamp/LastSeen) to identify failures occurring within the
`{start_time}` and `{end_time}` window.*


*注意：获取排序后的事件列表，并手动检查事件时间戳（CreationTimestamp/LastSeen），以识别发生在`{start_time}`和`{end_time}`窗口内的故障。*

Signature Identifiers:

特征标识：

FailedScheduling
: Node resource exhaustion. Look for messages like
```
0/3 nodes are available: 3 Insufficient memory.
```
or missing node affinity tolerations (e.g. Spot VM taints).

FailedMount
:

Missing PersistentVolumeClaim (
```
PVC
```
).
Missing Secret (
```
Secret "{secret_name}" not found
```
).
Missing ConfigMap (
```
ConfigMap "{configmap_name}" not found
```
).

Failed
/
BackOff
(Image Pull):
- Wrong image tag, missing image registry authentication (e.g., ImagePullBackOff).
- Resolution Steps for Wrong Image Tag:
- Identify the failing container image name and the invalid tag.
- Check the Git repository history for the last known working image tag for this workload. Run
```
git log -p -S "{image_name}" -- {manifest_file_path}
```
  (or use
```
git log
```
  on the folder containing manifests) to identify the previous working tag in Git.
- If the invalid tag is a recent change in git history, compare it to the tag from the last successful commit.
- Propose reverting the image tag to the last working version, or correcting the tag version in the manifest patch.

FailedScheduling
：节点资源耗尽。查找类似
```
0/3 nodes are available: 3 Insufficient memory.
```
的消息，或缺失的节点亲和性容忍度（例如Spot VM污点）。

FailedMount
：

缺少PersistentVolumeClaim（
```
PVC
```
）。
缺少Secret（
```
Secret "{secret_name}" not found
```
）。
缺少ConfigMap（
```
ConfigMap "{configmap_name}" not found
```
）。

Failed
/
BackOff
（镜像拉取）：
- 镜像标签错误、缺少镜像仓库认证（例如ImagePullBackOff）。
- 镜像标签错误的解决步骤：
- 识别失败的容器镜像名称和无效标签。
- 检查Git仓库历史，查找该工作负载最后已知可用的镜像标签。运行
```
git log -p -S "{image_name}" -- {manifest_file_path}
```
  （或对包含清单的文件夹运行
```
git log
```
  ），以在Git中找到之前可用的标签。
- 如果无效标签是Git历史中的最近变更，将其与最后一次成功提交的标签进行对比。
- 建议将镜像标签恢复到最后可用版本，或在清单补丁中修正标签版本。

Step 3: Inspect Application Logs

步骤3：检查应用日志

Extract exceptions and stack traces from the application runtime.

Diagnostic Commands:

bash

undefined

从应用运行时提取异常和堆栈跟踪信息。

诊断命令：

bash

undefined

Check current active log stream (handles multi-container pods)

检查当前活跃的日志流（支持多容器Pod）

kubectl logs {pod_name} -n {workload_namespace} --all-containers --tail=100

Check logs from previously terminated container instances (handles multi-container pods)

检查已终止容器实例的日志（支持多容器Pod）

kubectl logs {pod_name} -n {workload_namespace} --all-containers -p --tail=100

undefined

kubectl logs {pod_name} -n {workload_namespace} --all-containers -p --tail=100

undefined

Signature Identifiers:

特征标识：

Out-of-Memory (OOM) Analysis: Inspect container logs and startup commands (
```
spec.containers[*].command
```
). Differentiate between an Application Code Leak/Loop (unbounded array appending, memory leak signatures) vs an Infrastructure Capacity Ceiling Mismatch (legitimate workload demand exceeding limits).
Stack Trace / Unhandled Exception: Look for language-specific stack traces (e.g.,
```
panic:
```
,
```
NullPointerException
```
,
```
Traceback (most recent call)
```
). This indicates an application bug.
Egress Network Timeout: Look for connection timeouts (e.g.,
```
Connection timed out
```
,
```
dial tcp: i/o timeout
```
). Proceed to Step 4 (Verify Connectivity).
Permission Errors (ReadOnlyRootFilesystem): Look for write errors (e.g.,
```
Read-only file system
```
,
```
Permission denied
```
when writing to
```
/tmp
```
or
```
/var/log
```
). Propose adding an
```
emptyDir
```
volume mount to that directory in the manifest.

内存不足（OOM）分析：检查容器日志和启动命令（
```
spec.containers[*].command
```
）。区分应用代码泄漏/循环（无界数组追加、内存泄漏特征）与基础设施容量上限不匹配（合法工作负载需求超过限制）。
堆栈跟踪/未处理异常：查找特定语言的堆栈跟踪（例如
```
panic:
```
、
```
NullPointerException
```
、
```
Traceback (most recent call)
```
）。这表明存在应用程序bug。
出口网络超时：查找连接超时信息（例如
```
Connection timed out
```
、
```
dial tcp: i/o timeout
```
）。进入步骤4（验证连通性）。
权限错误（ReadOnlyRootFilesystem）：查找写入错误（例如
```
Read-only file system
```
、写入
```
/tmp
```
或
```
/var/log
```
时的
```
Permission denied
```
）。建议在清单中为该目录添加
```
emptyDir
```
卷挂载。

Step 4: Verify Service Connectivity and Network Policies

步骤4：验证服务连通性与网络策略

Troubleshoot connection drops to other services.

Diagnostic Commands:

bash

undefined

排查与其他服务的连接中断问题。

诊断命令：

bash

undefined

Verify target endpoint is active

验证目标端点是否活跃

kubectl get endpoints {target_service_name} -n {target_namespace}

Query network policies inside namespace

查询命名空间内的网络策略

kubectl get networkpolicies -n {workload_namespace} -o yaml

undefined

kubectl get networkpolicies -n {workload_namespace} -o yaml

undefined

Logic & Dry-Run Fallback:

逻辑与干运行回退：

Live Cluster Mode:
- If
```
kubectl get endpoints
```
  returns an empty list, the target microservice itself is failing to schedule or boot (troubleshoot target service).
- If endpoints exist but logs show timeouts, analyze
```
NetworkPolicy
```
  egress blocks to verify if egress traffic to the target service's IP/port is allowed.
Sandboxed / Dry-Run Mode:
- If live
```
kubectl
```
  queries fail or cluster connection is unavailable, do NOT retry live cluster access or enter repetitive connection attempts.
- Immediately inspect the application source code (e.g.
```
worker.py
```
  ,
```
app.go
```
  , DB connection strings) or Deployment manifests to identify the target service hostname (e.g.
```
account-db
```
  ) and destination port (e.g.
```
5432
```
  ).
- Present the exact
```
kubectl get endpoints
```
  and
```
kubectl get networkpolicies
```
  commands for the user, and synthesize the required
```
NetworkPolicy
```
  egress patch allowing traffic to the target service and port.

实时集群模式：
- 若
```
kubectl get endpoints
```
  返回空列表，说明目标微服务本身无法调度或启动（需排查目标服务）。
- 若端点存在但日志显示超时，分析
```
NetworkPolicy
```
  的出口规则，验证是否允许向目标服务的IP/端口发送出口流量。
沙箱/干运行模式：
- 若实时
```
kubectl
```
  查询失败或集群连接不可用，请勿重试集群访问或进入重复连接尝试。
- 立即检查应用源代码（例如
```
worker.py
```
  、
```
app.go
```
  、数据库连接字符串）或Deployment清单，识别目标服务主机名（例如
```
account-db
```
  ）和目标端口（例如
```
5432
```
  ）。
- 向用户展示完整的
```
kubectl get endpoints
```
  和
```
kubectl get networkpolicies
```
  命令，并生成允许向目标服务和端口发送流量的
```
NetworkPolicy
```
  出口补丁建议。

Step 5: Propose GitOps Correction

步骤5：提出GitOps修正方案

Following the GitOps boundary, do not apply patches directly to the cluster.

Synthesize the root cause analysis for the human operator (e.g. "payment-api is failing with exit code 137 because its memory limit is set to 256Mi while actual usage spiked to 270Mi").
Generate the corrected YAML manifest patch (e.g. increase memory limits, add missing Secret mounts, or add tolerations for Spot nodes).
Check if a branch or Pull Request (PR) already exists for this workload/failure. If so, update the existing branch/PR or notify the user instead of creating a duplicate. Otherwise, create a branch, commit the change, open a Pull Request (PR) on GitHub, and conclude the workflow (do not wait for human merge).

遵循GitOps边界，请勿直接向集群应用补丁。

为人工操作员综合分析根本原因（例如："payment-api因内存限制设为256Mi，而实际使用量飙升至270Mi，导致退出码137故障"）。
生成修正后的YAML清单补丁（例如增加内存限制、添加缺失的Secret挂载、或为Spot节点添加容忍度）。
检查是否已存在针对该工作负载/故障的分支或拉取请求（PR）。若存在，则更新现有分支/PR或通知用户，而非创建重复内容。否则，创建分支、提交变更、在GitHub上打开拉取请求（PR）并结束流程（无需等待人工合并）。