k8s-troubleshooter

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Kubernetes Troubleshooter & Incident Response

Kubernetes故障排查与事件响应

Systematic approach to diagnosing and resolving Kubernetes issues in production environments.

针对生产环境中Kubernetes问题的系统性诊断与解决方法。

Core Troubleshooting Workflow

核心故障排查流程

Follow this systematic approach for any Kubernetes issue:

针对任何Kubernetes问题，遵循以下系统性方法：

1. Gather Context

1. 收集上下文信息

What is the observed symptom?
When did it start?
What changed recently (deployments, config, infrastructure)?
What is the scope (single pod, service, node, cluster)?
What is the business impact (severity level)?

观察到的症状是什么？
症状何时开始出现？
最近有哪些变更（部署、配置、基础设施）？
影响范围是单个pod、服务、节点还是整个集群？
业务影响程度如何（严重级别）？

2. Initial Triage

2. 初步分类排查

Run cluster health check:

bash

undefined

运行集群健康检查：

bash

undefined

Check node status and health

检查节点状态与健康状况

kubectl get nodes

Find non-running pods across all namespaces

查找所有命名空间中未运行的pod

kubectl get pods -A --field-selector status.phase!=Running

Check node resource usage

检查节点资源使用情况

kubectl top nodes


This provides an overview of:
- Node health status
- Pending and failed pods across all namespaces
- Node resource utilization

kubectl top nodes


这将提供以下概览信息：
- 节点健康状态
- 所有命名空间中处于Pending和Failed状态的pod
- 节点资源利用率

3. Deep Dive Investigation

3. 深入调查

Based on triage results, focus investigation:

For Namespace-Level Issues:

bash

python3 scripts/check_namespace.py <namespace>

This provides comprehensive namespace health:

Pod status (running, pending, failed, crashlooping)
Service health and endpoints
Deployment availability
PVC status
Resource quota usage
Recent events
Actionable recommendations

For Pod Issues:

bash

undefined

根据初步排查结果，聚焦调查方向：

针对命名空间级问题：

bash

python3 scripts/check_namespace.py <namespace>

这将提供全面的命名空间健康信息：

Pod状态（运行中、Pending、Failed、CrashLoopBackOff）
服务健康状况与端点
部署可用性
PVC状态
资源配额使用情况
近期事件
可执行的建议

针对Pod问题：

bash

undefined

Get full pod details (status, events, conditions, resource config)

获取完整的Pod详情（状态、事件、条件、资源配置）

kubectl describe pod <pod-name> -n <namespace>

Check current and previous container logs

查看当前及之前的容器日志

kubectl logs <pod-name> -n <namespace> kubectl logs <pod-name> -n <namespace> --previous

Get events specific to the pod

获取与该Pod相关的特定事件

kubectl get events -n <namespace> --field-selector involvedObject.name=<pod-name>


This reveals:
- Pod phase and readiness
- Container statuses and states
- Restart counts and exit codes
- Recent events and scheduling decisions
- Resource requests and limits

**For additional investigations:**
- Check all namespace events: `kubectl get events -n <namespace> --sort-by='.lastTimestamp'`

kubectl get events -n <namespace> --field-selector involvedObject.name=<pod-name>


这将揭示：
- Pod阶段与就绪状态
- 容器状态
- 重启次数与退出码
- 近期事件与调度决策
- 资源请求与限制

**其他调查命令：**
- 查看所有命名空间事件：`kubectl get events -n <namespace> --sort-by='.lastTimestamp'`

4. Identify Root Cause

4. 确定根本原因

Consult references/common_issues.md for detailed information on:

ImagePullBackOff / ErrImagePull
CrashLoopBackOff
Pending Pods
OOMKilled
Node issues (NotReady, DiskPressure)
Networking failures
Storage/PVC issues
Resource quotas and throttling
RBAC permission errors

Each issue includes:

Symptoms
Common causes
Diagnostic commands
Remediation steps
Prevention strategies

参考references/common_issues.md获取以下问题的详细信息：

ImagePullBackOff / ErrImagePull
CrashLoopBackOff
Pending状态的Pod
OOMKilled
节点问题（NotReady、DiskPressure）
网络故障
存储/PVC问题
资源配额与限流
RBAC权限错误

每个问题包含：

症状
常见原因
诊断命令
修复步骤
预防策略

5. Apply Remediation

5. 实施修复

Follow remediation steps from common_issues.md based on root cause identified.

Always:

Test fixes in non-production first if possible
Document actions taken
Monitor for effectiveness
Have rollback plan ready

根据确定的根本原因，遵循common_issues.md中的修复步骤操作。

始终遵循：

尽可能先在非生产环境测试修复方案
记录已执行的操作
监控修复效果
准备回滚计划

6. Verify & Monitor

6. 验证与监控

After applying fix:

Verify issue is resolved
Monitor for recurrence (15-30 minutes minimum)
Check related systems
Update documentation

应用修复后：

验证问题已解决
监控是否复发（至少15-30分钟）
检查相关系统
更新文档

Incident Response

事件响应

For production incidents, follow structured response in references/incident_response.md:

Severity Assessment:

SEV-1 (Critical): Complete outage, data loss, security breach
SEV-2 (High): Major degradation, significant user impact
SEV-3 (Medium): Minor impairment, workaround available
SEV-4 (Low): Cosmetic, minimal impact

Incident Phases:

Detection - Identify and assess
Triage - Determine scope and impact
Investigation - Find root cause
Resolution - Apply fix
Post-Incident - Document and improve

Common Incident Scenarios:

Complete cluster outage
Service degradation
Node failure
Storage issues
Security incidents

See references/incident_response.md for detailed playbooks.

针对生产事件，遵循references/incident_response.md中的结构化响应流程：

严重级别评估：

SEV-1（关键）：完全 outage、数据丢失、安全漏洞
SEV-2（高）：严重性能下降、显著用户影响
SEV-3（中）：轻微功能受损、存在 workaround
SEV-4（低）：外观问题、影响极小

事件阶段：

检测 - 识别并评估事件
分类 - 确定范围与影响
调查 - 找到根本原因
解决 - 应用修复方案
事后复盘 - 记录并改进

常见事件场景：

集群完全 outage
服务性能下降
节点故障
存储问题
安全事件

查看references/incident_response.md获取详细的执行手册。

Quick Reference Commands

快速参考命令

Cluster Overview

集群概览

bash

kubectl cluster-info
kubectl get nodes
kubectl get pods --all-namespaces | grep -v Running
kubectl get events --all-namespaces --sort-by='.lastTimestamp' | tail -20

bash

kubectl cluster-info
kubectl get nodes
kubectl get pods --all-namespaces | grep -v Running
kubectl get events --all-namespaces --sort-by='.lastTimestamp' | tail -20

Pod Diagnostics

Pod诊断

bash

kubectl describe pod <pod> -n <namespace>
kubectl logs <pod> -n <namespace>
kubectl logs <pod> -n <namespace> --previous
kubectl exec -it <pod> -n <namespace> -- /bin/sh
kubectl get pod <pod> -n <namespace> -o yaml

bash

kubectl describe pod <pod> -n <namespace>
kubectl logs <pod> -n <namespace>
kubectl logs <pod> -n <namespace> --previous
kubectl exec -it <pod> -n <namespace> -- /bin/sh
kubectl get pod <pod> -n <namespace> -o yaml

Node Diagnostics

节点诊断

bash

kubectl describe node <node>
kubectl top nodes
kubectl top pods --all-namespaces
ssh <node> "systemctl status kubelet"
ssh <node> "journalctl -u kubelet -n 100"

bash

kubectl describe node <node>
kubectl top nodes
kubectl top pods --all-namespaces
ssh <node> "systemctl status kubelet"
ssh <node> "journalctl -u kubelet -n 100"

Service & Network

服务与网络

bash

kubectl describe svc <service> -n <namespace>
kubectl get endpoints <service> -n <namespace>
kubectl get networkpolicies --all-namespaces

bash

kubectl describe svc <service> -n <namespace>
kubectl get endpoints <service> -n <namespace>
kubectl get networkpolicies --all-namespaces

Storage

存储

bash

kubectl get pvc,pv --all-namespaces
kubectl describe pvc <pvc> -n <namespace>
kubectl get storageclass

bash

kubectl get pvc,pv --all-namespaces
kubectl describe pvc <pvc> -n <namespace>
kubectl get storageclass

Resource & Configuration

资源与配置

bash

kubectl describe resourcequota -n <namespace>
kubectl describe limitrange -n <namespace>
kubectl get rolebindings,clusterrolebindings -n <namespace>

bash

kubectl describe resourcequota -n <namespace>
kubectl describe limitrange -n <namespace>
kubectl get rolebindings,clusterrolebindings -n <namespace>

Diagnostic Scripts

诊断脚本

check_namespace.py

Namespace-level health check and diagnostics:

Pod health (running, pending, failed, crashlooping, image pull errors)
Service health and endpoints
Deployment availability status
PersistentVolumeClaim status
Resource quota usage and limits
Recent namespace events
Health status assessment
Actionable recommendations

Usage:

bash

undefined

命名空间级健康检查与诊断：

Pod健康状况（运行中、Pending、Failed、CrashLoopBackOff、镜像拉取错误）
服务健康状况与端点
部署可用性状态
PersistentVolumeClaim状态
资源配额使用情况与限制
近期命名空间事件
健康状态评估
可执行的建议

使用方法：

bash

undefined

Human-readable output

人类可读格式输出

python3 scripts/check_namespace.py <namespace>

JSON output for automation

JSON格式输出用于自动化

python3 scripts/check_namespace.py <namespace> --json

Include more events

显示更多事件

python3 scripts/check_namespace.py <namespace> --events 20


Best used when troubleshooting issues in a specific namespace or assessing overall namespace health.

python3 scripts/check_namespace.py <namespace> --events 20


适用于排查特定命名空间的问题或评估命名空间整体健康状况。

Cluster-Level Diagnostics (kubectl)

集群级诊断（kubectl）

For cluster-wide health checks, use kubectl directly:

bash

undefined

针对集群范围的健康检查，直接使用kubectl：

bash

undefined

Node health and status

节点健康与状态

kubectl get nodes kubectl top nodes

Find non-running pods across all namespaces

查找所有命名空间中未运行的pod

kubectl get pods -A --field-selector status.phase!=Running

System pod health

系统pod健康状况

kubectl get pods -n kube-system

undefined

kubectl get pods -n kube-system

undefined

Pod-Level Diagnostics (kubectl)

Pod级诊断（kubectl）

For detailed pod investigation, use kubectl directly:

bash

undefined

针对Pod的详细调查，直接使用kubectl：

bash

undefined

Full pod details (status, events, conditions, resource config)

获取完整的Pod详情（状态、事件、条件、资源配置）

kubectl describe pod <pod-name> -n <namespace>

Current and previous container logs

当前及之前的容器日志

kubectl logs <pod-name> -n <namespace> kubectl logs <pod-name> -n <namespace> --previous

Events specific to the pod

获取与该Pod相关的特定事件

kubectl get events -n <namespace> --field-selector involvedObject.name=<pod-name>

undefined

kubectl get events -n <namespace> --field-selector involvedObject.name=<pod-name>

undefined

Reference Documentation

参考文档

references/common_issues.md

Comprehensive guide to common Kubernetes issues with:

Detailed symptom descriptions
Root cause analysis
Step-by-step diagnostic procedures
Remediation instructions
Prevention strategies

Covers:

Pod issues (ImagePullBackOff, CrashLoopBackOff, Pending, OOMKilled)
Node issues (NotReady, DiskPressure)
Networking issues (pod-to-pod communication, service access)
Storage issues (PVC pending, volume mount failures)
Resource issues (quota exceeded, CPU throttling)
Security issues (vulnerabilities, RBAC)

Read this when you identify a specific issue type but need detailed remediation steps.

常见Kubernetes问题的综合指南，包含：

详细的症状描述
根本原因分析
分步诊断流程
修复说明
预防策略

涵盖：

Pod问题（ImagePullBackOff、CrashLoopBackOff、Pending、OOMKilled）
节点问题（NotReady、DiskPressure）
网络问题（Pod间通信、服务访问）
存储问题（PVC Pending、卷挂载失败）
资源问题（配额超限、CPU限流）
安全问题（漏洞、RBAC）

当你识别到特定问题类型但需要详细修复步骤时，可阅读此文档。

references/incident_response.md

Structured incident response framework including:

Incident response phases (Detection → Triage → Investigation → Resolution → Post-Incident)
Severity level definitions
Detailed playbooks for common incident scenarios
Communication guidelines
Post-incident review template
Best practices for prevention, preparedness, response, and recovery

Read this when responding to production incidents or planning incident response procedures.

结构化事件响应框架，包含：

事件响应阶段（检测 → 分类 → 调查 → 解决 → 事后复盘）
严重级别定义
常见事件场景的详细执行手册
沟通指南
事后复盘模板
预防、准备、响应与恢复的最佳实践

当响应生产事件或规划事件响应流程时，可阅读此文档。

references/performance_troubleshooting.md

Comprehensive performance diagnosis and optimization guide covering:

High Latency Issues - API response time, request latency troubleshooting
CPU Performance - Throttling detection, profiling, optimization
Memory Performance - OOM issues, leak detection, heap profiling
Network Performance - Latency, packet loss, DNS resolution
Storage I/O Performance - Disk performance testing, optimization
Application-Level Metrics - Prometheus integration, distributed tracing
Cluster-Wide Performance - Control plane, scheduler, resource utilization

Read this when:

Investigating slow application response times
Diagnosing CPU or memory performance issues
Troubleshooting network latency or connectivity
Optimizing storage I/O performance
Setting up performance monitoring

综合性能诊断与优化指南，涵盖：

高延迟问题 - API响应时间、请求延迟排查
CPU性能 - 限流检测、性能分析、优化
内存性能 - OOM问题、内存泄漏检测、堆分析
网络性能 - 延迟、丢包、DNS解析
存储I/O性能 - 磁盘性能测试、优化
应用级指标 - Prometheus集成、分布式追踪
集群级性能 - 控制平面、调度器、资源利用率

当出现以下情况时阅读此文档：

调查应用响应缓慢问题
诊断CPU或内存性能问题
排查网络延迟或连通性问题
优化存储I/O性能
设置性能监控

references/helm_troubleshooting.md

Complete guide to Helm troubleshooting including:

Release Issues - Stuck releases, missing resources, state problems
Installation Failures - Chart conflicts, validation errors, template rendering
Upgrade and Rollback - Failed upgrades, immutable field errors, rollback procedures
Values and Configuration - Values not applied, parsing errors, secret handling
Chart Dependencies - Dependency updates, version conflicts, subchart values
Hooks and Lifecycle - Hook failures, cleanup issues
Repository Issues - Chart access problems, version mismatches

Read this when:

Working with Helm-deployed applications
Troubleshooting chart installations or upgrades
Debugging Helm release states
Managing chart dependencies

完整的Helm故障排查指南，包含：

发布问题 - 卡住的发布、缺失的资源、状态问题
安装失败 - 图表冲突、验证错误、模板渲染问题
升级与回滚 - 升级失败、不可变字段错误、回滚流程
值与配置 - 值未生效、解析错误、密钥处理
图表依赖 - 依赖更新、版本冲突、子图表值
钩子与生命周期 - 钩子失败、清理问题
仓库问题 - 图表访问问题、版本不匹配

当出现以下情况时阅读此文档：

处理Helm部署的应用
排查图表安装或升级问题
调试Helm发布状态
管理图表依赖