k8s-incident

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Kubernetes Incident Response

Kubernetes事件响应

Runbooks and diagnostic workflows for common Kubernetes incidents.

针对常见Kubernetes事件的运行手册和诊断工作流。

When to Apply

适用场景

Use this skill when:

User mentions: "incident", "outage", "emergency", "down", "not working"
Operations: emergency response, production issues, service degradation
Keywords: "urgent", "broken", "fix", "restore", "recover"

在以下场景使用该技能：

用户提及：“事件”、“服务中断”、“紧急情况”、“宕机”、“无法正常工作”
操作场景：应急响应、生产环境问题、服务性能下降
关键词：“紧急”、“故障”、“修复”、“恢复”、“复原”

Priority Rules

优先级规则

Priority	Rule	Impact	Tools
1	Check control plane first	CRITICAL	`get_pods(namespace="kube-system")`
2	Assess node health	CRITICAL	`get_nodes`
3	Gather events before changes	HIGH	`get_events`
4	Document timeline	HIGH	Manual notes
5	Rollback if safe	MEDIUM	`rollback_deployment`

优先级	规则	影响程度	工具
1	首先检查控制平面	严重	`get_pods(namespace="kube-system")`
2	评估节点健康状态	严重	`get_nodes`
3	执行变更前收集事件日志	高	`get_events`
4	记录事件时间线	高	手动记录
5	若安全则执行回滚	中	`rollback_deployment`

Quick Reference

速查指南

Incident	First Tool	Next Steps
Pod failure	`get_pod_logs(previous=True)`	`describe_pod` , `get_events`
Node down	`describe_node`	Check kubelet logs
Service unreachable	`get_endpoints`	`get_network_policies`
Control plane	`get_pods(namespace="kube-system")`	Check API server logs

事件类型	首选工具	后续步骤
Pod故障	`get_pod_logs(previous=True)`	`describe_pod` , `get_events`
节点宕机	`describe_node`	检查kubelet日志
服务无法访问	`get_endpoints`	`get_network_policies`
控制平面问题	`get_pods(namespace="kube-system")`	检查API Server日志

Incident Triage

事件分诊

Quick Health Check

快速健康检查

python

get_nodes()
get_pods(namespace="kube-system")
get_events(namespace)

python

get_nodes()
get_pods(namespace="kube-system")
get_events(namespace)

Severity Assessment

严重程度评估

Indicator	Severity	Action
Multiple nodes NotReady	Critical	Escalate immediately
kube-system pods failing	Critical	Control plane issue
Single pod CrashLoop	Medium	Debug pod
High latency	Medium	Check resources

指标	严重程度	操作
多个节点处于NotReady状态	严重	立即升级处理
kube-system Pod故障	严重	控制平面问题
单个Pod出现CrashLoop	中	调试Pod
高延迟	中	检查资源使用情况

Runbook: Pod Failures

运行手册：Pod故障

CrashLoopBackOff

python

get_pod_logs(name, namespace, previous=True)
describe_pod(name, namespace)
get_events(namespace, field_selector="involvedObject.name=<pod>")
get_pod_metrics(name, namespace)

Common Causes:

OOMKilled → Increase memory limits
Exit code 1 → Application error in logs
Exit code 137 → Killed by OOM or SIGKILL
Exit code 143 → Graceful SIGTERM

python

get_pod_logs(name, namespace, previous=True)
describe_pod(name, namespace)
get_events(namespace, field_selector="involvedObject.name=<pod>")
get_pod_metrics(name, namespace)

常见原因：

OOMKilled → 提升内存限制
退出码1 → 日志中存在应用程序错误
退出码137 → 因OOM或SIGKILL被终止
退出码143 → 优雅终止（SIGTERM）

ImagePullBackOff

python

describe_pod(name, namespace)
get_secrets(namespace)

python

describe_pod(name, namespace)
get_secrets(namespace)

Pending Pod

处于Pending状态的Pod

python

describe_pod(name, namespace)
get_nodes()
get_events(namespace)

python

describe_pod(name, namespace)
get_nodes()
get_events(namespace)

Runbook: Node Issues

运行手册：节点问题

Node NotReady

节点处于NotReady状态

python

describe_node(name)
get_events(namespace="", field_selector="involvedObject.name=<node>")
node_logs_tool(name, "kubelet")

python

describe_node(name)
get_events(namespace="", field_selector="involvedObject.name=<node>")
node_logs_tool(name, "kubelet")

Node DiskPressure

节点磁盘压力过大

python

describe_node(name)
get_pods(field_selector="spec.nodeName=<node>")

python

describe_node(name)
get_pods(field_selector="spec.nodeName=<node>")

Runbook: Network Issues

运行手册：网络问题

Service Not Accessible

服务无法访问

python

get_services(namespace)
get_endpoints(namespace)
get_pods(namespace, label_selector="<service-selector>")
get_network_policies(namespace)

python

get_services(namespace)
get_endpoints(namespace)
get_pods(namespace, label_selector="<service-selector>")
get_network_policies(namespace)

DNS Resolution Failures

DNS解析失败

python

get_pods(namespace="kube-system", label_selector="k8s-app=kube-dns")
get_pod_logs("coredns-xxx", "kube-system")

python

get_pods(namespace="kube-system", label_selector="k8s-app=kube-dns")
get_pod_logs("coredns-xxx", "kube-system")

With Cilium

使用Cilium时

python

cilium_status_tool()
cilium_endpoints_list_tool(namespace)
hubble_flows_query_tool(namespace)

python

cilium_status_tool()
cilium_endpoints_list_tool(namespace)
hubble_flows_query_tool(namespace)

With Istio

使用Istio时

python

istio_analyze_tool(namespace)
istio_proxy_status_tool()

python

istio_analyze_tool(namespace)
istio_proxy_status_tool()

Runbook: Storage Issues

运行手册：存储问题

PVC Pending

PVC处于Pending状态

python

describe_pvc(name, namespace)
get_storage_classes()
get_events(namespace)

python

describe_pvc(name, namespace)
get_storage_classes()
get_events(namespace)

Pod Stuck in ContainerCreating

Pod卡在ContainerCreating状态

python

describe_pod(name, namespace)
get_pvc(namespace)
get_events(namespace)

python

describe_pod(name, namespace)
get_pvc(namespace)
get_events(namespace)

Runbook: Control Plane Issues

运行手册：控制平面问题

API Server Unavailable

API Server不可用

python

get_pods(namespace="kube-system", label_selector="component=kube-apiserver")
get_events(namespace="kube-system")

python

get_pods(namespace="kube-system", label_selector="component=kube-apiserver")
get_events(namespace="kube-system")

etcd Issues

etcd问题

python

get_pods(namespace="kube-system", label_selector="component=etcd")
get_pod_logs("etcd-xxx", "kube-system")

python

get_pods(namespace="kube-system", label_selector="component=etcd")
get_pod_logs("etcd-xxx", "kube-system")

Emergency Actions

应急操作

Force Delete Pod

强制删除Pod

python

delete_pod(name, namespace, grace_period=0, force=True)

python

delete_pod(name, namespace, grace_period=0, force=True)

Rollback Deployment

回滚Deployment

python

rollback_deployment(name, namespace, revision=0)

python

rollback_deployment(name, namespace, revision=0)

Helm Rollback

Helm回滚

python

rollback_helm_release(name, namespace, revision=1)

python

rollback_helm_release(name, namespace, revision=1)

Diagnostic Collection Script

诊断收集脚本

For comprehensive incident diagnostics, see scripts/collect-diagnostics.py.

如需全面的事件诊断，请查看 scripts/collect-diagnostics.py。

Multi-Cluster Incident Response

多集群事件响应

Check all clusters:

python

for context in ["prod-1", "prod-2", "staging"]:
    get_nodes(context=context)
    get_pods(namespace="kube-system", context=context)
    get_events(namespace="kube-system", context=context)

检查所有集群：

python

for context in ["prod-1", "prod-2", "staging"]:
    get_nodes(context=context)
    get_pods(namespace="kube-system", context=context)
    get_events(namespace="kube-system", context=context)

k8s-incident

Original

Translation

Kubernetes Incident Response

Kubernetes事件响应

When to Apply

适用场景

Priority Rules

优先级规则

Quick Reference

速查指南

Incident Triage

事件分诊

Quick Health Check

快速健康检查

Severity Assessment

严重程度评估

Runbook: Pod Failures

运行手册：Pod故障

CrashLoopBackOff

CrashLoopBackOff

ImagePullBackOff

ImagePullBackOff

Pending Pod

处于Pending状态的Pod

Runbook: Node Issues

运行手册：节点问题

Node NotReady

节点处于NotReady状态

Node DiskPressure

节点磁盘压力过大

Runbook: Network Issues

运行手册：网络问题

Service Not Accessible

服务无法访问

DNS Resolution Failures

DNS解析失败

With Cilium

使用Cilium时

With Istio

使用Istio时

Runbook: Storage Issues

运行手册：存储问题

PVC Pending

PVC处于Pending状态

Pod Stuck in ContainerCreating

Pod卡在ContainerCreating状态

Runbook: Control Plane Issues

运行手册：控制平面问题

API Server Unavailable

API Server不可用

etcd Issues

etcd问题

Emergency Actions

应急操作

Force Delete Pod

强制删除Pod

Rollback Deployment

回滚Deployment

Helm Rollback

Helm回滚

Diagnostic Collection Script

诊断收集脚本

Multi-Cluster Incident Response

多集群事件响应

Post-Incident

事件后处理

Document Timeline

记录时间线

Prevent Recurrence

预防复发

Related Skills

相关技能