gke-reliability

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

GKE Reliability

GKE 可靠性

This reference covers high availability and reliability configuration for GKE clusters and workloads.

MCP Tools:

get_cluster

get_k8s_resource

describe_k8s_resource

apply_k8s_manifest

list_k8s_events

本参考文档涵盖GKE集群和工作负载的高可用性与可靠性配置。

MCP工具：

get_cluster

get_k8s_resource

describe_k8s_resource

apply_k8s_manifest

list_k8s_events

Golden Path Reliability Defaults

黄金路径可靠性默认配置

Setting	Golden Path Value	Notes
Cluster type	Regional (4 zones:	Control plane replicated across
: : us-central1-a/b/c/f) : zones :
Upgrade strategy	SURGE ( `maxSurge: 1` )	Rolling upgrades with extra
: : : capacity :
Auto-repair	`true`	Unhealthy nodes replaced
: : : automatically :
Auto-upgrade	`true`	Nodes follow control plane
: : : version :
Release channel	REGULAR	Balanced freshness and stability
Stateful HA	Enabled	Leader election for stateful
: : : workloads :

设置项	黄金路径配置值	说明
集群类型	Regional（4个可用区：	控制平面跨可用区复制
: : us-central1-a/b/c/f) : :
升级策略	SURGE（ `maxSurge: 1` ）	带额外容量的滚动升级
: : : :
自动修复	`true`	自动替换不健康节点
: : : :
自动升级	`true`	节点跟随控制平面版本更新
: : : :
发布通道	REGULAR	平衡新鲜度与稳定性
有状态工作负载高可用	Enabled	为有状态工作负载启用领导者选举
: : : :

Workflows

工作流程

1. Verify Cluster High Availability

1. 验证集群高可用性

undefined

undefined

MCP (preferred)

MCP（推荐方式）

get_cluster(name="projects/<PROJECT>/locations/<REGION>/clusters/<CLUSTER>", readMask="location,locations,nodePools.locations")

gcloud fallback

gcloud 备选方案

gcloud container clusters describe <CLUSTER> --region <REGION>
--format="json(location, locations)"
--quiet


-   If `location` is a region (e.g., `us-central1`), the control plane is
    regional
-   If `locations` has multiple entries, nodes span multiple zones

gcloud container clusters describe <CLUSTER> --region <REGION>
--format="json(location, locations)"
--quiet


-   若`location`为地区（例如`us-central1`），则控制平面为区域级
-   若`locations`包含多个条目，则节点分布在多个可用区

2. Pod Disruption Budgets (PDBs)

2. Pod中断预算（PDB）

PDBs ensure minimum pod availability during voluntary disruptions (node upgrades, autoscaler scale-down).

Check existing PDBs:

undefined

PDB用于确保在自愿中断（节点升级、自动扩缩容缩容）期间保持最小Pod可用性。

检查现有PDB：

undefined

MCP (preferred)

MCP（推荐方式）

get_k8s_resource(parent="...", resourceType="poddisruptionbudget")

kubectl fallback

kubectl 备选方案

kubectl get pdb --all-namespaces


**Create PDB:**

```yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-app-pdb
  namespace: default
spec:
  minAvailable: 2       # Or use maxUnavailable: 1
  selector:
    matchLabels:
      app: my-app

Every production Deployment with 2+ replicas should have a PDB.

kubectl get pdb --all-namespaces


**创建PDB：**

```yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-app-pdb
  namespace: default
spec:
  minAvailable: 2       # 或使用 maxUnavailable: 1
  selector:
    matchLabels:
      app: my-app

所有包含2个及以上副本的生产环境Deployment都应配置PDB。

3. Health Probes

3. 健康探针

Every production container should have liveness and readiness probes. Startup probes are recommended for slow-starting apps.

Check existing probes:

undefined

每个生产环境容器都应配置存活探针和就绪探针。对于启动缓慢的应用，建议配置启动探针。

检查现有探针：

undefined

MCP (preferred)

MCP（推荐方式）

describe_k8s_resource(parent="...", resourceType="deployment", name="<APP>", namespace="<NS>")

kubectl fallback

kubectl 备选方案

kubectl get deployment <APP> -n <NS> -o yaml | grep -E "livenessProbe|readinessProbe|startupProbe"


**Recommended probe configuration:**

```yaml
spec:
  containers:
  - name: app
    livenessProbe:
      httpGet:
        path: /healthz
        port: 8080
      initialDelaySeconds: 15
      periodSeconds: 10
      timeoutSeconds: 2
      failureThreshold: 3
    readinessProbe:
      httpGet:
        path: /readyz
        port: 8080
      initialDelaySeconds: 5
      periodSeconds: 5
      timeoutSeconds: 2
      failureThreshold: 3
    startupProbe:             # For slow-starting apps
      httpGet:
        path: /healthz
        port: 8080
      initialDelaySeconds: 10
      periodSeconds: 5
      timeoutSeconds: 2
      failureThreshold: 30    # 30 * 5s = 150s max startup time

Readiness: Determines when a pod can accept traffic
Liveness: Determines when to restart a container
Startup: Disables liveness/readiness until the app is ready (prevents premature restarts)

kubectl get deployment <APP> -n <NS> -o yaml | grep -E "livenessProbe|readinessProbe|startupProbe"


**推荐探针配置：**

```yaml
spec:
  containers:
  - name: app
    livenessProbe:
      httpGet:
        path: /healthz
        port: 8080
      initialDelaySeconds: 15
      periodSeconds: 10
      timeoutSeconds: 2
      failureThreshold: 3
    readinessProbe:
      httpGet:
        path: /readyz
        port: 8080
      initialDelaySeconds: 5
      periodSeconds: 5
      timeoutSeconds: 2
      failureThreshold: 3
    startupProbe:             # 适用于启动缓慢的应用
      httpGet:
        path: /healthz
        port: 8080
      initialDelaySeconds: 10
      periodSeconds: 5
      timeoutSeconds: 2
      failureThreshold: 30    # 30 * 5秒 = 最长150秒启动时间

就绪探针：判断Pod何时可以接收流量
存活探针：判断何时需要重启容器
启动探针：在应用准备就绪前禁用存活/就绪探针（防止过早重启）

4. Graceful Shutdown

4. 优雅关闭

Ensure applications handle

SIGTERM

and drain in-flight requests:

yaml

spec:
  terminationGracePeriodSeconds: 30    # Default; increase for long-running requests
  containers:
  - name: app
    lifecycle:
      preStop:
        exec:
          command: ["/bin/sh", "-c", "sleep 5"]  # Allow LB to deregister

确保应用能处理

SIGTERM

信号并完成在处理请求的排空：

yaml

spec:
  terminationGracePeriodSeconds: 30    # 默认值；对于长时请求可适当增加
  containers:
  - name: app
    lifecycle:
      preStop:
        exec:
          command: ["/bin/sh", "-c", "sleep 5"]  # 允许负载均衡器注销节点

5. Topology Spread Constraints

5. 拓扑分布约束

Distribute pods across zones and nodes to survive failures:

yaml

spec:
  topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: my-app
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app: my-app

Zone spread (
```
DoNotSchedule
```
): Hard requirement -- pods must be balanced across zones
Node spread (
```
ScheduleAnyway
```
): Best-effort -- prefer distribution but don't block scheduling

跨可用区和节点分布Pod，以应对故障：

yaml

spec:
  topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: my-app
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app: my-app

可用区分布（
```
DoNotSchedule
```
）：硬性要求——Pod必须在可用区间平衡分布
节点分布（
```
ScheduleAnyway
```
）：尽力而为——优先分布但不阻塞调度

6. Replicas

6. 副本数

Workload Type	Minimum Replicas	Reason
Stateless web/API	2	Survive single pod/node
: : : failure :
Critical services	3	Survive zone failure with zone
: : : spread :
Stateful (databases)	3 (with replication)	Application-level quorum
Batch/jobs	1	Ephemeral by nature

工作负载类型	最小副本数	原因
无状态Web/API服务	2	应对单个Pod/节点故障
: : : :
核心服务	3	结合可用区分布应对可用区故障
: : : :
有状态服务（数据库）	3（带复制机制）	应用层面达成仲裁共识
批处理/任务型工作负载	1	本身为临时性质

Best Practices & Production Guidelines

最佳实践与生产环境指南

Regional clusters for production: Always use regional clusters to survive zone failures.
PDBs for everything: Every production workload with 2+ replicas needs a PodDisruptionBudget (PDB) to protect against voluntary disruptions.
Probes with Explicit Timeouts: Every production container must have both liveness and readiness probes defined. Always explicitly define
initialDelaySeconds
,
periodSeconds
, and
timeoutSeconds
for all probes. Never rely on the Kubernetes default timeout of 1 second if your application requires more, but always set a strict limit to prevent hanging connections.
Zone spreading: Use topology spread constraints to distribute pods across failure domains (zones and nodes).
Graceful shutdown: Handle
```
SIGTERM
```
and set appropriate
```
terminationGracePeriodSeconds
```
with a
```
preStop
```
sleep hook to allow load balancer deregistration.
Maintenance windows: Schedule upgrades during low-traffic periods (see the
```
gke-upgrades
```
skill).

生产环境使用区域集群：始终使用区域集群以应对可用区故障。
所有工作负载配置PDB：每个拥有2个及以上副本的生产环境工作负载都需要Pod中断预算（PDB），以防范自愿中断。
带明确超时的探针：每个生产环境容器必须同时定义存活和就绪探针。务必为所有探针显式定义
initialDelaySeconds
、
periodSeconds
和
timeoutSeconds
。如果应用需要更长时间，切勿依赖Kubernetes默认的1秒超时，但始终要设置严格限制以避免连接挂起。
可用区分布：使用拓扑分布约束将Pod分布在故障域（可用区和节点）中。
优雅关闭：处理
```
SIGTERM
```
信号，设置合适的
```
terminationGracePeriodSeconds
```
并配合
```
preStop
```
睡眠钩子，以允许负载均衡器完成节点注销。
维护窗口：在低流量时段安排升级（参考
```
gke-upgrades
```
技能）。