Back to Details

gke-workload-scaling

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

GKE Workload Scaling

GKE工作负载扩缩容

This skill provides workflows and best practices for scaling applications on Google Kubernetes Engine (GKE). It covers manual scaling, Horizontal Pod Autoscaling (HPA), and Vertical Pod Autoscaling (VPA).
本技能提供了在Google Kubernetes Engine (GKE)上进行应用扩缩容的工作流和最佳实践，涵盖手动扩缩容、Horizontal Pod Autoscaling (HPA)和Vertical Pod Autoscaler (VPA)。

Workflows

工作流

1. Manual Scaling

1. 手动扩缩容

Scale a deployment to a fixed number of replicas. Useful for immediate manual intervention or testing.
Command:
bash
kubectl scale deployment {deployment_name} --replicas={number} -n {namespace}
将部署扩缩容至固定数量的副本。适用于即时手动干预或测试场景。
命令：
bash
kubectl scale deployment {deployment_name} --replicas={number} -n {namespace}

Verify the scale event

Verify the scale event

kubectl get deployment {deployment_name} -n {namespace}
undefined
kubectl get deployment {deployment_name} -n {namespace}
undefined

2. Horizontal Pod Autoscaling (HPA)

2. Horizontal Pod Autoscaler (HPA)

Automatically scale the number of pods based on observed CPU utilization, memory utilization, or custom metrics.
Prerequisites:
  • Metrics Server must be running (enabled by default on GKE).
  • Containers clearly define resource requests/limits.
Quick Command:
bash
kubectl autoscale deployment {deployment_name} --cpu-percent=50 --min=1 --max=10
Manifest Approach (Recommended): Use a YAML manifest for version-controlled configuration. See assets/hpa-example.yaml for a template.
bash
kubectl apply -f assets/hpa-example.yaml
基于观测到的CPU使用率、内存使用率或自定义指标自动调整Pod数量。
前置条件：
  • 必须运行Metrics Server（GKE默认已启用）。
  • 容器需明确定义资源请求/限制。
快速命令：
bash
kubectl autoscale deployment {deployment_name} --cpu-percent=50 --min=1 --max=10
清单方式（推荐）： 使用YAML清单实现版本控制的配置。模板可参考assets/hpa-example.yaml。
bash
kubectl apply -f assets/hpa-example.yaml

Verify HPA is created and fetching metrics

Verify HPA is created and fetching metrics

kubectl get hpa

**Custom Metrics & External Metrics:** For GKE, the modern and recommended
approach for scaling based on Cloud Monitoring metrics (e.g., Pub/Sub queue
length) is to use the **External** metric type, which is natively supported by
the GKE control plane without requiring the Custom Metrics Adapter. For
application-specific metrics exposed via Prometheus, you can use **Google Cloud
Managed Service for Prometheus** or the Prometheus Adapter.
kubectl get hpa

**自定义指标与外部指标：** 在GKE中，基于Cloud Monitoring指标（如Pub/Sub队列长度）进行扩缩容的现代推荐方式是使用**External**指标类型，GKE控制平面原生支持该类型，无需自定义指标适配器。对于通过Prometheus暴露的应用特定指标，可使用**Google Cloud Managed Service for Prometheus**或Prometheus适配器。

3. Vertical Pod Autoscaling (VPA)

3. Vertical Pod Autoscaler (VPA)

Automatically adjust the CPU and memory reservations for your pods to match actual usage. This is critical for right-sizing workloads.
Prerequisites:
  • VPA must be enabled on the cluster.
    • Autopilot: Enabled by default.
    • Standard: Must be enabled manually.
Enable VPA on Standard Cluster:
bash
gcloud container clusters update {cluster_name} --enable-vertical-pod-autoscaling --zone {zone}
Update Modes:
  • Off
    : Calculates recommendations but does not apply them. Good for "dry run" analysis.
  • Initial
    : Assigns resources only at pod creation time.
  • Auto
    : Updates running pods by restarting them if recommendations differ significantly from requests.
  • InPlaceOrRecreate
    : Attempts to update Pod resources without recreating the Pod. If in-place update is not possible, it reverts to 
    Auto
    mode (requires GKE 1.34+).
Example: See assets/vpa-example.yaml for a configuration template.
自动调整Pod的CPU和内存预留，以匹配实际使用情况。这对于合理调整工作负载规模至关重要。
前置条件：
  • 集群必须启用VPA。
    • Autopilot模式： 默认已启用。
    • Standard模式： 需手动启用。
在Standard集群上启用VPA：
bash
gcloud container clusters update {cluster_name} --enable-vertical-pod-autoscaling --zone {zone}
更新模式：
  • Off
    ：计算推荐配置但不应用。适用于"试运行"分析。
  • Initial
    ：仅在Pod创建时分配资源。
  • Auto
    ：若推荐配置与请求差异显著，则通过重启Pod来更新运行中的Pod。
  • InPlaceOrRecreate
    ：尝试在不重建Pod的情况下更新Pod资源。若无法进行原地更新，则回退到
    Auto
    模式（需要GKE 1.34+版本）。
示例： 配置模板可参考assets/vpa-example.yaml。

Best Practices

最佳实践

  1. Define Resource Requests: HPA and VPA rely on accurate resource requests. Always define them in your container specs.
  2. Avoid Metric Conflicts: Do not configure HPA and VPA to use the same metric (e.g., both CPU). This causes thrashing.
    • Typical Pattern: HPA on CPU, VPA on Memory.
  3. Pod Disruption Budgets (PDBs): Define PDBs to ensure application availability during scaling events or node upgrades.
  4. HPA Lag: HPA has a stabilization window (default 5 mins) to prevent rapid fluctuation.
  5. VPA "Auto" Mode Risks: In "Auto" mode, VPA restarts pods to change resources. Ensure your application handles restarts gracefully (e.g., handles SIGTERM).
    • Note: By default, VPA requires at least 2 replicas to perform evictions (to prevent a situation where the only running replica is evicted, causing downtime). In GKE 1.22+, you can override this by setting 
      minReplicas
      in 
      PodUpdatePolicy
      .
  1. 定义资源请求： HPA和VPA依赖准确的资源请求。务必在容器规格中定义资源请求。
  2. 避免指标冲突： 不要同时配置HPA和VPA使用同一指标（如均使用CPU），这会导致频繁波动。
    • 典型模式： HPA基于CPU，VPA基于内存。
  3. Pod中断预算（PDBs）： 定义PDB以确保扩缩容事件或节点升级期间的应用可用性。
  4. HPA延迟： HPA设有稳定窗口（默认5分钟）以防止快速波动。
  5. VPA "Auto"模式风险： 在"Auto"模式下，VPA会重启Pod以更改资源配置。确保应用能优雅处理重启（如正确处理SIGTERM信号）。
    • 注意： 默认情况下，VPA需要至少2个副本才能执行驱逐操作（以避免仅有的运行副本被驱逐导致停机）。在GKE 1.22+版本中，可通过在
      PodUpdatePolicy
      中设置
      minReplicas
      来覆盖此限制。

Rightsizing Workflow

规模合理调整工作流

  1. Deploy VPA in 
    Off
    mode for 24+ hours
  2. Read recommendations: 
    kubectl describe vpa {deployment_name}-vpa -n {namespace}
  3. Compare 
    target
    values against current 
    requests
  4. Apply with 20% buffer: 
    new_request = target * 1.2
  5. Use patch format or update deployment manifest to apply new resource requests
ConditionRecommendationRisk
CPU request >5x P95 actualReduce to 
P95 * 1.2
Medium
Memory request >3x P95 actualReduce to 
P95 * 1.2
Medium
CPU request >2x P95 actualRightsizing with 20% bufferLow
No resource limits setAdd limits to prevent noisy-neighborLow
  1. 以
    Off
    模式部署VPA并运行24小时以上
  2. 查看推荐配置：
    kubectl describe vpa {deployment_name}-vpa -n {namespace}
  3. 对比
    target
    值与当前
    requests
  4. 增加20%缓冲后应用：
    new_request = target * 1.2
  5. 使用补丁格式或更新部署清单来应用新的资源请求
条件建议风险级别
CPU请求 > 实际P95值的5倍降低至
P95 * 1.2
中等
内存请求 > 实际P95值的3倍降低至
P95 * 1.2
中等
CPU请求 > 实际P95值的2倍增加20%缓冲后调整规模低
未设置资源限制添加限制以防止"嘈杂邻居"问题低