Back to Details

gke-upgrades

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

GKE Upgrades & Maintenance

GKE 升级与维护

Produce clear, actionable documents — upgrade plans, runbooks, or checklists — tailored to the user's environment. Output should be specific to their cluster mode, release channel, version, and workload types rather than generic advice.
Always frame guidance around the auto-upgrade model: auto-upgrade with maintenance windows and exclusions is the preferred control mechanism.
生成清晰、可执行的文档——升级计划、操作手册或检查清单——并根据用户环境量身定制。输出内容需针对用户的集群模式、发布通道、版本以及工作负载类型，而非通用建议。
始终围绕自动升级模型提供指导：带维护窗口和排除规则的自动升级是首选控制机制。

Context Gathering

上下文收集

Before producing any upgrade artifact, establish:
  • Cluster mode — Standard or Autopilot? (Autopilot has no node pool management, mandatory resource requests, no SSH)
  • Current and target versions — Node version skew must be within 2 minor versions of control plane.
  • Release channel — Rapid, Regular, Stable, or Extended.
  • Environment topology & Rollout Sequencing — Single vs multi-cluster, dev/staging/prod tiers, and whether Rollout Sequencing is used.
  • Workload sensitivity — StatefulSets, databases, GPU, long-running batch need special handling.
If the user provides these upfront, skip straight to the deliverable. If they're vague, fill in reasonable defaults and flag assumptions.
在生成任何升级文档前，需明确以下信息：
  • 集群模式——是Standard还是Autopilot？（Autopilot无节点池管理、强制资源请求、不支持SSH）
  • 当前版本与目标版本——节点版本与控制平面版本的差异不得超过2个小版本。
  • 发布通道——Rapid、Regular、Stable还是Extended。
  • 环境拓扑与发布序列——单集群还是多集群，开发/预发布/生产环境层级，是否使用Rollout Sequencing。
  • 工作负载敏感度——StatefulSets、数据库、GPU、长时间运行的批处理任务需要特殊处理。
如果用户提前提供了这些信息，直接生成交付物即可。如果信息模糊，填充合理默认值并标注假设条件。

Core Principles

核心原则

GKE versions follow Kubernetes version terminology: Major.Minor.Patch (e.g., 1.30.1-gke.1187000). A Minor version bump (e.g., 1.29 → 1.30) introduces new features and APIs. A Patch version bump (e.g., 1.30.1 → 1.30.2) introduces security and bug fixes. Ensure the user understands this distinction.
  1. Sequential control plane, skip-level node pools -- Control plane upgrades are sequential (N → N+1 → N+2). Node pools support skip-level (N+2) upgrades.
  2. Control plane first -- Control plane must be upgraded before node pools. Nodes can trail by up to 2 minor versions.
  3. Environment progression -- Always upgrade dev/staging before production. Use Rollout Sequencing (preferred) to automate and enforce this progression across environments (e.g., dev → staging → prod), or manually coordinate version progression if Rollout Sequencing is not used.
  4. Workload-aware -- Upgrade strategy depends on what's running (stateless, stateful, GPU, batch).
  5. Release channels first -- Always recommend release channels. Note that "No channel" (static versioning) is deprecated and clusters should be migrated to release channels.
  6. Rollback/Downgrade -- Control Plane patches and Node Pools (minor and patches) can be rolled back (downgraded to a target version). GKE supports a 2-step Control Plane minor upgrade where step 1 is rollbackable. Other Control Plane minor version rollbacks are NOT customer-doable and require GKE Support.
  7. Node pool upgrade ordering -- When upgrading multiple node pools, always recommend sequential ordering: upgrade non-critical/stateless pools first (acting as a canary) to verify cluster health before upgrading critical stateful (database) or GPU pools.
GKE版本遵循Kubernetes版本命名规则：主版本.次版本.补丁版本（例如：1.30.1-gke.1187000）。次版本升级（例如：1.29 → 1.30）会引入新功能和API。补丁版本升级（例如：1.30.1 → 1.30.2）会引入安全修复和Bug修复。需确保用户理解这一区别。
  1. 控制平面顺序升级，节点池跨版本升级——控制平面升级需按顺序进行（N → N+1 → N+2）。节点池支持跨版本（N+2）升级。
  2. 先升级控制平面——控制平面必须在节点池之前完成升级。节点版本最多可滞后控制平面2个小版本。
  3. 按环境梯度升级——始终先升级开发/预发布环境，再升级生产环境。优先使用Rollout Sequencing自动化并强制环境间的升级顺序（例如：开发 → 预发布 → 生产）；若未使用Rollout Sequencing，则手动协调版本升级顺序。
  4. 适配工作负载——升级策略取决于运行的工作负载类型（无状态、有状态、GPU、批处理）。
  5. 优先使用发布通道——始终推荐使用发布通道。注意“无通道”（静态版本控制）已被弃用，集群应迁移至发布通道。
  6. 回滚/降级——控制平面补丁版本和节点池（次版本与补丁版本）可回滚（降级至目标版本）。GKE支持控制平面次版本的两步升级，第一步可回滚。其他控制平面次版本回滚无法由用户自行操作，需联系GKE支持团队。
  7. 节点池升级顺序——升级多个节点池时，始终建议按顺序进行：先升级非关键/无状态节点池（作为金丝雀验证），确认集群健康后再升级关键有状态（数据库）或GPU节点池。

Release Channels

发布通道

ChannelBest forSLA
RapidDev/test, early feature accessNo upgrade stability SLA
Regular (default)Most productionFull SLA
StableMission-critical, stability-firstFull SLA
ExtendedCompliance, EoS enforcement controlFull SLA
通道类型适用场景SLA
Rapid开发/测试环境、提前体验新功能无升级稳定性SLA
Regular（默认）大多数生产环境完整SLA
Stable关键业务场景、优先保障稳定性完整SLA
Extended合规场景、需控制终止支持时间完整SLA

Support Lifecycle

支持生命周期

Standard GKE versions are supported for 14 months after they become available in the Regular channel. This means:
  • Rapid channel versions may be supported for longer than 14 months (since they enter Rapid before Regular).
  • Stable channel versions may be supported for less than 14 months (since they enter Stable after Regular).
  • Extended support extends this period up to 24 months. Note that extra cost applies only during the extended support period (months 15-24).
标准GKE版本在Regular通道发布后，支持周期为14个月。这意味着：
  • Rapid通道版本的支持周期可能超过14个月（因为它们先于Regular通道发布）。
  • Stable通道版本的支持周期可能不足14个月（因为它们晚于Regular通道发布）。
  • Extended支持将周期延长至24个月。注意仅在扩展支持期（第15-24个月）会产生额外费用。

Maintenance Windows & Exclusions

维护窗口与排除规则

Configure maintenance windows to control auto-upgrade timing. GKE also supports node pool level maintenance exclusions (in addition to cluster level) to block upgrades for specific workloads.
Exclusion types & Limits:
  • "No upgrades" (Scope: 
    no_upgrades
    )    : Blocks all upgrades (minor, patch, node).
    • Limit: Max 90 days of total exclusion duration in any rolling 365-day window.
    • Chaining constraint: Because of the rolling 365-day limit, you cannot chain multiple exclusions to cover a continuous period longer than 90 days (e.g., you cannot cover a 100-day freeze using 
      no_upgrades
      ).
  • "No minor or node upgrades" (Scope: 
    no_minor_or_node_upgrades
    )    : Blocks minor and node upgrades, but allows control plane patch upgrades (low risk).
    • Limit: Up to 180 days per exclusion. Can be extended (by adding new exclusions) up to the minor version's End of Support (EoS).
  • "No minor upgrades" (Scope: 
    no_minor_upgrades
    )    : Blocks minor upgrades, but allows control plane patches and node upgrades.
    • Limit: Up to 180 days per exclusion. Can be extended up to EoS.
Important Exclusion Rules (MUST follow when recommending exclusions and MUST include in the final text response):
  1. Auto-upgrades only: Maintenance exclusions only block automatic upgrades. Manual upgrades initiated by the user will bypass exclusions. You MUST explain this to the user.
  2. Warn against "No channel": You MUST explicitly warn that disabling release channels ("No channel" / static versioning) is deprecated and must not be used as a replacement for exclusions.
  3. Compare Scopes: You MUST explain the difference between 'No upgrades' (limitations, blocks patches) and 'No minor or node upgrades' (allows patches, longer duration). Recommend 'No minor or node upgrades' when the user wants to allow security patches/fixes while blocking minor version jumps.
  4. Handle periods > 90 days: If the user needs to block upgrades for more than 90 days, you MUST explain that 'No upgrades' is limited to 90 days in a rolling 365-day window (preventing chaining for longer continuous periods) and advise using 'No minor or node upgrades' (which can last up to 180 days per exclusion, extendable until EoS) or persistent exclusions for minor upgrades until End of Support.
  5. Version skew: Be mindful of version skew (between control plane and node pools) when using exclusions. Ensure skew does not exceed the supported 2 minor versions. Use 
    --add-maintenance-exclusion-until-end-of-support
    for persistent exclusions.
  6. Correct gcloud syntax: When providing 
    gcloud
    commands for exclusions, you MUST use the separate flag syntax: 
    --add-maintenance-exclusion-name
    , 
    --add-maintenance-exclusion-start
    , 
    --add-maintenance-exclusion-end
    (or 
    --add-maintenance-exclusion-until-end-of-support
    ), and 
    --add-maintenance-exclusion-scope
    (do NOT use a single comma-separated 
    --add-maintenance-exclusion
    flag).
配置维护窗口以控制自动升级的时间。GKE还支持节点池级别的维护排除规则（除集群级别外），可针对特定工作负载阻止升级。
排除类型与限制：
  • “禁止所有升级”（范围：
    no_upgrades
    ）    ：阻止所有升级（次版本、补丁版本、节点升级）。
    • 限制：在任意滚动365天窗口内，总排除时长最多为90天。
    • 链式约束：由于滚动365天限制，无法通过多个排除规则链式覆盖超过90天的连续周期（例如，无法用
      no_upgrades
      覆盖100天的冻结期）。
  • “禁止次版本或节点升级”（范围：
    no_minor_or_node_upgrades
    ）    ：阻止次版本和节点升级，但允许控制平面补丁版本升级（低风险）。
    • 限制：每次排除最长可达180天。可通过添加新的排除规则延长至次版本的**终止支持（EoS）**时间。
  • “禁止次版本升级”（范围：
    no_minor_upgrades
    ）    ：阻止次版本升级，但允许控制平面补丁版本和节点升级。
    • 限制：每次排除最长可达180天。可延长至EoS时间。
重要排除规则（推荐排除规则时必须遵循，且必须包含在最终文本响应中）：
  1. 仅适用于自动升级：维护排除规则仅阻止自动升级。用户发起的手动升级会绕过排除规则。必须向用户说明这一点。
  2. 警告“无通道”模式：必须明确警告禁用发布通道（“无通道”/静态版本控制）已被弃用，不得作为排除规则的替代方案。
  3. 对比范围差异：必须解释“禁止所有升级”（有限制、阻止补丁）与“禁止次版本或节点升级”（允许补丁、时长更长）之间的区别。当用户希望允许安全补丁/修复但阻止次版本升级时，推荐使用“禁止次版本或节点升级”。
  4. 处理超过90天的场景：如果用户需要阻止升级超过90天，必须说明“禁止所有升级”在滚动365天窗口内限制为90天（无法链式覆盖更长连续周期），并建议使用“禁止次版本或节点升级”（每次排除最长180天，可延长至EoS）或针对次版本的持久排除规则直至终止支持。
  5. 版本差异：使用排除规则时需注意版本差异（控制平面与节点池之间），确保差异不超过支持的2个小版本。使用
    --add-maintenance-exclusion-until-end-of-support
    设置持久排除规则。
  6. 正确的gcloud语法：提供排除规则的
    gcloud
    命令时，必须使用分离的标志语法：
    --add-maintenance-exclusion-name
    、
    --add-maintenance-exclusion-start
    、
    --add-maintenance-exclusion-end
    （或
    --add-maintenance-exclusion-until-end-of-support
    ）以及
    --add-maintenance-exclusion-scope
    （不得使用单个逗号分隔的
    --add-maintenance-exclusion
    标志）。

Mandatory Upgrade Overrides

强制升级覆盖

GKE reserves the right to override user-defined maintenance windows and exclusions for mandatory operations. These overrides cannot be disabled or blocked.
Common Override Scenarios:
  • Critical Security Patches: Urgent vulnerability fixes that must be applied immediately to protect infrastructure.
  • End of Support (EoS) / End of Life (EOL) Enforcement: If a cluster is running an unsupported version, GKE will force upgrade it to a supported version.
  • Expiring Certificates: If control plane certificates (CAs) are expiring (within 30 days) and rotation is required to prevent cluster unrecoverability.
  • Maintenance Starvation: GKE requires at least 48 hours of maintenance availability in any rolling 32-day window. If exclusions block too much, GKE may force an upgrade.
Guidance (MUST follow when overrides are discussed):
  1. Correlate with Bulletins: If GKE performs an unexpected upgrade, you MUST explicitly suggest checking GKE Release Notes or Security Bulletins to correlate the event with emergency patches (do not just suggest checking Cloud Audit Logs).
  2. Design for Resilience: Workloads must be designed to survive unexpected control plane or node rotation. You MUST recommend:
    • Regional clusters (multi-master) to ensure API availability during control plane upgrades.
    • Multi-zone workload deployments.
    • Replicas > 1 for critical deployments.
    • Properly configured Pod Disruption Budgets (PDBs) that are not overly restrictive.
GKE保留覆盖用户定义的维护窗口和排除规则的权利，用于强制操作。这些覆盖无法禁用或阻止。
常见覆盖场景：
  • 关键安全补丁：必须立即应用的紧急漏洞修复，以保护基础设施。
  • 终止支持（EoS）/生命周期结束（EOL）强制执行：如果集群运行不受支持的版本，GKE将强制升级至受支持版本。
  • 证书过期：如果控制平面证书（CA）即将过期（30天内），需进行轮换以避免集群无法恢复。
  • 维护饥饿：GKE要求在任意滚动32天窗口内至少有48小时的维护可用时间。如果排除规则阻止了过多维护时间，GKE可能强制进行升级。
指导原则（讨论覆盖场景时必须遵循）：
  1. 关联公告：如果GKE执行了意外升级，必须明确建议查看GKE发布说明或安全公告，将事件与紧急补丁关联（不要仅建议查看Cloud Audit Logs）。
  2. 设计弹性架构：工作负载必须设计为能够承受意外的控制平面或节点轮换。必须推荐：
    • 区域集群（多主节点），确保控制平面升级期间API可用性。
    • 多区域工作负载部署。
    • 关键部署的副本数>1。
    • 正确配置Pod Disruption Budgets（PDB），避免过于严格。

Upgrade Planning

升级规划

When asked to plan an upgrade, produce a structured document covering:
  • Version compatibility (breaking changes, deprecated APIs) (minor version upgrades only)
  • Upgrade path (sequential minor version upgrades) (minor version upgrades only)
  • Node pool upgrade strategy (Standard only)
  • Workload readiness (PDBs, resource requests)
  • Rollback/Contingency procedure (how to revert node pools or coordinate with GKE Support for master rollback)
Compatibility Search Rule:
  • If compatibility information (e.g., third-party operator compatibility, GPU driver/CUDA compatibility matrix) is not immediately available in the workspace or via a quick web search, do NOT loop or make multiple search attempts. Instead, list the compatibility verification as a critical pre-upgrade action item for the user in the checklist.
当用户要求规划升级时，生成结构化文档，涵盖：
  • 版本兼容性（破坏性变更、已弃用API）（仅适用于次版本升级）
  • 升级路径（顺序次版本升级）（仅适用于次版本升级）
  • 节点池升级策略（仅适用于Standard集群）
  • 工作负载就绪性（PDB、资源请求）
  • 回滚/应急流程（如何回滚节点池或联系GKE支持团队进行控制平面回滚）
兼容性搜索规则：
  • 如果兼容性信息（例如，第三方Operator兼容性、GPU驱动/CUDA兼容性矩阵）无法在工作区或快速网络搜索中立即获取，不要循环或多次尝试搜索。相反，将兼容性验证列为用户的关键升级前操作项，加入检查清单。

Node Pool Strategy (Standard Only)

节点池策略（仅适用于Standard集群）

Recommend Surge upgrade as the default and most common strategy, with per-pool settings:
  • Stateless: Higher 
    maxSurge
    (2-3) for speed, 
    maxUnavailable=0
    for safety.
  • Stateful/DB: 
    maxSurge=1, maxUnavailable=0
    (conservative).
  • GPU (fixed reservation): 
    maxSurge=0, maxUnavailable=1
    (no surge capacity).
  • Large (50+ nodes): 
    maxSurge=20, maxUnavailable=0
    (max parallelism).
For mission-critical workloads requiring fast rollback or strict validation, recommend Standard Blue-Green upgrades. Acknowledge Autoscaled Blue-Green as an option for disruption-sensitive workloads, but note it is currently in preview and may have capacity requirements.
Upgrade Ordering (User-initiated only): When planning manual upgrades, specify the sequence of node pool upgrades. Recommend upgrading stateless pools first, verifying cluster stability, and then upgrading stateful/GPU pools. For auto-upgrades, GKE automatically manages sequential node pool upgrades.
For standard command sequences and runbook templates, see 
references/runbook-template.md
.
推荐**滚动扩容升级（Surge upgrade）**作为默认且最常用的策略，针对不同节点池设置如下：
  • 无状态负载：较高的
    maxSurge
    （2-3）以提升速度，
    maxUnavailable=0
    保障安全。
  • 有状态/数据库负载：
    maxSurge=1, maxUnavailable=0
    （保守策略）。
  • GPU（固定预留）：
    maxSurge=0, maxUnavailable=1
    （无扩容容量）。
  • 大型节点池（50+节点）：
    maxSurge=20, maxUnavailable=0
    （最大并行度）。
对于需要快速回滚或严格验证的关键业务工作负载，推荐标准蓝绿升级（Standard Blue-Green）。确认**自动扩缩容蓝绿升级（Autoscaled Blue-Green）**是对中断敏感的工作负载的选项，但需注意该功能目前处于预览阶段，可能有容量要求。
升级顺序（仅用户发起的手动升级）：规划手动升级时，指定节点池的升级顺序。建议先升级无状态节点池，验证集群稳定性后再升级有状态/GPU节点池。对于自动升级，GKE会自动管理节点池的顺序升级。
标准命令序列和操作手册模板，请参阅
references/runbook-template.md
。

Large-Scale AI/ML Clusters (GPU/TPU)

大规模AI/ML集群（GPU/TPU）

  • No Live Migration: GPU VMs do not support live migration; GKE upgrades will force pod restarts. You MUST explain this.
  • Fixed Reservations & Quota: H100/A100 typically use fixed reservations with no spare quota.
    • Recommend rolling upgrade with zero surge: 
      maxSurge=0, maxUnavailable=1
      . This releases the reservation of the node being upgraded before provisioning its replacement.
    • You MUST explain that Blue-Green upgrades are not feasible because they require double (2x) the GPU resources (both quota and reservations) during the transition.
  • Driver Coupling: The GPU driver is tightly coupled with the target node OS image version.
    • You MUST explain that node upgrades update the underlying OS image, introducing new Linux Kernels and hardware drivers (NVIDIA).
    • You MUST warn that driver updates can break CUDA compatibility.
    • You MUST recommend comparing OS image, kernel version (
      uname -r
      ), and driver versions between old (working) and new (non-working) nodes to diagnose driver issues.
    • You MUST recommend deploying a test pod (e.g., vector addition) to verify GPU access.
    • You MUST recommend rolling back the node pool to the previous version as a quick mitigation if production is blocked.
    • You MUST advise updating workload dependencies (CUDA version in container images) to match the new driver before attempting the upgrade again.
    • You MUST advise upgrading and testing CUDA compatibility in a staging environment/cluster before applying the upgrade to the production GPU node pools.
  • Operational Safety:
    • You MUST recommend using GKE maintenance exclusions to block auto-upgrades during active training campaigns.
    • Prior to manual upgrades, cordon GPU nodes and wait for active training jobs to checkpoint/complete.
  • TPU Considerations: TPU slices are recreated atomically (not rolling); maintenance on one slice restarts all slices in the environment.
  • 无实时迁移：GPU虚拟机不支持实时迁移；GKE升级会强制重启Pod。必须向用户说明这一点。
  • 固定预留与配额：H100/A100通常使用固定预留，无备用配额。
    • 推荐零扩容滚动升级：
      maxSurge=0, maxUnavailable=1
      。该策略会在配置替换节点前释放待升级节点的预留资源。
    • 必须说明蓝绿升级不可行，因为过渡期间需要双倍（2倍）的GPU资源（配额和预留均需翻倍）。
  • 驱动耦合：GPU驱动与目标节点OS镜像版本紧密耦合。
    • 必须说明节点升级会更新底层OS镜像，引入新的Linux内核和硬件驱动（NVIDIA）。
    • 必须警告驱动更新可能破坏CUDA兼容性。
    • 必须推荐对比新旧节点的OS镜像、内核版本（
      uname -r
      ）和驱动版本，以诊断驱动问题。
    • 必须推荐部署测试Pod（例如向量加法）验证GPU访问。
    • 必须推荐若生产环境受阻，快速回滚节点池至先前版本作为临时缓解方案。
    • 必须建议在再次尝试升级前，更新工作负载依赖（容器镜像中的CUDA版本）以匹配新驱动。
    • 必须建议在预发布环境/集群中升级并测试CUDA兼容性，再将升级应用到生产GPU节点池。
  • 操作安全：
    • 必须推荐使用GKE维护排除规则在活跃训练期间阻止自动升级。
    • 手动升级前，标记GPU节点为不可调度（cordon），等待活跃训练任务完成 checkpoint 或结束。
  • TPU注意事项：TPU切片会被原子重建（非滚动）；对一个切片的维护会重启环境中的所有切片。

Checklists

检查清单

Produce checklists as copyable markdown with checkboxes. See 
references/checklists.md
 for the full pre-upgrade and post-upgrade checklist templates. Adapt them to the user's environment.
Stateful Workloads: When stateful workloads (databases) are present, always include checks for PV backup completion and verification of PV reclaim policies (e.g., Retain vs Delete) in the pre-upgrade checklist.
Autopilot Checklists: For Autopilot clusters, ensure the checklists include:
  • Verification of 
    resources.requests
    on all containers (Autopilot requirement).
  • You MUST include specific 
    kubectl
    commands for API deprecation checks, specifically: 
    kubectl get --raw /metrics | grep apiserver_request_total | grep deprecated
    to check if any active workloads are using deprecated APIs.
  • Verifying PDBs to ensure they don't block node drain (even though GKE manages nodes, PDBs are still respected).
  • Identifying and deleting "bare pods" (pods not managed by a ReplicaSet/Deployment/StatefulSet) as they won't be rescheduled during node recreation.
  • Verification of 
    terminationGracePeriodSeconds
    to ensure pods have enough time to shut down gracefully during node recreation.
生成可复制的Markdown格式检查清单，包含复选框。完整的升级前和升级后检查清单模板，请参阅
references/checklists.md
。根据用户环境调整模板内容。
有状态工作负载：当存在有状态工作负载（数据库）时，升级前检查清单必须包含PV备份完成检查和PV回收策略验证（例如Retain vs Delete）。
Autopilot集群检查清单：针对Autopilot集群，检查清单需包含：
  • 验证所有容器的
    resources.requests
    （Autopilot要求）。
  • 必须包含特定的
    kubectl
    命令用于检查API弃用情况，具体为：
    kubectl get --raw /metrics | grep apiserver_request_total | grep deprecated
    ，以检查是否有活跃工作负载正在使用已弃用的API。
  • 验证PDB，确保它们不会阻止节点驱逐（尽管GKE管理节点，但仍会遵循PDB规则）。
  • 识别并删除“裸Pod”（未由ReplicaSet/Deployment/StatefulSet管理的Pod），因为节点重建时这些Pod不会被重新调度。
  • 验证
    terminationGracePeriodSeconds
    ，确保节点重建时Pod有足够时间优雅关闭。

Maintenance runbooks

维护操作手册

Produce step-by-step runbooks with actual 
gcloud
and 
kubectl
commands. See 
references/runbook-template.md
for the standard command sequences.
生成包含实际
gcloud
和
kubectl
命令的分步操作手册。标准命令序列请参阅
references/runbook-template.md
。

Maintenance Window Pauses

维护窗口暂停

When diagnosing a "stuck" upgrade, consider if it was paused by a maintenance window:
  • Silent Pause Behavior: If a maintenance window closes before an upgrade (auto or manual) completes, GKE intentionally pauses the rollout to prevent disruption outside allowed times.
  • Mixed-Version State: The cluster is left in a stable mixed-version state (some nodes upgraded, some not). You MUST explicitly state that this is a supported and safe intended outcome.
  • Resumption: The upgrade will automatically resume when the next maintenance window opens.
  • Mitigation for immediate completion: If the user wants to complete the upgrade immediately, you MUST suggest temporarily widening the maintenance window to include the current time (e.g., using 
    gcloud container clusters update ... --maintenance-window-start ... --maintenance-window-duration ...
    ). Do not suggest re-triggering the manual upgrade or bypassing the window.
诊断“停滞”升级时，需考虑是否因维护窗口导致升级暂停：
  • 静默暂停行为：如果维护窗口在升级（自动或手动）完成前关闭，GKE会有意暂停滚动升级，以避免在允许时间外造成中断。
  • 混合版本状态：集群会处于稳定的混合版本状态（部分节点已升级，部分未升级）。必须明确说明这是受支持且安全的预期结果。
  • 恢复升级：升级会在下一个维护窗口开启时自动恢复。
  • 立即完成升级的缓解方案：如果用户希望立即完成升级，必须建议临时扩大维护窗口以包含当前时间（例如，使用
    gcloud container clusters update ... --maintenance-window-start ... --maintenance-window-duration ...
    ）。不要建议重新触发手动升级或绕过窗口。

Troubleshooting

故障排除

When a user reports a stuck or failing upgrade, you MUST systematically analyze and address ALL 5 potential causes in your final response. Do not omit checks even if you suspect one is the primary cause:
  1. PDB blocking drain: Identify if any PDB has 
    ALLOWED DISRUPTIONS = 0
    using 
    kubectl get pdb -A
    .
  2. Resource constraints: Check if pods are stuck in 
    Pending
    due to capacity limits.
  3. Bare pods: Identify pods without owner references that are blocking the drain (recommend deleting them).
  4. Admission webhooks: Check if Validating/Mutating webhooks are rejecting pod creation on new nodes.
  5. PVC attachment issues: Check for volume attachment failures (especially zone constraints).
Stockout / Quota Exhaustion Rule:
  • If the upgrade is stuck due to 
    ZONE_RESOURCE_POOL_EXHAUSTED
    (stockout) or 
    QUOTA_EXCEEDED
    for Compute Engine resources:
    1. Recommend modifying the upgrade strategy to 
      maxSurge=0
      (rolling in-place) to bypass quota limits.
    2. For 
      QUOTA_EXCEEDED
      , suggest requesting a quota increase from Google Cloud.
    3. You MUST suggest migrating workloads or creating new node pools in a different zone or region where capacity/quota is available as a mitigation.
Refer to 
references/troubleshooting.md
 for the exact diagnostic commands and fix procedures for each step.
当用户报告升级停滞或失败时，必须系统分析并解决以下所有5个潜在原因，不得遗漏任何检查，即使你怀疑某一个是主要原因：
  1. PDB阻止驱逐：使用
    kubectl get pdb -A
    检查是否有PDB的
    ALLOWED DISRUPTIONS = 0
    。
  2. 资源约束：检查Pod是否因容量限制处于
    Pending
    状态。
  3. 裸Pod：识别无所有者引用的Pod，这些Pod会阻止驱逐（建议删除它们）。
  4. 准入Webhook：检查Validating/Mutating Webhook是否拒绝新节点上的Pod创建。
  5. PVC挂载问题：检查卷挂载失败（尤其是区域约束问题）。
资源耗尽/配额用尽规则：
  • 如果升级因
    ZONE_RESOURCE_POOL_EXHAUSTED
    （资源耗尽）或
    QUOTA_EXCEEDED
    （配额用尽）而停滞：
    1. 建议将升级策略修改为
      maxSurge=0
      （原地滚动升级）以绕过配额限制。
    2. 对于
      QUOTA_EXCEEDED
      ，建议向Google Cloud申请配额提升。
    3. 必须建议将工作负载迁移至容量/配额可用的其他区域或地区，或在该区域创建新节点池作为缓解方案。
详细的诊断命令和修复步骤，请参阅
references/troubleshooting.md
。

References

参考资料