gke-workload-scaling-troubleshooting
Compare original and translation side by side
🇺🇸
Original
English🇨🇳
Translation
ChineseGKE Workload Scaling Troubleshooting Skill
GKE工作负载扩缩容故障排查技能
Use this skill to systematically diagnose and resolve HorizontalPodAutoscaler
(HPA) failures on GKE — metrics reported as , events,
missing resource requests, custom/external metrics-pipeline breakage, HPA that
refuses to scale up or down, scale-to/from-zero issues, and slow HPA reaction on
large clusters. This skill operates non-interactively and enforces a read-only
diagnostics boundary before proposing manifest or configuration corrections.
<unknown>FailedGet*For configuring HPA/VPA objects and scaling best practices, use theskill instead. This skill focuses on failure diagnosis.gke-workload-scaling
使用本技能可系统诊断和解决GKE上的HorizontalPodAutoscaler (HPA)故障——包括指标上报为、类事件、资源请求缺失、自定义/外部指标管道中断、HPA无法扩缩容、零副本扩缩容问题,以及大型集群中HPA响应缓慢的问题。本技能以非交互方式运行,在提出清单或配置修正方案前,严格遵循只读诊断边界。
<unknown>FailedGet*如需配置HPA/VPA对象以及了解扩缩容最佳实践,请改用技能。本技能专注于故障诊断。gke-workload-scaling
🔍 Diagnosis & Resolution Workflow
🔍 诊断与解决流程
Step 0: Non-Interactive Context Discovery & Dry-Run Fallback
步骤0:非交互上下文发现与空运行回退机制
-
Parameter Extraction: Extract required context (,
project_id,cluster_name,cluster_location,hpa_name,workload_name) non-interactively from the user prompt, activeworkload_namespace, or environment defaults:SETTINGS.md- Default to
workload_namespaceif omitted.default - Infer missing cluster parameters from the active environment (or
kubectl config current-context).gcloud config get-value project
- Default
-
Cluster Credentials & Fallback Mode:
- Attempt credential fetch: .
gcloud container clusters get-credentials {cluster_name} --location {cluster_location} --project {project_id} - Fallback / Dry-Run Mode: If the cluster is unreachable,
non-existent, or live command execution fails (such as in sandboxed
evaluations, dry-run mode, or offline analysis):
- Limit retry attempts to avoid resource exhaustion and context overflow.
- Immediately present the exact /
kubectldiagnostic commands for the human operator to run.gcloud - Synthesize the root-cause analysis and output the proposed GitOps correction based on the reported symptoms.
- Attempt credential fetch:
-
参数提取:从用户提示、当前或环境默认值中非交互地提取所需上下文(
SETTINGS.md、project_id、cluster_name、cluster_location、hpa_name、workload_name):workload_namespace- 若未指定,默认值为
workload_namespace。default - 从当前环境推断缺失的集群参数(或
kubectl config current-context)。gcloud config get-value project
- 若未指定
-
集群凭证与回退模式:
- 尝试获取凭证:。
gcloud container clusters get-credentials {cluster_name} --location {cluster_location} --project {project_id} - 回退/空运行模式:如果集群不可达、不存在,或者实时命令执行失败(例如在沙箱评估、空运行模式或离线分析场景下):
- 限制重试次数,避免资源耗尽和上下文溢出。
- 立即提供准确的/
kubectl诊断命令供运维人员执行。gcloud - 基于上报的症状综合分析根本原因,并输出建议的GitOps修正方案。
- 尝试获取凭证:
Step 1: Inspect the HPA and Classify the Symptom
步骤1:检查HPA并对症状分类
Start every investigation with , then route to the matching
branch. The three key sections are (an value means the
HPA hasn't fetched the metric or the pipeline is broken),
(, , — a status marks a
failure), and (specific reasons such as or
).
kubectl describe hpaMetrics<unknown>ConditionsAbleToScaleScalingActiveScalingLimitedFalseEventsFailedGetScaleFailedGetResourceMetricDiagnostic Commands:
bash
kubectl describe hpa {hpa_name} -n {workload_namespace}
kubectl get hpa {hpa_name} -n {workload_namespace} -o yamlFor historical events, query Cloud Logging (the HPA events survive after the live
list rolls over):
Eventsresource.type="k8s_cluster"
resource.labels.cluster_name="{cluster_name}"
resource.labels.location="{cluster_location}"
logName="projects/{project_id}/logs/events"
jsonPayload.involvedObject.kind="HorizontalPodAutoscaler"Route by signal:
- ,
FailedGetScale,FailedComputeMetricsReplicas, or fluctuating replicas from competing HPAs → Branch A (Configuration Errors).Error 400 ... label is not allowed - ,
FailedGetResourceMetric, orunable to fetch pod metrics→ Branch B (Workload & Service Errors).multiple services selecting the same target - custom/external metric,
<unknown>/FailedGetExternalMetric, orFailedGetCustomMetric→ Branch C (Metrics API & Data Availability).no known available metric versions found - Conditions all / no errors but the workload won't scale up or down → Branch D (Healthy but Unexpected Scaling).
True - Workload configured with won't scale to or from zero → Branch E (Scale To / From Zero).
minReplicas: 0 - Correct HPA but slow reaction on a cluster with many HPA objects → Branch F (Slow Recalculation on Large Clusters).
所有排查都从执行开始,然后根据结果路由到对应的分支。三个关键部分是**(值为表示HPA未获取到指标或指标管道中断)、(、、——状态为表示存在故障)和**(例如或等具体原因)。
kubectl describe hpaMetrics<unknown>ConditionsAbleToScaleScalingActiveScalingLimitedFalseEventsFailedGetScaleFailedGetResourceMetric诊断命令:
bash
kubectl describe hpa {hpa_name} -n {workload_namespace}
kubectl get hpa {hpa_name} -n {workload_namespace} -o yaml如需查询历史事件,请使用Cloud Logging(实时列表滚动过期后,HPA事件仍会保留在Cloud Logging中):
Eventsresource.type="k8s_cluster"
resource.labels.cluster_name="{cluster_name}"
resource.labels.location="{cluster_location}"
logName="projects/{project_id}/logs/events"
jsonPayload.involvedObject.kind="HorizontalPodAutoscaler"按信号路由:
- 、
FailedGetScale、FailedComputeMetricsReplicas,或因HPA竞争导致副本数波动 → 分支A(配置错误)。Error 400 ... label is not allowed - 、
FailedGetResourceMetric或unable to fetch pod metrics→ 分支B(工作负载与服务错误)。multiple services selecting the same target - 自定义/外部指标为、
<unknown>/FailedGetExternalMetric,或FailedGetCustomMetric→ 分支C(指标API与数据可用性)。no known available metric versions found - 所有Conditions均为/无错误,但工作负载无法扩缩容 → 分支D(状态正常但扩缩容行为不符合预期)。
True - 配置了的工作负载无法扩缩容到零副本或从零副本扩容 → 分支E(零副本扩缩容)。
minReplicas: 0 - HPA配置正确,但在包含大量HPA对象的集群中响应缓慢 → 分支F(大型集群重计算缓慢)。
Step 2: Resolution — Route to the Matching Branch
步骤2:解决问题——路由到对应分支
Based on the signal you classified in Step 1, jump to one of the
mutually-exclusive branches below (A–F). These are alternatives — you do not
run them in sequence. After applying the branch's fix, go to Step 3 to present it
as a reviewable GitOps change.
根据你在步骤1中分类的信号,跳转到以下互斥分支中的一个(A-F)。这些分支是替代方案,无需按顺序执行。应用对应分支的修复方案后,前往步骤3,将其以可审核的GitOps变更形式呈现。
Branch A: HorizontalPodAutoscaler Configuration Errors
分支A:HorizontalPodAutoscaler配置错误
-
—
FailedGetScale: theunable to get the target's current scale: ... "TARGET" not founddoesn't resolve to an existing scalable workload.scaleTargetRef- Verify the
scaleTargetRef,name, andkindexactly match the target workload's metadata.apiVersion - Confirm the target workload exists in the same namespace as the HPA
(a missing puts objects in
-n, causing a mismatch).default - The target must be a scalable kind (Deployment, StatefulSet, ReplicaSet) — you cannot autoscale a DaemonSet.
- Verify the
-
—
FailedComputeMetricsReplicas: the metricinvalid metrics (1 invalid out of 1)andtypedon't match.target- If , the target must be
type: Utilization.averageUtilization - If , the target must be
type: AverageValue.averageValue
- If
-
: an invalid key in
unable to fetch metrics from external metrics API: googleapi: Error 400: Metric label: 'LABEL' is not allowed.metric.selector.matchLabels- Remove or correct the disallowed label; find valid filterable labels in the Cloud Monitoring metric documentation.
-
Replica count fluctuates / contradictoryevents from different HPAs: more than one HPA targets the same workload via
SuccessfulRescale, and they compete. There is no dedicated condition for this — confirm withspec.scaleTargetRefand look for duplicatekubectl get hpa -n {workload_namespace} -o yamlvalues.scaleTargetRef- Consolidate all metrics into one HPA object (it takes the highest of
its ) and delete the duplicates.
spec.metrics
- Consolidate all metrics into one HPA object (it takes the highest of
its
-
—
FailedGetScale:unable to get the target's current scale: ... "TARGET" not found无法解析到现有的可扩缩容工作负载。scaleTargetRef- 验证的
scaleTargetRef、name和kind与目标工作负载的元数据完全匹配。apiVersion - 确认目标工作负载与HPA位于同一命名空间(缺少参数会将对象放入
-n命名空间,导致不匹配)。default - 目标必须是可扩缩容的类型(Deployment、StatefulSet、ReplicaSet)——不能对DaemonSet进行自动扩缩容。
- 验证
-
—
FailedComputeMetricsReplicas:指标invalid metrics (1 invalid out of 1)与type不匹配。target- 如果,则target必须是
type: Utilization。averageUtilization - 如果,则target必须是
type: AverageValue。averageValue
- 如果
-
:
unable to fetch metrics from external metrics API: googleapi: Error 400: Metric label: 'LABEL' is not allowed中存在无效键。metric.selector.matchLabels- 移除或修正不被允许的标签;可在Cloud Monitoring指标文档中查找有效的可过滤标签。
-
副本数波动 / 来自不同HPA的事件相互矛盾:多个HPA通过
SuccessfulRescale以同一工作负载为目标,产生竞争。此问题没有专门的condition——可通过spec.scaleTargetRef确认,查找重复的kubectl get hpa -n {workload_namespace} -o yaml值。scaleTargetRef- 将所有指标整合到一个HPA对象中(HPA会取中计算出的最高副本数),并删除重复的HPA。
spec.metrics
- 将所有指标整合到一个HPA对象中(HPA会取
Branch B: Workload & Service Errors
分支B:工作负载与服务错误
-
, reason
ScalingActive: False, messageFailedGetResourceMetric(or a persistentunable to compute the replica count): the HPA computes utilization as a percentage of the container resource request, but at least one container in the Pod is missing aunable to fetch pod metricsentry for the scaled resource (resources.requestsorcpu).memory- Add for the scaled resource to every container in the Pod spec (including sidecars). A brief
resources.requestsright after the metrics server starts is normal and self-heals.unable to fetch pod metrics
- Add
-
: traffic-based autoscaling requires a one-to-one Service↔workload relationship, but more than one Service's selector matches the workload's Pods.
multiple services selecting the same target of HPA_NAME: SERVICE- Make the intended Service's selector unique (add a distinct label to the workload and to that one Service), or tighten the other Services' selectors so they no longer match the workload's Pods.
-
,原因为
ScalingActive: False,消息为FailedGetResourceMetric(或持续出现unable to compute the replica count):HPA将利用率计算为容器资源请求的百分比,但Pod中至少有一个容器缺少对应扩缩容资源(unable to fetch pod metrics或cpu)的memory条目。resources.requests- 为Pod规约中的每个容器(包括sidecar)添加对应扩缩容资源的。在metrics server启动后短暂出现
resources.requests是正常现象,会自行恢复。unable to fetch pod metrics
- 为Pod规约中的每个容器(包括sidecar)添加对应扩缩容资源的
-
:基于流量的自动扩缩容要求Service与工作负载是一对一关系,但有多个Service的选择器匹配到了该工作负载的Pod。
multiple services selecting the same target of HPA_NAME: SERVICE- 让目标Service的selector具有唯一性(为工作负载和该Service添加一个独特的标签),或者收紧其他Service的selector,使其不再匹配该工作负载的Pod。
Branch C: Metrics API & Data Availability (Custom / External Metrics)
分支C:指标API与数据可用性(自定义/外部指标)
The custom/external pipeline is: HPA controller → Kubernetes metrics API server
→ metrics adapter (for example ) → metric
source (Cloud Monitoring / Prometheus). Symptoms are metric values or
/ events.
custom-metrics-stackdriver-adapter<unknown>FailedGetExternalMetricFailedGetCustomMetric-
Is the adapter registered and available?bash
kubectl get apiservice | grep -E 'NAME|metrics.k8s.io'Expectand/orv1beta1.custom.metrics.k8s.iowithv1beta1.external.metrics.k8s.io. IfAVAILABLE: True/missing, the adapter is crashed or misconfigured — inspect its Pod logs in theFalseorcustom-metricsnamespace for permission, connectivity, or "metric not found" errors.kube-system -
Query the metrics API directly (bypasses the HPA to test the whole pipeline;optional):
jqbashkubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/{workload_namespace}/{metric_name}" | jq . kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/{workload_namespace}/pods/*/{metric_name}" | jq . -
Interpret the result:
- Valid JSON with a value → the pipeline works; the fault is in the HPA
manifest (metric-name typo or wrong ).
matchLabels - → network isolation is blocking the control plane from reaching the adapter. Add the adapter's
Error from server (Service Unavailable)to the control-plane firewall rule (in addition to the existingtargetPortandtcp:443). Identify the rule withtcp:10250, and also confirm no NetworkPolicy blocks ingress to the adapter Pods.gcloud compute firewall-rules list --filter="name~gke-{cluster_name}-[0-9a-z]*-master" - Empty list → the adapter runs but can't retrieve the metric. Inspect the adapter Pod logs, and confirm in Metrics Explorer that the metric actually exists in the source with the expected name and labels.
[]
- Valid JSON with a value → the pipeline works; the fault is in the HPA
manifest (metric-name typo or wrong
-
: a communication breakdown (control plane briefly unavailable during an upgrade/repair, or the adapter Pods are unhealthy or not registered) — not a problem at the metric source. Check control-plane health/notifications, confirm the adapter Pods are
unable to fetch metrics from custom metrics API: no known available metric versions foundwith no restarts (Running), and re-verify the APIServices arekubectl get pods -n custom-metrics,kube-system -o wide. Often transient.AVAILABLE: True -
: the query is valid but no data matched (different from a value of
googleapi: Error 400: The supplied filter ... will not return any time series) — the application wasn't writing the metric during the window. Verify the metric name/labels match what the app emits, confirm the app had permission and was active, and check the app logs for metric-emission errors.0
自定义/外部指标的管道为:HPA controller → Kubernetes metrics API server → metrics adapter(例如)→ 指标源(Cloud Monitoring / Prometheus)。症状为指标值为或出现 / 事件。
custom-metrics-stackdriver-adapter<unknown>FailedGetExternalMetricFailedGetCustomMetric-
adapter是否已注册且可用?bash
kubectl get apiservice | grep -E 'NAME|metrics.k8s.io'预期和/或v1beta1.custom.metrics.k8s.io的v1beta1.external.metrics.k8s.io状态为AVAILABLE。如果状态为True或缺失,说明adapter已崩溃或配置错误——请检查False或custom-metrics命名空间中其Pod的日志,查找权限、连通性或“metric not found”相关错误。kube-system -
直接查询指标API(绕过HPA测试整个管道;为可选工具):
jqbashkubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/{workload_namespace}/{metric_name}" | jq . kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/{workload_namespace}/pods/*/{metric_name}" | jq . -
结果解读:
- 返回带值的有效JSON → 管道正常;问题出在HPA清单上(指标名称拼写错误或不正确)。
matchLabels - → 网络隔离阻止了控制平面访问adapter。将adapter的
Error from server (Service Unavailable)添加到控制平面防火墙规则中(除了现有的targetPort和tcp:443之外)。可通过tcp:10250命令识别对应规则,同时确认没有NetworkPolicy阻止adapter Pod的入站流量。gcloud compute firewall-rules list --filter="name~gke-{cluster_name}-[0-9a-z]*-master" - 返回空列表→ adapter运行正常但无法获取指标。检查adapter Pod的日志,并在Metrics Explorer中确认指标源中确实存在具有预期名称和标签的指标。
[]
- 返回带值的有效JSON → 管道正常;问题出在HPA清单上(指标名称拼写错误或
-
:通信中断(升级/修复期间控制平面短暂不可用,或adapter Pod不健康或未注册)——并非指标源的问题。检查控制平面的健康状态/通知,确认adapter Pod处于
unable to fetch metrics from custom metrics API: no known available metric versions found状态且无重启(Running),并重新验证APIServices的kubectl get pods -n custom-metrics,kube-system -o wide状态为AVAILABLE。该问题通常是暂时性的。True -
:查询有效但没有匹配的数据(与值为
googleapi: Error 400: The supplied filter ... will not return any time series不同)——应用程序在该时间窗口内未写入指标。验证指标名称/标签与应用发出的是否匹配,确认应用具有权限且处于运行状态,并检查应用日志中是否有指标上报错误。0
Branch D: Healthy but Unexpected Scaling Behavior
分支D:状态正常但扩缩容行为不符合预期
The HPA's conditions are and it shows no errors, but scaling doesn't
happen as expected.
True-
Won't scale up — check, in order:
- Replica limits: is already at
currentReplicas/minReplicas(see themaxReplicascondition); adjust the bounds.ScalingLimited - Tolerance window: Kubernetes ignores changes while the
current/target ratio stays within –
0.9(default 10% tolerance). Example: target 85% CPU, current 93% → ratio ≈ 1.094 < 1.1, so no scale-up. Wait for the metric to move outside the band, or configure a different tolerance.1.1 - Unready Pods: /not-
PendingPods are excluded from the calculation — resolve the underlying scheduling/probe issue.Ready - Sync delay: a 15–30s delay between threshold crossing and action is normal.
- Replica limits:
-
Won't scale down — check, in order:
- Multiple metrics: the HPA uses the metric demanding the most replicas, so it won't scale down unless all metrics agree.
- Unavailable metric halts scale-down: if any metric goes the HPA conservatively refuses to scale down. Common with rate-based custom metrics that stop reporting at zero traffic. Prefer gauge metrics (for example
<unknown>) or make the source publishnum_undelivered_messagesduring inactivity rather than sending no data.0 - Scale-down stabilization window: the default
is 300s (5 min). Lower it if scale-down must be faster.
behavior.scaleDown.stabilizationWindowSeconds
HPA的所有conditions均为,且未显示任何错误,但扩缩容未按预期发生。
True-
无法扩容——按以下顺序检查:
- 副本限制:已达到
currentReplicas/minReplicas(请参阅maxReplicascondition);调整上下限即可。ScalingLimited - 容差窗口:当当前值/目标值的比率保持在–
0.9范围内时,Kubernetes会忽略变更(默认10%容差)。例如:CPU目标为85%,当前为93% → 比率≈1.094 < 1.1,因此不会扩容。可等待指标超出该范围,或配置不同的容差值。1.1 - 未就绪Pod:/非
Pending状态的Pod会被排除在计算之外——请解决底层的调度/探针问题。Ready - 同步延迟:从指标超过阈值到执行扩缩容操作之间有15-30秒的延迟是正常的。
- 副本限制:
-
无法缩容——按以下顺序检查:
- 多指标场景:HPA会采用要求最多副本数的指标,因此除非所有指标都支持缩容,否则不会执行缩容。
- 指标不可用会暂停缩容:如果任何指标变为,HPA会出于保守考虑拒绝缩容。这在基于速率的自定义指标中很常见,这类指标在流量为零时会停止上报。建议优先使用gauge类型的指标(例如
<unknown>),或者让指标源在无活动时发布num_undelivered_messages值,而不是不发送任何数据。0 - 缩容稳定窗口:的默认值为300秒(5分钟)。如果需要更快的缩容速度,可降低该值。
behavior.scaleDown.stabilizationWindowSeconds
Branch E: Scale To / From Zero (GKE 1.37+)
分支E:零副本扩缩容(GKE 1.37+)
Scaling a workload to and from zero replicas with HPA () is
supported on GKE 1.37 or later.
minReplicas: 0-
Won't scale to zero:
- must be set.
minReplicas: 0 - The HPA cannot scale to zero using only (CPU/memory) metrics — configure at least one
ResourceorExternalmetric.Object - With multiple metrics, all must evaluate to zero.
- GKE waits the 5-minute scale-down stabilization window at zero demand before going from 1→0.
- Any metric showing pauses scale-down (see Branch D).
<unknown>
-
Won't scale up from zero: runand check:
kubectl describe hpa {hpa_name}- Metrics: the value must be and not
> 0; if missing, confirm the external source (for example Pub/Sub) is publishing and Cloud Monitoring is receiving.<unknown> - Conditions: a healthy idle state shows ,
AbleToScale: True,ScalingActive: True. IfScaledToZero: Truewith reasonScalingActive: FalseorScalingDisabledExternalScaleToZero, the workload was manually scaled to zero (for exampleScalingDisabledReplicaCountZero), which pauses autoscaling. Resume it by scaling the Deployment back tokubectl scale.--replicas=1 - Cold-start latency from node provisioning can add delay; Capacity Buffers keep standby capacity ready.
- Metrics: the value must be
GKE 1.37或更高版本支持使用HPA()将工作负载扩缩容到零副本或从零副本扩容。
minReplicas: 0-
无法缩容到零副本:
- 必须设置。
minReplicas: 0 - HPA不能仅使用(CPU/内存)指标缩容到零——至少需要配置一个
Resource或External类型的指标。Object - 在多指标场景下,所有指标的计算结果都必须为零。
- 在需求为零时,GKE会等待5分钟的缩容稳定窗口后才会从1副本缩容到0副本。
- 任何指标显示为都会暂停缩容(请参阅分支D)。
<unknown>
- 必须设置
-
无法从零副本扩容:执行并检查:
kubectl describe hpa {hpa_name}- Metrics:值必须且不为
> 0;如果缺失,确认外部指标源(例如Pub/Sub)正在发布数据,且Cloud Monitoring正在接收。<unknown> - Conditions:健康的空闲状态会显示、
AbleToScale: True、ScalingActive: True。如果ScaledToZero: True且原因为ScalingActive: False或ScalingDisabledExternalScaleToZero,说明工作负载是手动缩容到零的(例如通过ScalingDisabledReplicaCountZero),这会暂停自动扩缩容。将Deployment重新扩容到kubectl scale即可恢复自动扩缩容。--replicas=1 - 节点供应带来的冷启动延迟会增加等待时间;Capacity Buffers可提供备用容量以缩短延迟。
- Metrics:值必须
Branch F: Slow HPA Recalculation on Large Clusters
分支F:大型集群中HPA重计算缓慢
If HPAs are correct but react slowly, the cluster may exceed the HPA object count
the standard controller keeps within a 15-second recalculation period.
- Standard controller: within 15s for up to 300 HPA objects (GKE 1.22+).
- Performance HPA profile: within 15s for up to 1,000 HPA objects (GKE 1.31+) or 5,000 HPA objects (GKE 1.33+, where it is enabled by default on eligible clusters).
Enable it on an eligible cluster (this is a cluster mutation — present it for the
operator to run, don't execute it):
bash
gcloud container clusters update {cluster_name} \
--location {cluster_location} --project {project_id} \
--hpa-profile=performanceScaling on many metrics per HPA and slow (>~50 ms) custom-metric adapters also
lengthen the recalculation period. For visibility into scaling decisions, enable
HPA event logging and review the structured HPA decision logs.
如果HPA配置正确但响应缓慢,可能是集群中的HPA对象数量超过了标准controller在15秒重计算周期内可处理的上限。
- 标准controller:在GKE 1.22+版本中,最多可处理300个HPA对象,重计算周期在15秒以内。
- Performance HPA配置文件:在GKE 1.31+版本中,最多可处理1000个HPA对象,重计算周期在15秒以内;在GKE 1.33+版本中,最多可处理5000个HPA对象(符合条件的集群默认启用该配置)。
在符合条件的集群上启用该配置(这是集群变更操作——请提供给运维人员执行,不要自动执行):
bash
gcloud container clusters update {cluster_name} \
--location {cluster_location} --project {project_id} \
--hpa-profile=performance每个HPA配置的指标数量过多,以及自定义指标adapter响应缓慢(>~50ms),也会延长重计算周期。如需了解扩缩容决策的详细信息,可启用HPA事件日志并查看结构化的HPA决策日志。
Step 3: Propose the GitOps Correction
步骤3:提出GitOps修正方案
Enforce the read-only diagnostics boundary: do not apply live mutations. This
includes cluster/manifest changes (, , , , ) and Google Cloud / gcloud changes
(cluster updates such as , and
updates). Instead, present the corrected HorizontalPodAutoscaler, PodSpec
(), Service selector, workload autoscaling
configuration, or the command as a reviewable patch/command to be
applied through the user's GitOps pipeline (for example Config Sync, Argo CD, or
Flux) or by an authorized operator.
kubectl editkubectl patchkubectl applykubectl scalekubectl delete--hpa-profilegcloud compute firewall-rulesresources.requestsgcloud严格遵守只读诊断边界:不要直接应用实时变更。这包括集群/清单变更(、、、、)以及Google Cloud / gcloud相关变更(例如等集群更新,以及更新)。相反,应将修正后的HorizontalPodAutoscaler、PodSpec()、Service selector、工作负载自动扩缩容配置或命令以可审核的补丁/命令形式呈现,由用户通过其GitOps管道(例如Config Sync、Argo CD或Flux)或授权运维人员执行。
kubectl editkubectl patchkubectl applykubectl scalekubectl delete--hpa-profilegcloud compute firewall-rulesresources.requestsgcloudReferences
参考资料
- Troubleshoot horizontal Pod autoscaling in GKE
- Horizontal Pod autoscaling (concepts, Performance HPA profile & limits)
- Configure horizontal Pod autoscaling (hpa-profile)
- View horizontal Pod autoscaling events
- Scale to and from zero using HPA
- Kubernetes HorizontalPodAutoscaler (tolerance & scaling policies)