compute-resource-availability-troubleshooting
Compare original and translation side by side
🇺🇸
Original
English🇨🇳
Translation
ChineseCompute Engine Resource Availability and Stockout Troubleshooting
Compute Engine 资源可用性与容量库存耗尽故障排查
MCP Tools:,list_machine_types,list_accelerator_types,get_zone_operation,list_instance_group_managers,list_managed_instances,list_reservationsget_reservation_details
This skill diagnoses and resolves Google Compute Engine resource availability
errors (capacity stockouts). It provides immediate tactical workarounds to get
blocked workloads running and long-term architectural mitigations using
Regional Managed Instance Group (MIG) distribution shapes, Instance
Flexibility, Dynamic Workload Scheduler (DWS) Flex-Start, and On-Demand
Capacity Reservations.
MCP 工具:、list_machine_types、list_accelerator_types、get_zone_operation、list_instance_group_managers、list_managed_instances、list_reservationsget_reservation_details
此 skill 用于诊断并解决 Google Compute Engine 资源可用性错误(容量库存耗尽)。它既提供可立即采用的临时变通方案以让受阻工作负载恢复运行,也提供基于 Regional Managed Instance Group(MIG)distribution shapes、Instance Flexibility、Dynamic Workload Scheduler(DWS)Flex-Start 和 On-Demand Capacity Reservations 的长期架构缓解措施。
Usage Attribution
使用归属
MUST READ: Before executing any , REST API, or MCP command, you
MUST read and follow the instructions in
Usage Attribution.
gcloud必须阅读:在执行任何 、REST API 或 MCP 命令之前,您必须阅读并遵循使用归属中的说明。
gcloudPrerequisites & Environment Setup
前置条件与环境配置
Any host, project, IAM, or Compute Engine resource modifications (installing
, configuring MCP, enabling APIs or Log Analytics, creating instances
or reservations, updating MIGs, or deleting resources) must be explicitly
approved by the user before execution.
gcloud- Google Cloud SDK: Verify is installed (
gcloud). If missing or outdated, ask the user for approval before installing via Install the Google Cloud SDK.gcloud --version - Compute Engine MCP Server & Capability Boundary: When the Compute Engine
MCP server is configured (see
Use the Compute Engine MCP server;
requires ), prefer its read-only discovery tools before falling back to
roles/mcp.toolUser:gcloud- and
list_machine_typesto check whether a machine shape or GPU accelerator is offered in a target zone.list_accelerator_types - to inspect a failed zonal operation's
get_zone_operationarray (error.errors,ZONE_RESOURCE_POOL_EXHAUSTED, orZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS).QUOTA_EXCEEDED - and
list_instance_group_managersto inspect zonal MIGs and per-instancelist_managed_instances/currentAction(note: all Compute Engine MCP MIG tools—lastAttempt.errors,list_instance_group_managers, andlist_managed_instances—only accept aget_instance_group_manager_basic_infoparameter and cannot inspect regional MIGs; usezonewithgcloudor REST for regional MIGs).--region={region} - and
list_reservationsto inspect existing zonal reservation capacity (get_reservation_detailsvs.specificReservation.count).specificReservation.inUseCount - MCP Limitations (Use or REST Instead): MCP
gclouddoes not exposecreate_instance(provisioning_model/FLEX_START),SPOT,max_run_duration, orinstance_termination_action. MCP also lacks tools for Cloud Logging queries (reservation_affinity), Spot Capacity Advisor (gcloud logging read), Bulk Insert (gcloud beta compute advice capacity), MIG distribution updates (gcloud compute instances bulk create), and reservation creation or deletion (gcloud compute instance-groups managed update/gcloud compute reservations create). Always usedeleteor the Compute Engine REST API for those operations.gcloud
- Authentication: Local CLI (), Application Default Credentials (
gcloud auth login), or headless/CI-CD viagcloud auth application-default loginor Workload Identity Federation.GOOGLE_APPLICATION_CREDENTIALS - Project, Region, Billing & API Enablement:
Note: An active billing account attachment is required (even for Free Tier eligible resources such assh
gcloud config set project {project_id} gcloud config set compute/region {region} gcloud config set compute/zone {zone} gcloud services enable compute.googleapis.com logging.googleapis.com). Never hardcode regions or zones; confirme2-micro,{project_id}, and{region}(defaulting to a well-supported region such as{zone}/us-central1if unspecified), and consult Google Cloud Locations to verify regional and zonal availability (preview features and specialized GPUs/TPUs are restricted to specific locations).us-central1-a - Required IAM Roles:
- Read-Only Diagnostics & Capacity Advice: and
roles/compute.viewer.roles/logging.viewer - Log Analytics SQL ():
_Default._AllLogs(plusroles/logging.viewAccessorandroles/bigquery.jobUserwhen querying via a linked BigQuery dataset).roles/bigquery.dataViewer - Instance & MIG Remediation: (plus
roles/compute.instanceAdmin.v1when attaching a service account to VMs).roles/iam.serviceAccountUser - Reservation Creation & Teardown: (or a custom role granting
roles/compute.admin,compute.reservations.create, andcompute.reservations.update, which are not included incompute.reservations.delete).roles/compute.instanceAdmin.v1 - MCP Tool Invocation: .
roles/mcp.toolUser
- Read-Only Diagnostics & Capacity Advice:
任何针对主机、项目、IAM 或 Compute Engine 资源的修改(安装 、配置 MCP、启用 API 或 Log Analytics、创建实例或预留、更新 MIG 或删除资源),执行前均必须获得用户明确批准。
gcloud- Google Cloud SDK:确认已安装 (
gcloud)。如果缺失或版本过旧,应请求用户批准后再通过安装 Google Cloud SDK进行安装。gcloud --version - Compute Engine MCP Server 与能力边界:配置 Compute Engine MCP server 后(参见使用 Compute Engine MCP server;需要 ),应优先使用其只读发现工具,再回退到
roles/mcp.toolUser:gcloud- 使用 和
list_machine_types检查目标可用区是否提供特定 machine shape 或 GPU accelerator。list_accelerator_types - 使用 检查失败的可用区级操作的
get_zone_operation数组(error.errors、ZONE_RESOURCE_POOL_EXHAUSTED或ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS)。QUOTA_EXCEEDED - 使用 和
list_instance_group_managers检查可用区级 MIG,以及各实例的list_managed_instances/currentAction(注意:所有 Compute Engine MCP MIG 工具——lastAttempt.errors、list_instance_group_managers和list_managed_instances——仅接受get_instance_group_manager_basic_info参数,无法检查区域级 MIG;请使用带zone的--region={region}或 REST)。gcloud - 使用 和
list_reservations检查现有可用区级预留容量(get_reservation_details与specificReservation.count)。specificReservation.inUseCount - MCP 限制(改用 或 REST):MCP
gcloud未暴露create_instance(provisioning_model/FLEX_START)、SPOT、max_run_duration或instance_termination_action。MCP 也不支持 Cloud Logging 查询(reservation_affinity)、Spot Capacity Advisor(gcloud logging read)、Bulk Insert(gcloud beta compute advice capacity)、MIG distribution 更新(gcloud compute instances bulk create),以及预留创建或删除(gcloud compute instance-groups managed update/gcloud compute reservations create)。此类操作始终应使用delete或 Compute Engine REST API。gcloud
- 使用
- 身份验证:本地 CLI()、Application Default Credentials(
gcloud auth login),或通过gcloud auth application-default login、Workload Identity Federation 在无头/CI-CD 环境中进行身份验证。GOOGLE_APPLICATION_CREDENTIALS - 项目、区域、计费与 API 启用:
注意:必须绑定有效的结算账户,即使资源符合 Free Tier 条件(如sh
gcloud config set project {project_id} gcloud config set compute/region {region} gcloud config set compute/zone {zone} gcloud services enable compute.googleapis.com logging.googleapis.com)也不例外。切勿硬编码 region 或 zone;应确认e2-micro、{project_id}和{region}(如果未指定,则默认使用{zone}/us-central1等支持范围较广的区域),并查阅 Google Cloud Locations 以核实区域和可用区可用性(preview features 以及专用 GPU/TPU 仅在特定 location 提供)。us-central1-a - 必需的 IAM Roles:
- 只读诊断与容量建议:和
roles/compute.viewer。roles/logging.viewer - Log Analytics SQL():
_Default._AllLogs(通过关联的 BigQuery dataset 查询时,还需要roles/logging.viewAccessor和roles/bigquery.jobUser)。roles/bigquery.dataViewer - 实例与 MIG 修复:(向 VM 附加 service account 时还需要
roles/compute.instanceAdmin.v1)。roles/iam.serviceAccountUser - 预留创建与拆除:(或授予
roles/compute.admin、compute.reservations.create和compute.reservations.update的自定义 role;这些权限不包含在compute.reservations.delete中)。roles/compute.instanceAdmin.v1 - MCP 工具调用:。
roles/mcp.toolUser
- 只读诊断与容量建议:
Step 1: Verify Stockout Scope & Distinguish from Quota Errors
第 1 步:验证库存耗尽范围并与配额错误区分
A. Core Domain Rules (Explain to User)
A. 核心领域规则(向用户说明)
Before taking action, clarify these three foundational Compute Engine behaviors
(see
Troubleshooting resource availability errors):
- Stockout != Quota: (and
ZONE_RESOURCE_POOL_EXHAUSTED) means physical hardware capacity for that specific resource pool in that specific zone is temporarily full. Increasing project quota (IAM & Admin > Quotas & System Limits) will not resolve a physical stockout. Conversely,ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILSmeans the project's administrative quota ceiling has been reached regardless of physical hardware availability. When returned,QUOTA_EXCEEDEDalso includesZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS(such aserrorInfo.reasonfor unsupported hardware combinations that will not succeed on retry) andconfiguration_availabilitylisting sibling zones with capacity.errorInfo.metadata.zonesAvailable - Scoped to Specific Shape & Zone: A stockout applies only to the exact
machine shape or accelerator (for example, or
g2-standard-4) in that zone (for example,c4-standard-32). Sibling zones (us-central1-a,us-central1-b,us-central1-c) or smaller shapes (us-central1-f) frequently have ample capacity.c4-standard-16 - Existing Running VMs Are Safe: Resource availability errors affect only
new capacity allocations (creating new VMs, starting stopped
VMs, changing machine types, or MIG scale-out). Currently running
TERMINATEDVMs in the zone are unaffected.RUNNING
采取任何操作之前,应先说明以下三个 Compute Engine 基础行为(参见资源可用性错误故障排查):
- 库存耗尽不等于配额问题:(以及
ZONE_RESOURCE_POOL_EXHAUSTED)表示特定可用区中特定资源池的物理硬件容量暂时已满。提高项目配额(IAM & Admin > Quotas & System Limits)无法解决物理库存耗尽问题。相反,ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS表示无论物理硬件是否可用,项目的管理配额上限均已达到。返回QUOTA_EXCEEDED时,还会包含ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS(例如errorInfo.reason,表示不支持的硬件组合,重试也不会成功),以及configuration_availability,其中列出了有容量的相邻可用区。errorInfo.metadata.zonesAvailable - 限定于特定规格与可用区:库存耗尽仅适用于该可用区中的确切 machine shape 或 accelerator(例如 或
g2-standard-4),例如c4-standard-32。相邻可用区(us-central1-a、us-central1-b、us-central1-c)或更小的规格(us-central1-f)通常仍有充足容量。c4-standard-16 - 现有运行中的 VM 不受影响:资源可用性错误仅影响新容量分配(创建新 VM、启动已停止的 VM、更改 machine type 或 MIG 扩容)。该可用区中当前处于
TERMINATED状态的 VM 不受影响。RUNNING
B. Query Cloud Audit Logs for Resource Availability vs. Quota Failures
B. 查询 Cloud Audit Logs,区分资源可用性问题与配额失败
Run with an explicit window ( defaults to , so multi-day investigations require
or ). Inspect
(not ) to distinguish
/
from :
gcloud logging read--freshnessgcloud logging read1d--freshness=7d--freshness=14dprotoPayload.response.error.errors[0].code.reasonZONE_RESOURCE_POOL_EXHAUSTEDZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILSQUOTA_EXCEEDEDsh
gcloud logging read \
'resource.type="gce_instance" AND log_id("cloudaudit.googleapis.com/activity") AND (protoPayload.status.message:("ZONE_RESOURCE_POOL_EXHAUSTED" OR "does not have enough resources available" OR "resource pool exhausted" OR "does not exist in zone" OR "QUOTA_EXCEEDED" OR "Quota") OR protoPayload.response.error.errors.code:("ZONE_RESOURCE_POOL_EXHAUSTED" OR "ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS" OR "QUOTA_EXCEEDED"))' \
--project={project_id} --freshness=7d --limit=20 \
--format="table(timestamp, resource.labels.zone, protoPayload.resourceName, protoPayload.response.error.errors[0].code, protoPayload.status.message)"- If is
code(Always Include Regional Quota Inspection): Do not treat this as a physical stockout. Always provideQUOTA_EXCEEDEDto inspect regional quota limits and usage, and direct the user to IAM & Admin > Quotas & System Limits (or switch to a machine family with quota headroom):gcloud compute regions describeshgcloud compute regions describe {region} --project={project_id} \ --format="table(quotas:format='table(metric,limit,usage)')" - If a MIG is failing to scale out: Also inspect MIG provisioning errors
directly (via Compute Engine MCP for zonal MIGs, or
list_managed_instances):gcloudsh# Zonal MIG (--zone={zone}) vs. Regional MIG (--region={region}): gcloud compute instance-groups managed list-errors {mig_name} \ --project={project_id} --zone={zone} gcloud compute instance-groups managed list-errors {mig_name} \ --project={project_id} --region={region}
使用明确的 时间窗口运行 ( 默认使用 ,因此持续多天的调查需要 或 )。检查 (而不是 ),以区分 / 与 :
--freshnessgcloud logging readgcloud logging read1d--freshness=7d--freshness=14dprotoPayload.response.error.errors[0].code.reasonZONE_RESOURCE_POOL_EXHAUSTEDZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILSQUOTA_EXCEEDEDsh
gcloud logging read \
'resource.type="gce_instance" AND log_id("cloudaudit.googleapis.com/activity") AND (protoPayload.status.message:("ZONE_RESOURCE_POOL_EXHAUSTED" OR "does not have enough resources available" OR "resource pool exhausted" OR "does not exist in zone" OR "QUOTA_EXCEEDED" OR "Quota") OR protoPayload.response.error.errors.code:("ZONE_RESOURCE_POOL_EXHAUSTED" OR "ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS" OR "QUOTA_EXCEEDED"))' \
--project={project_id} --freshness=7d --limit=20 \
--format="table(timestamp, resource.labels.zone, protoPayload.resourceName, protoPayload.response.error.errors[0].code, protoPayload.status.message)"- 如果 为
code(始终检查区域配额): 不要将其视为物理库存耗尽。始终提供QUOTA_EXCEEDED以检查区域配额上限和使用量,并引导用户前往 IAM & Admin > Quotas & System Limits(或切换到具有配额余量的 machine family):gcloud compute regions describeshgcloud compute regions describe {region} --project={project_id} \ --format="table(quotas:format='table(metric,limit,usage)')" - 如果 MIG 扩容失败:还应直接检查 MIG 配置错误(对于可用区级 MIG,可使用 Compute Engine MCP ;也可使用
list_managed_instances):gcloudsh# Zonal MIG (--zone={zone}) vs. Regional MIG (--region={region}): gcloud compute instance-groups managed list-errors {mig_name} \ --project={project_id} --zone={zone} gcloud compute instance-groups managed list-errors {mig_name} \ --project={project_id} --region={region}
C. Analyze Historical Stockout Frequency with Log Analytics SQL
C. 使用 Log Analytics SQL 分析历史库存耗尽频率
Because failed entries have severity
(), Cloud Logging routes them to the
log bucket (), whereas
only stores Admin Activity logs below and System
Event logs.
cloudaudit.googleapis.com/activityERRORseverity_number >= 500_Default{project_id}.global._Default._AllLogs_Required._AllLogsERROR- Prerequisite: Upgrade to Log Analytics (see Query and view logs in Log Analytics):
_Defaultshgcloud logging buckets update _Default \ --project={project_id} --location=global --enable-analytics - JSON Column Extraction Rule: In ,
_AllLogsandresource.labelshave BigQuery typeproto_payload.audit_log.request. Never referenceJSONdirectly inresource.labels.zone—always extract scalarGROUP BYvalues withSTRING:JSON_VALUE(...)sqlSELECT JSON_VALUE(resource.labels.zone) AS zone, REGEXP_EXTRACT(JSON_VALUE(proto_payload.audit_log.request.machineType), r'([^/]+)$') AS machine_type, proto_payload.audit_log.status.message AS status_message, COUNT(*) AS stockout_count, MIN(timestamp) AS first_seen, MAX(timestamp) AS last_seen FROM `{project_id}.global._Default._AllLogs` WHERE timestamp >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 14 DAY) AND log_id = 'cloudaudit.googleapis.com/activity' AND resource.type = 'gce_instance' AND ( proto_payload.audit_log.status.message LIKE '%ZONE_RESOURCE_POOL_EXHAUSTED%' OR proto_payload.audit_log.status.message LIKE '%does not have enough resources available%' OR TO_JSON_STRING(proto_payload.audit_log.response.error.errors) LIKE '%ZONE_RESOURCE_POOL_EXHAUSTED%' ) GROUP BY zone, machine_type, status_message ORDER BY stockout_count DESC;
由于失败的 条目严重级别为 (),Cloud Logging 会将其路由到 log bucket(),而 仅存储严重级别低于 的 Admin Activity 日志和 System Event 日志。
cloudaudit.googleapis.com/activityERRORseverity_number >= 500_Default{project_id}.global._Default._AllLogs_Required._AllLogsERROR- 前置条件:将 升级为 Log Analytics(参见在 Log Analytics 中查询和查看日志):
_Defaultshgcloud logging buckets update _Default \ --project={project_id} --location=global --enable-analytics - JSON 列提取规则:在 中,
_AllLogs和resource.labels的 BigQuery 类型为proto_payload.audit_log.request。切勿在JSON中直接引用GROUP BY;必须使用resource.labels.zone提取标量JSON_VALUE(...)值:STRINGsqlSELECT JSON_VALUE(resource.labels.zone) AS zone, REGEXP_EXTRACT(JSON_VALUE(proto_payload.audit_log.request.machineType), r'([^/]+)$') AS machine_type, proto_payload.audit_log.status.message AS status_message, COUNT(*) AS stockout_count, MIN(timestamp) AS first_seen, MAX(timestamp) AS last_seen FROM `{project_id}.global._Default._AllLogs` WHERE timestamp >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 14 DAY) AND log_id = 'cloudaudit.googleapis.com/activity' AND resource.type = 'gce_instance' AND ( proto_payload.audit_log.status.message LIKE '%ZONE_RESOURCE_POOL_EXHAUSTED%' OR proto_payload.audit_log.status.message LIKE '%does not have enough resources available%' OR TO_JSON_STRING(proto_payload.audit_log.response.error.errors) LIKE '%ZONE_RESOURCE_POOL_EXHAUSTED%' ) GROUP BY zone, machine_type, status_message ORDER BY stockout_count DESC;
Step 2: Immediate Tactical Mitigations (Unblock Workload Now)
第 2 步:立即采取临时缓解措施(解除当前工作负载阻塞)
Execute these five steps in priority order to provision the blocked workload
immediately:
按优先级执行以下五个步骤,立即为受阻工作负载提供资源:
1. Discover Sibling Zones Supporting the Machine Shape or GPU
1. 发现支持目标 machine shape 或 GPU 的相邻可用区
If a zone returns or , check which sibling zones in
offer the target machine shape and (if applicable) GPU accelerator using
Compute Engine MCP / or :
ZONE_RESOURCE_POOL_EXHAUSTEDMachine type with name '...' does not exist in zone '...'{region}list_machine_typeslist_accelerator_typesgcloudsh
undefined如果可用区返回 或 ,请使用 Compute Engine MCP / 或 ,检查 中哪些相邻可用区提供目标 machine shape,以及在适用时提供 GPU accelerator:
ZONE_RESOURCE_POOL_EXHAUSTEDMachine type with name '...' does not exist in zone '...'list_machine_typeslist_accelerator_typesgcloud{region}sh
undefinedList all zones in {region} that offer {machine_type}:
List all zones in {region} that offer {machine_type}:
gcloud compute machine-types list --project={project_id}
--filter="name={machine_type} AND zone ~ ^{region}-"
--format="table(name, zone, guestCpus, memoryMb)"
--filter="name={machine_type} AND zone ~ ^{region}-"
--format="table(name, zone, guestCpus, memoryMb)"
gcloud compute machine-types list --project={project_id}
--filter="name={machine_type} AND zone ~ ^{region}-"
--format="table(name, zone, guestCpus, memoryMb)"
--filter="name={machine_type} AND zone ~ ^{region}-"
--format="table(name, zone, guestCpus, memoryMb)"
For GPU workloads (especially N1 with attached GPUs), verify accelerator zones:
For GPU workloads (especially N1 with attached GPUs), verify accelerator zones:
gcloud compute accelerator-types list --project={project_id}
--filter="name={accelerator_type} AND zone ~ ^{region}-"
--format="table(name, zone, maximumCardsPerInstance)"
--filter="name={accelerator_type} AND zone ~ ^{region}-"
--format="table(name, zone, maximumCardsPerInstance)"
undefinedgcloud compute accelerator-types list --project={project_id}
--filter="name={accelerator_type} AND zone ~ ^{region}-"
--format="table(name, zone, maximumCardsPerInstance)"
--filter="name={accelerator_type} AND zone ~ ^{region}-"
--format="table(name, zone, maximumCardsPerInstance)"
undefined2. Use Bulk Insert (bulk create
) to Pick an Available Zone
bulk create2. 使用 Bulk Insert(bulk create
)选择有容量的可用区
bulk createInstead of guessing and retrying individual zones manually, use with and
(for colocated VMs or )
or (to spread across available zones).
Compute Engine automatically selects zone(s) in with capacity (see
Create VMs in bulk):
gcloud compute instances bulk create--region={region}--target-distribution-shape=ANY_SINGLE_ZONE--count=1--target-distribution-shape=BALANCED{region}sh
undefined不要手动猜测并逐一重试可用区,而应使用 ,并设置 和 (用于共置 VM 或 ),或设置 (用于分散到多个可用区)。Compute Engine 会自动选择 中具有容量的可用区(参见批量创建 VM):
gcloud compute instances bulk create--region={region}--target-distribution-shape=ANY_SINGLE_ZONE--count=1--target-distribution-shape=BALANCED{region}sh
undefinedOn instances bulk create, --target-distribution-shape is UPPERCASE (ANY, ANY_SINGLE_ZONE, BALANCED):
On instances bulk create, --target-distribution-shape is UPPERCASE (ANY, ANY_SINGLE_ZONE, BALANCED):
gcloud compute instances bulk create
--project={project_id} --region={region}
--name-pattern="{vm_name}-#" --count={vm_count}
--target-distribution-shape=ANY_SINGLE_ZONE
--machine-type={machine_type}
--project={project_id} --region={region}
--name-pattern="{vm_name}-#" --count={vm_count}
--target-distribution-shape=ANY_SINGLE_ZONE
--machine-type={machine_type}
undefinedgcloud compute instances bulk create
--project={project_id} --region={region}
--name-pattern="{vm_name}-#" --count={vm_count}
--target-distribution-shape=ANY_SINGLE_ZONE
--machine-type={machine_type}
--project={project_id} --region={region}
--name-pattern="{vm_name}-#" --count={vm_count}
--target-distribution-shape=ANY_SINGLE_ZONE
--machine-type={machine_type}
undefined3. Check Spot VM Obtainability via Spot Capacity Advisor (advice capacity
)
advice capacity3. 通过 Spot Capacity Advisor(advice capacity
)检查 Spot VM 可获取性
advice capacityWhen provisioning Spot VMs, use (the
Spot Capacity Advisor API) to compare real-time Spot obtainability scores and
estimated uptime () across zones and machine types in
before creating instances (use
for 30-day historical preemption
rates; see
View the availability of Spot VMs):
gcloud beta compute advice capacityestimatedUptime{region}gcloud beta compute advice capacity-history- Critical Scope & Flag Rules:
- supports only Spot VMs (
gcloud beta compute advice capacity, the only supported value; it cannot query--provisioning-model=SPOTon-demand capacity).STANDARD - There is no flag on
--machine-type—specify machine types withadvice capacityor--instance-selection-machine-types={machine_type_1},{machine_type_2}.--instance-selection="name={selection_name},machine-type={machine_type}" - On ,
advice capacitychoices are UPPERCASE (--target-distribution-shape,ANY, orANY_SINGLE_ZONE). Always includeBALANCED,--provisioning-model=SPOT, and uppercase--instance-selection-machine-types(or--target-distribution-shape=ANY/ANY_SINGLE_ZONE):BALANCED
sh
gcloud beta compute advice capacity \
--project={project_id} --region={region} \
--provisioning-model=SPOT --size={vm_count} \
--instance-selection-machine-types={machine_type_1},{machine_type_2} \
--target-distribution-shape=ANY配置 Spot VMs 时,使用 (Spot Capacity Advisor API),在创建实例之前比较 中各可用区和 machine type 的实时 Spot 可获取性评分及预计正常运行时间();使用 查看过去 30 天的抢占率历史(参见查看 Spot VM 的可用性):
gcloud beta compute advice capacity{region}estimatedUptimegcloud beta compute advice capacity-history- 关键范围与标志规则:
- 仅支持 Spot VMs(
gcloud beta compute advice capacity是唯一受支持的值;无法查询--provisioning-model=SPOT按需容量)。STANDARD - 没有
advice capacity标志;应通过--machine-type或--instance-selection-machine-types={machine_type_1},{machine_type_2}指定 machine type。--instance-selection="name={selection_name},machine-type={machine_type}" - 在 中,
advice capacity的可选值为大写(--target-distribution-shape、ANY或ANY_SINGLE_ZONE)。始终包含BALANCED、--provisioning-model=SPOT和大写的--instance-selection-machine-types(或--target-distribution-shape=ANY/ANY_SINGLE_ZONE):BALANCED
sh
gcloud beta compute advice capacity \
--project={project_id} --region={region} \
--provisioning-model=SPOT --size={vm_count} \
--instance-selection-machine-types={machine_type_1},{machine_type_2} \
--target-distribution-shape=ANY4. Right-Size Large Shapes (> 32 vCPUs)
4. 优化大型规格(> 32 vCPUs)
Physical host fragmentation makes large machine shapes (such as ,
, or ) significantly scarcer than smaller shapes.
If the workload can scale horizontally, split a single 64-vCPU request into two
32-vCPU VMs or four 16-vCPU VMs.
*-standard-64*-standard-96*-standard-128物理主机碎片化会使大型 machine shape(例如 、 或 )比小型 shape 明显更难获得。如果工作负载可以横向扩展,可将单个 64-vCPU 请求拆分为两个 32-vCPU VM,或四个 16-vCPU VM。
*-standard-64*-standard-96*-standard-1285. Use Dynamic Workload Scheduler (DWS) Flex-Start for Scarce GPUs
5. 对稀缺 GPU 使用 Dynamic Workload Scheduler(DWS)Flex-Start
When scarce GPU shapes (, , , or
) fail immediate creation with ,
use Dynamic Workload Scheduler (DWS) Flex-Start to queue and automatically
provision capacity when hardware frees up (see
About Dynamic Workload Scheduler
and
Create a standalone Flex-start VM):
a4-highgpu-8ga3-highgpu-8ga2-highgpu-1gg2-standard-4ZONE_RESOURCE_POOL_EXHAUSTED- Mandatory CLI Rules for DWS Flex-Start Standalone VMs:
- Machine Family Accelerator & Maintenance Rule: For (
G2),nvidia-l4,G4(A2),nvidia-tesla-a100, andA3machine series, GPUs are automatically attached by the machine type (omitA4unless attaching GPUs to--acceleratoror using vWS). Standalone Flex-Start GPU creation requiresN1,--reservation-affinity=none(--max-run-durationto10m),7d(or--instance-termination-action=DELETE),STOP, and--maintenance-policy=TERMINATE(--request-valid-for-durationto90sto queue in a zone, or2hfor immediate-only allocation).0s - For multi-instance GPU jobs or longer/multi-zone queueing, use a MIG
with a Flex-Start instance template and resize requests (see
Create resize requests in a MIG).
The MIG requires autoscaling disabled,
, and (on a Regional MIG)
--default-action-on-vm-failure=do-nothingwith--target-distribution-shape=any-single-zone.--instance-redistribution-type=none
- Machine Family Accelerator & Maintenance Rule: For
sh
undefined当稀缺 GPU shape(、、 或 )因 而无法立即创建时,使用 Dynamic Workload Scheduler(DWS)Flex-Start 对容量进行排队,并在硬件释放后自动预配资源(参见关于 Dynamic Workload Scheduler和创建独立 Flex-Start VM):
a4-highgpu-8ga3-highgpu-8ga2-highgpu-1gg2-standard-4ZONE_RESOURCE_POOL_EXHAUSTED- DWS Flex-Start 独立 VM 的强制 CLI 规则:
- Machine Family Accelerator 与维护规则:对于 (
G2)、nvidia-l4、G4(A2)、nvidia-tesla-a100和A3machine series,GPU 会由 machine type 自动附加(除非为A4附加 GPU 或使用 vWS,否则省略N1)。创建独立 Flex-Start GPU 需要--accelerator、--reservation-affinity=none(--max-run-duration到10m)、7d(或--instance-termination-action=DELETE)、STOP和--maintenance-policy=TERMINATE(使用--request-valid-for-duration到90s在某个可用区排队,或使用2h仅立即分配)。0s - 对于多实例 GPU 作业或需要更长时间/跨可用区排队的工作负载,使用带有 Flex-Start instance template 和 resize request 的 MIG(参见在 MIG 中创建 resize request)。该 MIG 要求禁用 autoscaling,设置 ;对于 Regional MIG,还要求设置
--default-action-on-vm-failure=do-nothing和--target-distribution-shape=any-single-zone。--instance-redistribution-type=none
- Machine Family Accelerator 与维护规则:对于
sh
undefinedOption A: Standalone VM with DWS Flex-Start (queues up to 2h via --request-valid-for-duration):
Option A: Standalone VM with DWS Flex-Start (queues up to 2h via --request-valid-for-duration):
gcloud compute instances create {vm_name}
--project={project_id} --zone={zone}
--machine-type=g2-standard-4
--provisioning-model=FLEX_START
--reservation-affinity=none
--request-valid-for-duration=2h
--max-run-duration=7d
--instance-termination-action=DELETE
--maintenance-policy=TERMINATE
--project={project_id} --zone={zone}
--machine-type=g2-standard-4
--provisioning-model=FLEX_START
--reservation-affinity=none
--request-valid-for-duration=2h
--max-run-duration=7d
--instance-termination-action=DELETE
--maintenance-policy=TERMINATE
gcloud compute instances create {vm_name}
--project={project_id} --zone={zone}
--machine-type=g2-standard-4
--provisioning-model=FLEX_START
--reservation-affinity=none
--request-valid-for-duration=2h
--max-run-duration=7d
--instance-termination-action=DELETE
--maintenance-policy=TERMINATE
--project={project_id} --zone={zone}
--machine-type=g2-standard-4
--provisioning-model=FLEX_START
--reservation-affinity=none
--request-valid-for-duration=2h
--max-run-duration=7d
--instance-termination-action=DELETE
--maintenance-policy=TERMINATE
Option B: Instance Template + Zonal or Regional MIG Resize Request (longer queueing):
Option B: Instance Template + Zonal or Regional MIG Resize Request (longer queueing):
gcloud compute instance-templates create {template_name}
--project={project_id} --machine-type=g2-standard-4
--provisioning-model=FLEX_START
--reservation-affinity=none
--max-run-duration=7d
--instance-termination-action=DELETE
--maintenance-policy=TERMINATE
--project={project_id} --machine-type=g2-standard-4
--provisioning-model=FLEX_START
--reservation-affinity=none
--max-run-duration=7d
--instance-termination-action=DELETE
--maintenance-policy=TERMINATE
gcloud compute instance-templates create {template_name}
--project={project_id} --machine-type=g2-standard-4
--provisioning-model=FLEX_START
--reservation-affinity=none
--max-run-duration=7d
--instance-termination-action=DELETE
--maintenance-policy=TERMINATE
--project={project_id} --machine-type=g2-standard-4
--provisioning-model=FLEX_START
--reservation-affinity=none
--max-run-duration=7d
--instance-termination-action=DELETE
--maintenance-policy=TERMINATE
Queue capacity on a MIG configured per bullet 2 (--zone={zone} or --region={region}):
Queue capacity on a MIG configured per bullet 2 (--zone={zone} or --region={region}):
gcloud compute instance-groups managed resize-requests create {mig_name}
--project={project_id} --region={region}
--resize-request={request_name} --resize-by={vm_count}
--project={project_id} --region={region}
--resize-request={request_name} --resize-by={vm_count}
---gcloud compute instance-groups managed resize-requests create {mig_name}
--project={project_id} --region={region}
--resize-request={request_name} --resize-by={vm_count}
--project={project_id} --region={region}
--resize-request={request_name} --resize-by={vm_count}
---Step 3: Long-Term Architectural Prevention
第 3 步:长期架构预防
To permanently eliminate single-zone stockouts for production services,
implement one or more of the following patterns:
要彻底消除生产服务的单可用区库存耗尽问题,应实施以下一种或多种模式:
A. Configure Regional MIGs with balanced
or any
Distribution Shape
balancedanyA. 使用 balanced
或 any
Distribution Shape 配置 Regional MIG
balancedanyA Regional MIG spreads instances across zones in a region. Configure its Target
Distribution Shape (see
Regional MIG distribution shape documentation):
- (
balanced): Prioritizes even zonal distribution, but automatically spills over to healthy sibling zones during a stockout.BALANCED - (
any): Provisions capacity in whichever zone has available hardware fastest, maximizing obtainability under regional pressure.ANY - (
any-single-zone): Colocates all VMs in one healthy zone with sufficient capacity (ideal for low-latency inter-node workloads).ANY_SINGLE_ZONE
Critical Operational Rule ():
--instance-redistribution-type- Regional MIGs default to (
even) +EVEN(proactive), which periodically deletes running VMs in over-represented zones and recreates them in under-represented or recovering zones to enforce strict zonal balance—disrupting stateful workloads and failing during stockouts.PROACTIVE - (
balanced),BALANCED(any), andANY(any-single-zone) all requireANY_SINGLE_ZONE. Compute Engine supports--instance-redistribution-type=noneredistribution only withproactiveand rejectsevenwithproactive,balanced, orany(see Set a target distribution shape).any-single-zone - Calliope Flag Case Rule: Unlike (uppercase),
instances bulk createrequires lowercase values forgcloud compute instance-groups managed update(--target-distribution-shape,any,any-single-zone,balanced) andeven(--instance-redistribution-type,none):proactive
sh
undefinedRegional MIG 将实例分布到区域内的多个可用区。请配置其 Target Distribution Shape(参见Regional MIG distribution shape 文档):
- (
balanced):优先实现均匀的可用区分布,但发生库存耗尽时会自动溢出到健康的相邻可用区。BALANCED - (
any):在最快获得可用硬件的可用区中预配容量,从而在区域资源紧张时最大化资源可获取性。ANY - (
any-single-zone):将所有 VM 共置在一个具有足够容量的健康可用区中,适用于对节点间延迟敏感的低延迟工作负载。ANY_SINGLE_ZONE
关键运维规则():
--instance-redistribution-type- Regional MIG 默认使用 (
even)+EVEN(proactive),会定期删除分布过多的可用区中正在运行的 VM,并将其重新创建在分布不足或正在恢复的可用区中,以维持严格的可用区均衡;这会中断有状态工作负载,并且可能在库存耗尽期间失败。PROACTIVE - (
balanced)、BALANCED(any)和ANY(any-single-zone)均要求设置ANY_SINGLE_ZONE。Compute Engine 仅支持将--instance-redistribution-type=noneredistribution 与proactive搭配使用,并会拒绝将even与proactive、balanced或any搭配使用(参见设置 Target Distribution Shape)。any-single-zone - Calliope 标志大小写规则:与 不同(使用大写),
instances bulk create要求gcloud compute instance-groups managed update(--target-distribution-shape、any、any-single-zone、balanced)和even(--instance-redistribution-type、none)使用小写值:proactive
sh
undefinedAllow spillover to healthy zones without proactively deleting running VMs:
Allow spillover to healthy zones without proactively deleting running VMs:
gcloud compute instance-groups managed update {mig_name}
--project={project_id} --region={region}
--instance-redistribution-type=none --target-distribution-shape=balanced
--project={project_id} --region={region}
--instance-redistribution-type=none --target-distribution-shape=balanced
gcloud compute instance-groups managed update {mig_name}
--project={project_id} --region={region}
--instance-redistribution-type=none --target-distribution-shape=balanced
--project={project_id} --region={region}
--instance-redistribution-type=none --target-distribution-shape=balanced
Or maximize obtainability in any zone (requires --instance-redistribution-type=none):
Or maximize obtainability in any zone (requires --instance-redistribution-type=none):
gcloud compute instance-groups managed update {mig_name}
--project={project_id} --region={region}
--instance-redistribution-type=none --target-distribution-shape=any
--project={project_id} --region={region}
--instance-redistribution-type=none --target-distribution-shape=any
undefinedgcloud compute instance-groups managed update {mig_name}
--project={project_id} --region={region}
--instance-redistribution-type=none --target-distribution-shape=any
--project={project_id} --region={region}
--instance-redistribution-type=none --target-distribution-shape=any
undefinedB. Enable MIG Instance Flexibility (Multi-Family Machine Type Fallback)
B. 启用 MIG Instance Flexibility(Multi-Family Machine Type Fallback)
Configure the MIG with Instance Flexibility ()
so it can fall back across equivalent machine families (for example, rank 1
, rank 2 or ) when the primary
family is exhausted (see
About instance flexibility
and
Add instance flexibility):
instanceFlexibilityPolicyc3-standard-8n2-standard-8n2d-standard-8- Constraints & Disk/Repair Rules:
- Supported only on Regional MIGs (not Zonal MIGs) with
set to
--target-distribution-shape,balanced, orany(defaultany-single-zoneis rejected) andeven. Configure with--instance-redistribution-type=none(gcloud beta compute instance-groups managed update/--instance-selection) or REST--remove-instance-selections.instanceFlexibilityPolicy.instanceSelections - Falling back across series with incompatible boot disk support (such as
Hyperdisk-only to Persistent-Disk-only
n4-standard-8) requires per-selection disk overrides vian2-standard-8(or--instance-flexibility-policy) rather than--flags-filealone.--instance-selection - Enable so MIG VM repairs can fall back to alternative machine types (and whenever using disk or CPU overrides).
--force-update-on-repair
- Supported only on Regional MIGs (not Zonal MIGs) with
sh
gcloud beta compute instance-groups managed update {mig_name} \
--project={project_id} --region={region} \
--target-distribution-shape=balanced \
--instance-redistribution-type=none \
--instance-selection="name=primary-tier,machine-type=c3-standard-8,rank=1" \
--instance-selection="name=fallback-tier,machine-type=n2-standard-8,rank=2"为 MIG 配置 Instance Flexibility(),使其在主要 machine family 耗尽时,可以回退到等效 machine family(例如优先级 1 的 ,优先级 2 的 或 )(参见关于 Instance Flexibility和添加 Instance Flexibility):
instanceFlexibilityPolicyc3-standard-8n2-standard-8n2d-standard-8- 约束与磁盘/修复规则:
- 仅支持在 Regional MIG 上使用(不支持 Zonal MIG),并且必须将 设置为
--target-distribution-shape、balanced或any(默认的any-single-zone会被拒绝),同时设置even。通过--instance-redistribution-type=none(gcloud beta compute instance-groups managed update/--instance-selection)或 REST 中的--remove-instance-selections进行配置。instanceFlexibilityPolicy.instanceSelections - 如果在不同 series 之间回退时存在不兼容的 boot disk 支持(例如仅支持 Hyperdisk 的 回退到仅支持 Persistent Disk 的
n4-standard-8),则必须通过n2-standard-8(或--instance-flexibility-policy)为每个 selection 设置磁盘覆盖项,不能只使用--flags-file。--instance-selection - 启用 ,使 MIG VM 修复操作能够回退到替代 machine type;使用磁盘或 CPU 覆盖项时也应启用此选项。
--force-update-on-repair
- 仅支持在 Regional MIG 上使用(不支持 Zonal MIG),并且必须将
sh
gcloud beta compute instance-groups managed update {mig_name} \
--project={project_id} --region={region} \
--target-distribution-shape=balanced \
--instance-redistribution-type=none \
--instance-selection="name=primary-tier,machine-type=c3-standard-8,rank=1" \
--instance-selection="name=fallback-tier,machine-type=n2-standard-8,rank=2"C. Purchase On-Demand Capacity Reservations
C. 购买 On-Demand Capacity Reservations
For mission-critical production workloads requiring 100% guaranteed zonal
capacity assurance, create an On-Demand Zonal Reservation (see
About reservations of zonal resources
and
Create a reservation for a single project):
- Operational & IAM Rules:
- Requires (or custom
roles/compute.admin) and succeeds only when physical capacity is currently available incompute.reservations.*—if{zone}is actively stocked out, create the reservation in an available sibling zone or wait until capacity replenishes.{zone} - For N1 with GPUs or Local SSDs, and
--acceleratorflags must exactly match the consuming VM's properties.--local-ssd
- Requires
sh
gcloud compute reservations create {reservation_name} \
--project={project_id} --zone={zone} \
--vm-count={vm_count} --machine-type={machine_type}- 运维与 IAM 规则:
- 需要 (或自定义
roles/compute.admin),并且仅当compute.reservations.*当前存在物理容量时才能成功;如果{zone}正处于库存耗尽状态,请在有容量的相邻可用区创建预留,或等待容量补充。{zone} - 对于带 GPU 的 N1 或使用 Local SSD 的实例,和
--accelerator标志必须与使用该预留的 VM 属性完全一致。--local-ssd
- 需要
sh
gcloud compute reservations create {reservation_name} \
--project={project_id} --zone={zone} \
--vm-count={vm_count} --machine-type={machine_type}Safe Rollback, Teardown & Cost Governance
安全回滚、拆除与成本治理
- Restore Regional MIG Distribution Settings: To roll back from to opportunistic zonal balance (
anyrequiresbalancedand converges on resizes), or to return a stateless MIG to strict--instance-redistribution-type=none+evendistribution (after manually rebalancing if zone counts differ by more than 1 VM):proactivesh# Option A: Converge opportunistically on resizes without deleting VMs: gcloud compute instance-groups managed update {mig_name} \ --project={project_id} --region={region} \ --target-distribution-shape=balanced --instance-redistribution-type=none # Option B: Stateless strict even + proactive (rebalance manually first if >1 VM diff): gcloud compute instance-groups managed update {mig_name} \ --project={project_id} --region={region} \ --target-distribution-shape=even --instance-redistribution-type=none gcloud compute instance-groups managed update {mig_name} \ --project={project_id} --region={region} \ --instance-redistribution-type=proactive - Remove MIG Instance Selections: To roll back fallback selections:
sh
gcloud beta compute instance-groups managed update {mig_name} \ --project={project_id} --region={region} \ --remove-instance-selections=fallback-tier - Delete Unused Capacity Reservations & Temporary VMs:
- Cost Warning: On-Demand Capacity Reservations are billed at standard Compute Engine rates from creation whether or not VMs run inside them (and count toward regional quota). Consult Google Cloud pricing and Compute Engine pricing for current rates rather than quoting hardcoded prices.
- Omit on manual destructive commands so interactive confirmation prevents accidental deletion in production:
--quietshgcloud compute reservations delete {reservation_name} \ --project={project_id} --zone={zone} gcloud compute instances delete {vm_name} \ --project={project_id} --zone={zone}
- 恢复 Regional MIG Distribution 设置:如需从 回滚到机会式可用区均衡(
any要求balanced,并通过 resize 逐步收敛),或将无状态 MIG 恢复为严格的--instance-redistribution-type=none+even分布(如果各可用区的 VM 数量相差超过 1 个,必须先执行手动重新平衡):proactivesh# Option A: Converge opportunistically on resizes without deleting VMs: gcloud compute instance-groups managed update {mig_name} \ --project={project_id} --region={region} \ --target-distribution-shape=balanced --instance-redistribution-type=none # Option B: Stateless strict even + proactive (rebalance manually first if >1 VM diff): gcloud compute instance-groups managed update {mig_name} \ --project={project_id} --region={region} \ --target-distribution-shape=even --instance-redistribution-type=none gcloud compute instance-groups managed update {mig_name} \ --project={project_id} --region={region} \ --instance-redistribution-type=proactive - 移除 MIG Instance Selections:如需回退 fallback selections:
sh
gcloud beta compute instance-groups managed update {mig_name} \ --project={project_id} --region={region} \ --remove-instance-selections=fallback-tier - 删除未使用的 Capacity Reservations 和临时 VM:
- 成本警告:On-Demand Capacity Reservations 从创建之时起即按标准 Compute Engine 费率计费,无论其内部是否运行 VM(并且会占用区域配额)。请查阅 Google Cloud pricing 和 Compute Engine pricing 获取当前费率,不要引用硬编码价格。
- 手动执行破坏性命令时不要省略 ,让交互式确认机制防止生产环境中的误删除:
--quietshgcloud compute reservations delete {reservation_name} \ --project={project_id} --zone={zone} gcloud compute instances delete {vm_name} \ --project={project_id} --zone={zone}