compute-resource-availability-troubleshooting

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Compute Engine Resource Availability and Stockout Troubleshooting

Compute Engine 资源可用性与容量库存耗尽故障排查

MCP Tools:
list_machine_types
,
list_accelerator_types
,
get_zone_operation
,
list_instance_group_managers
,
list_managed_instances
,
list_reservations
,
get_reservation_details
This skill diagnoses and resolves Google Compute Engine resource availability errors (capacity stockouts). It provides immediate tactical workarounds to get blocked workloads running and long-term architectural mitigations using Regional Managed Instance Group (MIG) distribution shapes, Instance Flexibility, Dynamic Workload Scheduler (DWS) Flex-Start, and On-Demand Capacity Reservations.
MCP 工具:
list_machine_types
、
list_accelerator_types
、
get_zone_operation
、
list_instance_group_managers
、
list_managed_instances
、
list_reservations
、
get_reservation_details
此 skill 用于诊断并解决 Google Compute Engine 资源可用性错误(容量库存耗尽)。它既提供可立即采用的临时变通方案以让受阻工作负载恢复运行,也提供基于 Regional Managed Instance Group(MIG)distribution shapes、Instance Flexibility、Dynamic Workload Scheduler(DWS)Flex-Start 和 On-Demand Capacity Reservations 的长期架构缓解措施。

Usage Attribution

使用归属

MUST READ: Before executing any
gcloud
, REST API, or MCP command, you MUST read and follow the instructions in Usage Attribution.

必须阅读:在执行任何
gcloud
、REST API 或 MCP 命令之前,您必须阅读并遵循使用归属中的说明。

Prerequisites & Environment Setup

前置条件与环境配置

Any host, project, IAM, or Compute Engine resource modifications (installing
gcloud
, configuring MCP, enabling APIs or Log Analytics, creating instances or reservations, updating MIGs, or deleting resources) must be explicitly approved by the user before execution.
  1. Google Cloud SDK: Verify
    gcloud
    is installed (
    gcloud --version
    ). If missing or outdated, ask the user for approval before installing via Install the Google Cloud SDK.
  2. Compute Engine MCP Server & Capability Boundary: When the Compute Engine MCP server is configured (see Use the Compute Engine MCP server; requires
    roles/mcp.toolUser
    ), prefer its read-only discovery tools before falling back to
    gcloud
    :
    • list_machine_types
      and
      list_accelerator_types
      to check whether a machine shape or GPU accelerator is offered in a target zone.
    • get_zone_operation
      to inspect a failed zonal operation's
      error.errors
      array (
      ZONE_RESOURCE_POOL_EXHAUSTED
      ,
      ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
      , or
      QUOTA_EXCEEDED
      ).
    • list_instance_group_managers
      and
      list_managed_instances
      to inspect zonal MIGs and per-instance
      currentAction
      /
      lastAttempt.errors
      (note: all Compute Engine MCP MIG tools—
      list_instance_group_managers
      ,
      list_managed_instances
      , and
      get_instance_group_manager_basic_info
      —only accept a
      zone
      parameter and cannot inspect regional MIGs; use
      gcloud
      with
      --region={region}
      or REST for regional MIGs).
    • list_reservations
      and
      get_reservation_details
      to inspect existing zonal reservation capacity (
      specificReservation.count
      vs.
      specificReservation.inUseCount
      ).
    • MCP Limitations (Use
      gcloud
      or REST Instead)
      : MCP
      create_instance
      does not expose
      provisioning_model
      (
      FLEX_START
      /
      SPOT
      ),
      max_run_duration
      ,
      instance_termination_action
      , or
      reservation_affinity
      . MCP also lacks tools for Cloud Logging queries (
      gcloud logging read
      ), Spot Capacity Advisor (
      gcloud beta compute advice capacity
      ), Bulk Insert (
      gcloud compute instances bulk create
      ), MIG distribution updates (
      gcloud compute instance-groups managed update
      ), and reservation creation or deletion (
      gcloud compute reservations create
      /
      delete
      ). Always use
      gcloud
      or the Compute Engine REST API for those operations.
  3. Authentication: Local CLI (
    gcloud auth login
    ), Application Default Credentials (
    gcloud auth application-default login
    ), or headless/CI-CD via
    GOOGLE_APPLICATION_CREDENTIALS
    or Workload Identity Federation.
  4. Project, Region, Billing & API Enablement:
    sh
    gcloud config set project {project_id}
    gcloud config set compute/region {region}
    gcloud config set compute/zone {zone}
    gcloud services enable compute.googleapis.com logging.googleapis.com
    Note: An active billing account attachment is required (even for Free Tier eligible resources such as
    e2-micro
    ). Never hardcode regions or zones; confirm
    {project_id}
    ,
    {region}
    , and
    {zone}
    (defaulting to a well-supported region such as
    us-central1
    /
    us-central1-a
    if unspecified), and consult Google Cloud Locations to verify regional and zonal availability (preview features and specialized GPUs/TPUs are restricted to specific locations).
  5. Required IAM Roles:
    • Read-Only Diagnostics & Capacity Advice:
      roles/compute.viewer
      and
      roles/logging.viewer
      .
    • Log Analytics SQL (
      _Default._AllLogs
      )
      :
      roles/logging.viewAccessor
      (plus
      roles/bigquery.jobUser
      and
      roles/bigquery.dataViewer
      when querying via a linked BigQuery dataset).
    • Instance & MIG Remediation:
      roles/compute.instanceAdmin.v1
      (plus
      roles/iam.serviceAccountUser
      when attaching a service account to VMs).
    • Reservation Creation & Teardown:
      roles/compute.admin
      (or a custom role granting
      compute.reservations.create
      ,
      compute.reservations.update
      , and
      compute.reservations.delete
      , which are not included in
      roles/compute.instanceAdmin.v1
      ).
    • MCP Tool Invocation:
      roles/mcp.toolUser
      .

任何针对主机、项目、IAM 或 Compute Engine 资源的修改(安装
gcloud
、配置 MCP、启用 API 或 Log Analytics、创建实例或预留、更新 MIG 或删除资源),执行前均必须获得用户明确批准。
  1. Google Cloud SDK:确认已安装
    gcloud
    (
    gcloud --version
    )。如果缺失或版本过旧,应请求用户批准后再通过安装 Google Cloud SDK进行安装。
  2. Compute Engine MCP Server 与能力边界:配置 Compute Engine MCP server 后(参见使用 Compute Engine MCP server;需要
    roles/mcp.toolUser
    ),应优先使用其只读发现工具,再回退到
    gcloud
    :
    • 使用
      list_machine_types
      和
      list_accelerator_types
      检查目标可用区是否提供特定 machine shape 或 GPU accelerator。
    • 使用
      get_zone_operation
      检查失败的可用区级操作的
      error.errors
      数组(
      ZONE_RESOURCE_POOL_EXHAUSTED
      、
      ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
      或
      QUOTA_EXCEEDED
      )。
    • 使用
      list_instance_group_managers
      和
      list_managed_instances
      检查可用区级 MIG,以及各实例的
      currentAction
      /
      lastAttempt.errors
      (注意:所有 Compute Engine MCP MIG 工具——
      list_instance_group_managers
      、
      list_managed_instances
      和
      get_instance_group_manager_basic_info
      ——仅接受
      zone
      参数,无法检查区域级 MIG;请使用带
      --region={region}
      的
      gcloud
      或 REST)。
    • 使用
      list_reservations
      和
      get_reservation_details
      检查现有可用区级预留容量(
      specificReservation.count
      与
      specificReservation.inUseCount
      )。
    • MCP 限制(改用
      gcloud
      或 REST)
      :MCP
      create_instance
      未暴露
      provisioning_model
      (
      FLEX_START
      /
      SPOT
      )、
      max_run_duration
      、
      instance_termination_action
      或
      reservation_affinity
      。MCP 也不支持 Cloud Logging 查询(
      gcloud logging read
      )、Spot Capacity Advisor(
      gcloud beta compute advice capacity
      )、Bulk Insert(
      gcloud compute instances bulk create
      )、MIG distribution 更新(
      gcloud compute instance-groups managed update
      ),以及预留创建或删除(
      gcloud compute reservations create
      /
      delete
      )。此类操作始终应使用
      gcloud
      或 Compute Engine REST API。
  3. 身份验证:本地 CLI(
    gcloud auth login
    )、Application Default Credentials(
    gcloud auth application-default login
    ),或通过
    GOOGLE_APPLICATION_CREDENTIALS
    、Workload Identity Federation 在无头/CI-CD 环境中进行身份验证。
  4. 项目、区域、计费与 API 启用:
    sh
    gcloud config set project {project_id}
    gcloud config set compute/region {region}
    gcloud config set compute/zone {zone}
    gcloud services enable compute.googleapis.com logging.googleapis.com
    注意:必须绑定有效的结算账户,即使资源符合 Free Tier 条件(如
    e2-micro
    )也不例外。切勿硬编码 region 或 zone;应确认
    {project_id}
    、
    {region}
    和
    {zone}
    (如果未指定,则默认使用
    us-central1
    /
    us-central1-a
    等支持范围较广的区域),并查阅 Google Cloud Locations 以核实区域和可用区可用性(preview features 以及专用 GPU/TPU 仅在特定 location 提供)。
  5. 必需的 IAM Roles:
    • 只读诊断与容量建议:
      roles/compute.viewer
      和
      roles/logging.viewer
      。
    • Log Analytics SQL(
      _Default._AllLogs
      )
      :
      roles/logging.viewAccessor
      (通过关联的 BigQuery dataset 查询时,还需要
      roles/bigquery.jobUser
      和
      roles/bigquery.dataViewer
      )。
    • 实例与 MIG 修复:
      roles/compute.instanceAdmin.v1
      (向 VM 附加 service account 时还需要
      roles/iam.serviceAccountUser
      )。
    • 预留创建与拆除:
      roles/compute.admin
      (或授予
      compute.reservations.create
      、
      compute.reservations.update
      和
      compute.reservations.delete
      的自定义 role;这些权限不包含在
      roles/compute.instanceAdmin.v1
      中)。
    • MCP 工具调用:
      roles/mcp.toolUser
      。

Step 1: Verify Stockout Scope & Distinguish from Quota Errors

第 1 步:验证库存耗尽范围并与配额错误区分

A. Core Domain Rules (Explain to User)

A. 核心领域规则(向用户说明)

Before taking action, clarify these three foundational Compute Engine behaviors (see Troubleshooting resource availability errors):
  1. Stockout != Quota:
    ZONE_RESOURCE_POOL_EXHAUSTED
    (and
    ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
    ) means physical hardware capacity for that specific resource pool in that specific zone is temporarily full. Increasing project quota (IAM & Admin > Quotas & System Limits) will not resolve a physical stockout. Conversely,
    QUOTA_EXCEEDED
    means the project's administrative quota ceiling has been reached regardless of physical hardware availability. When returned,
    ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
    also includes
    errorInfo.reason
    (such as
    configuration_availability
    for unsupported hardware combinations that will not succeed on retry) and
    errorInfo.metadata.zonesAvailable
    listing sibling zones with capacity.
  2. Scoped to Specific Shape & Zone: A stockout applies only to the exact machine shape or accelerator (for example,
    g2-standard-4
    or
    c4-standard-32
    ) in that zone (for example,
    us-central1-a
    ). Sibling zones (
    us-central1-b
    ,
    us-central1-c
    ,
    us-central1-f
    ) or smaller shapes (
    c4-standard-16
    ) frequently have ample capacity.
  3. Existing Running VMs Are Safe: Resource availability errors affect only new capacity allocations (creating new VMs, starting stopped
    TERMINATED
    VMs, changing machine types, or MIG scale-out). Currently running
    RUNNING
    VMs in the zone are unaffected.
采取任何操作之前,应先说明以下三个 Compute Engine 基础行为(参见资源可用性错误故障排查):
  1. 库存耗尽不等于配额问题:
    ZONE_RESOURCE_POOL_EXHAUSTED
    (以及
    ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
    )表示特定可用区中特定资源池的物理硬件容量暂时已满。提高项目配额(IAM & Admin > Quotas & System Limits)无法解决物理库存耗尽问题。相反,
    QUOTA_EXCEEDED
    表示无论物理硬件是否可用,项目的管理配额上限均已达到。返回
    ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
    时,还会包含
    errorInfo.reason
    (例如
    configuration_availability
    ,表示不支持的硬件组合,重试也不会成功),以及
    errorInfo.metadata.zonesAvailable
    ,其中列出了有容量的相邻可用区。
  2. 限定于特定规格与可用区:库存耗尽仅适用于该可用区中的确切 machine shape 或 accelerator(例如
    g2-standard-4
    或
    c4-standard-32
    ),例如
    us-central1-a
    。相邻可用区(
    us-central1-b
    、
    us-central1-c
    、
    us-central1-f
    )或更小的规格(
    c4-standard-16
    )通常仍有充足容量。
  3. 现有运行中的 VM 不受影响:资源可用性错误仅影响新容量分配(创建新 VM、启动已停止的
    TERMINATED
    VM、更改 machine type 或 MIG 扩容)。该可用区中当前处于
    RUNNING
    状态的 VM 不受影响。

B. Query Cloud Audit Logs for Resource Availability vs. Quota Failures

B. 查询 Cloud Audit Logs,区分资源可用性问题与配额失败

Run
gcloud logging read
with an explicit
--freshness
window (
gcloud logging read
defaults to
1d
, so multi-day investigations require
--freshness=7d
or
--freshness=14d
). Inspect
protoPayload.response.error.errors[0].code
(not
.reason
) to distinguish
ZONE_RESOURCE_POOL_EXHAUSTED
/
ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
from
QUOTA_EXCEEDED
:
sh
gcloud logging read \
    'resource.type="gce_instance" AND log_id("cloudaudit.googleapis.com/activity") AND (protoPayload.status.message:("ZONE_RESOURCE_POOL_EXHAUSTED" OR "does not have enough resources available" OR "resource pool exhausted" OR "does not exist in zone" OR "QUOTA_EXCEEDED" OR "Quota") OR protoPayload.response.error.errors.code:("ZONE_RESOURCE_POOL_EXHAUSTED" OR "ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS" OR "QUOTA_EXCEEDED"))' \
    --project={project_id} --freshness=7d --limit=20 \
    --format="table(timestamp, resource.labels.zone, protoPayload.resourceName, protoPayload.response.error.errors[0].code, protoPayload.status.message)"
  • If
    code
    is
    QUOTA_EXCEEDED
    (Always Include Regional Quota Inspection)
    : Do not treat this as a physical stockout. Always provide
    gcloud compute regions describe
    to inspect regional quota limits and usage, and direct the user to IAM & Admin > Quotas & System Limits (or switch to a machine family with quota headroom):
    sh
    gcloud compute regions describe {region} --project={project_id} \
        --format="table(quotas:format='table(metric,limit,usage)')"
  • If a MIG is failing to scale out: Also inspect MIG provisioning errors directly (via Compute Engine MCP
    list_managed_instances
    for zonal MIGs, or
    gcloud
    ):
    sh
    # Zonal MIG (--zone={zone}) vs. Regional MIG (--region={region}):
    gcloud compute instance-groups managed list-errors {mig_name} \
        --project={project_id} --zone={zone}
    gcloud compute instance-groups managed list-errors {mig_name} \
        --project={project_id} --region={region}
使用明确的
--freshness
时间窗口运行
gcloud logging read
(
gcloud logging read
默认使用
1d
,因此持续多天的调查需要
--freshness=7d
或
--freshness=14d
)。检查
protoPayload.response.error.errors[0].code
(而不是
.reason
),以区分
ZONE_RESOURCE_POOL_EXHAUSTED
/
ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
与
QUOTA_EXCEEDED
:
sh
gcloud logging read \
    'resource.type="gce_instance" AND log_id("cloudaudit.googleapis.com/activity") AND (protoPayload.status.message:("ZONE_RESOURCE_POOL_EXHAUSTED" OR "does not have enough resources available" OR "resource pool exhausted" OR "does not exist in zone" OR "QUOTA_EXCEEDED" OR "Quota") OR protoPayload.response.error.errors.code:("ZONE_RESOURCE_POOL_EXHAUSTED" OR "ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS" OR "QUOTA_EXCEEDED"))' \
    --project={project_id} --freshness=7d --limit=20 \
    --format="table(timestamp, resource.labels.zone, protoPayload.resourceName, protoPayload.response.error.errors[0].code, protoPayload.status.message)"
  • 如果
    code
    为
    QUOTA_EXCEEDED
    (始终检查区域配额)
    : 不要将其视为物理库存耗尽。始终提供
    gcloud compute regions describe
    以检查区域配额上限和使用量,并引导用户前往 IAM & Admin > Quotas & System Limits(或切换到具有配额余量的 machine family):
    sh
    gcloud compute regions describe {region} --project={project_id} \
        --format="table(quotas:format='table(metric,limit,usage)')"
  • 如果 MIG 扩容失败:还应直接检查 MIG 配置错误(对于可用区级 MIG,可使用 Compute Engine MCP
    list_managed_instances
    ;也可使用
    gcloud
    ):
    sh
    # Zonal MIG (--zone={zone}) vs. Regional MIG (--region={region}):
    gcloud compute instance-groups managed list-errors {mig_name} \
        --project={project_id} --zone={zone}
    gcloud compute instance-groups managed list-errors {mig_name} \
        --project={project_id} --region={region}

C. Analyze Historical Stockout Frequency with Log Analytics SQL

C. 使用 Log Analytics SQL 分析历史库存耗尽频率

Because failed
cloudaudit.googleapis.com/activity
entries have severity
ERROR
(
severity_number >= 500
), Cloud Logging routes them to the
_Default
log bucket
(
{project_id}.global._Default._AllLogs
), whereas
_Required._AllLogs
only stores Admin Activity logs below
ERROR
and System Event logs.
  • Prerequisite: Upgrade
    _Default
    to Log Analytics (see Query and view logs in Log Analytics):
    sh
    gcloud logging buckets update _Default \
        --project={project_id} --location=global --enable-analytics
  • JSON Column Extraction Rule: In
    _AllLogs
    ,
    resource.labels
    and
    proto_payload.audit_log.request
    have BigQuery type
    JSON
    . Never reference
    resource.labels.zone
    directly in
    GROUP BY
    —always extract scalar
    STRING
    values with
    JSON_VALUE(...)
    :
    sql
    SELECT
      JSON_VALUE(resource.labels.zone) AS zone,
      REGEXP_EXTRACT(JSON_VALUE(proto_payload.audit_log.request.machineType), r'([^/]+)$') AS machine_type,
      proto_payload.audit_log.status.message AS status_message,
      COUNT(*) AS stockout_count,
      MIN(timestamp) AS first_seen,
      MAX(timestamp) AS last_seen
    FROM `{project_id}.global._Default._AllLogs`
    WHERE
      timestamp >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 14 DAY)
      AND log_id = 'cloudaudit.googleapis.com/activity'
      AND resource.type = 'gce_instance'
      AND (
        proto_payload.audit_log.status.message LIKE '%ZONE_RESOURCE_POOL_EXHAUSTED%'
        OR proto_payload.audit_log.status.message LIKE '%does not have enough resources available%'
        OR TO_JSON_STRING(proto_payload.audit_log.response.error.errors) LIKE '%ZONE_RESOURCE_POOL_EXHAUSTED%'
      )
    GROUP BY zone, machine_type, status_message
    ORDER BY stockout_count DESC;

由于失败的
cloudaudit.googleapis.com/activity
条目严重级别为
ERROR
(
severity_number >= 500
),Cloud Logging 会将其路由到
_Default
log bucket
(
{project_id}.global._Default._AllLogs
),而
_Required._AllLogs
仅存储严重级别低于
ERROR
的 Admin Activity 日志和 System Event 日志。
  • 前置条件:将
    _Default
    升级为 Log Analytics(参见在 Log Analytics 中查询和查看日志):
    sh
    gcloud logging buckets update _Default \
        --project={project_id} --location=global --enable-analytics
  • JSON 列提取规则:在
    _AllLogs
    中,
    resource.labels
    和
    proto_payload.audit_log.request
    的 BigQuery 类型为
    JSON
    。切勿在
    GROUP BY
    中直接引用
    resource.labels.zone
    ;必须使用
    JSON_VALUE(...)
    提取标量
    STRING
    值:
    sql
    SELECT
      JSON_VALUE(resource.labels.zone) AS zone,
      REGEXP_EXTRACT(JSON_VALUE(proto_payload.audit_log.request.machineType), r'([^/]+)$') AS machine_type,
      proto_payload.audit_log.status.message AS status_message,
      COUNT(*) AS stockout_count,
      MIN(timestamp) AS first_seen,
      MAX(timestamp) AS last_seen
    FROM `{project_id}.global._Default._AllLogs`
    WHERE
      timestamp >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 14 DAY)
      AND log_id = 'cloudaudit.googleapis.com/activity'
      AND resource.type = 'gce_instance'
      AND (
        proto_payload.audit_log.status.message LIKE '%ZONE_RESOURCE_POOL_EXHAUSTED%'
        OR proto_payload.audit_log.status.message LIKE '%does not have enough resources available%'
        OR TO_JSON_STRING(proto_payload.audit_log.response.error.errors) LIKE '%ZONE_RESOURCE_POOL_EXHAUSTED%'
      )
    GROUP BY zone, machine_type, status_message
    ORDER BY stockout_count DESC;

Step 2: Immediate Tactical Mitigations (Unblock Workload Now)

第 2 步:立即采取临时缓解措施(解除当前工作负载阻塞)

Execute these five steps in priority order to provision the blocked workload immediately:
按优先级执行以下五个步骤,立即为受阻工作负载提供资源:

1. Discover Sibling Zones Supporting the Machine Shape or GPU

1. 发现支持目标 machine shape 或 GPU 的相邻可用区

If a zone returns
ZONE_RESOURCE_POOL_EXHAUSTED
or
Machine type with name '...' does not exist in zone '...'
, check which sibling zones in
{region}
offer the target machine shape and (if applicable) GPU accelerator using Compute Engine MCP
list_machine_types
/
list_accelerator_types
or
gcloud
:
sh
undefined
如果可用区返回
ZONE_RESOURCE_POOL_EXHAUSTED
或
Machine type with name '...' does not exist in zone '...'
,请使用 Compute Engine MCP
list_machine_types
/
list_accelerator_types
或
gcloud
,检查
{region}
中哪些相邻可用区提供目标 machine shape,以及在适用时提供 GPU accelerator:
sh
undefined

List all zones in {region} that offer {machine_type}:

List all zones in {region} that offer {machine_type}:

gcloud compute machine-types list --project={project_id}
--filter="name={machine_type} AND zone ~ ^{region}-"
--format="table(name, zone, guestCpus, memoryMb)"
gcloud compute machine-types list --project={project_id}
--filter="name={machine_type} AND zone ~ ^{region}-"
--format="table(name, zone, guestCpus, memoryMb)"

For GPU workloads (especially N1 with attached GPUs), verify accelerator zones:

For GPU workloads (especially N1 with attached GPUs), verify accelerator zones:

gcloud compute accelerator-types list --project={project_id}
--filter="name={accelerator_type} AND zone ~ ^{region}-"
--format="table(name, zone, maximumCardsPerInstance)"
undefined
gcloud compute accelerator-types list --project={project_id}
--filter="name={accelerator_type} AND zone ~ ^{region}-"
--format="table(name, zone, maximumCardsPerInstance)"
undefined

2. Use Bulk Insert (
bulk create
) to Pick an Available Zone

2. 使用 Bulk Insert(
bulk create
)选择有容量的可用区

Instead of guessing and retrying individual zones manually, use
gcloud compute instances bulk create
with
--region={region}
and
--target-distribution-shape=ANY_SINGLE_ZONE
(for colocated VMs or
--count=1
) or
--target-distribution-shape=BALANCED
(to spread across available zones). Compute Engine automatically selects zone(s) in
{region}
with capacity (see Create VMs in bulk):
sh
undefined
不要手动猜测并逐一重试可用区,而应使用
gcloud compute instances bulk create
,并设置
--region={region}
和
--target-distribution-shape=ANY_SINGLE_ZONE
(用于共置 VM 或
--count=1
),或设置
--target-distribution-shape=BALANCED
(用于分散到多个可用区)。Compute Engine 会自动选择
{region}
中具有容量的可用区(参见批量创建 VM):
sh
undefined

On instances bulk create, --target-distribution-shape is UPPERCASE (ANY, ANY_SINGLE_ZONE, BALANCED):

On instances bulk create, --target-distribution-shape is UPPERCASE (ANY, ANY_SINGLE_ZONE, BALANCED):

gcloud compute instances bulk create
--project={project_id} --region={region}
--name-pattern="{vm_name}-#" --count={vm_count}
--target-distribution-shape=ANY_SINGLE_ZONE
--machine-type={machine_type}
undefined
gcloud compute instances bulk create
--project={project_id} --region={region}
--name-pattern="{vm_name}-#" --count={vm_count}
--target-distribution-shape=ANY_SINGLE_ZONE
--machine-type={machine_type}
undefined

3. Check Spot VM Obtainability via Spot Capacity Advisor (
advice capacity
)

3. 通过 Spot Capacity Advisor(
advice capacity
)检查 Spot VM 可获取性

When provisioning Spot VMs, use
gcloud beta compute advice capacity
(the Spot Capacity Advisor API) to compare real-time Spot obtainability scores and estimated uptime (
estimatedUptime
) across zones and machine types in
{region}
before creating instances (use
gcloud beta compute advice capacity-history
for 30-day historical preemption rates; see View the availability of Spot VMs):
  • Critical Scope & Flag Rules:
    1. gcloud beta compute advice capacity
      supports only Spot VMs (
      --provisioning-model=SPOT
      , the only supported value; it cannot query
      STANDARD
      on-demand capacity).
    2. There is no
      --machine-type
      flag
      on
      advice capacity
      —specify machine types with
      --instance-selection-machine-types={machine_type_1},{machine_type_2}
      or
      --instance-selection="name={selection_name},machine-type={machine_type}"
      .
    3. On
      advice capacity
      ,
      --target-distribution-shape
      choices are UPPERCASE (
      ANY
      ,
      ANY_SINGLE_ZONE
      , or
      BALANCED
      ). Always include
      --provisioning-model=SPOT
      ,
      --instance-selection-machine-types
      , and uppercase
      --target-distribution-shape=ANY
      (or
      ANY_SINGLE_ZONE
      /
      BALANCED
      ):
sh
gcloud beta compute advice capacity \
    --project={project_id} --region={region} \
    --provisioning-model=SPOT --size={vm_count} \
    --instance-selection-machine-types={machine_type_1},{machine_type_2} \
    --target-distribution-shape=ANY
配置 Spot VMs 时,使用
gcloud beta compute advice capacity
(Spot Capacity Advisor API),在创建实例之前比较
{region}
中各可用区和 machine type 的实时 Spot 可获取性评分及预计正常运行时间(
estimatedUptime
);使用
gcloud beta compute advice capacity-history
查看过去 30 天的抢占率历史(参见查看 Spot VM 的可用性):
  • 关键范围与标志规则:
    1. gcloud beta compute advice capacity
      仅支持 Spot VMs(
      --provisioning-model=SPOT
      是唯一受支持的值;无法查询
      STANDARD
      按需容量)。
    2. advice capacity
      没有
      --machine-type
      标志;应通过
      --instance-selection-machine-types={machine_type_1},{machine_type_2}
      或
      --instance-selection="name={selection_name},machine-type={machine_type}"
      指定 machine type。
    3. 在
      advice capacity
      中,
      --target-distribution-shape
      的可选值为大写(
      ANY
      、
      ANY_SINGLE_ZONE
      或
      BALANCED
      )。始终包含
      --provisioning-model=SPOT
      、
      --instance-selection-machine-types
      和大写的
      --target-distribution-shape=ANY
      (或
      ANY_SINGLE_ZONE
      /
      BALANCED
      ):
sh
gcloud beta compute advice capacity \
    --project={project_id} --region={region} \
    --provisioning-model=SPOT --size={vm_count} \
    --instance-selection-machine-types={machine_type_1},{machine_type_2} \
    --target-distribution-shape=ANY

4. Right-Size Large Shapes (> 32 vCPUs)

4. 优化大型规格(> 32 vCPUs)

Physical host fragmentation makes large machine shapes (such as
*-standard-64
,
*-standard-96
, or
*-standard-128
) significantly scarcer than smaller shapes. If the workload can scale horizontally, split a single 64-vCPU request into two 32-vCPU VMs or four 16-vCPU VMs.
物理主机碎片化会使大型 machine shape(例如
*-standard-64
、
*-standard-96
或
*-standard-128
)比小型 shape 明显更难获得。如果工作负载可以横向扩展,可将单个 64-vCPU 请求拆分为两个 32-vCPU VM,或四个 16-vCPU VM。

5. Use Dynamic Workload Scheduler (DWS) Flex-Start for Scarce GPUs

5. 对稀缺 GPU 使用 Dynamic Workload Scheduler(DWS)Flex-Start

When scarce GPU shapes (
a4-highgpu-8g
,
a3-highgpu-8g
,
a2-highgpu-1g
, or
g2-standard-4
) fail immediate creation with
ZONE_RESOURCE_POOL_EXHAUSTED
, use Dynamic Workload Scheduler (DWS) Flex-Start to queue and automatically provision capacity when hardware frees up (see About Dynamic Workload Scheduler and Create a standalone Flex-start VM):
  • Mandatory CLI Rules for DWS Flex-Start Standalone VMs:
    1. Machine Family Accelerator & Maintenance Rule: For
      G2
      (
      nvidia-l4
      ),
      G4
      ,
      A2
      (
      nvidia-tesla-a100
      ),
      A3
      , and
      A4
      machine series, GPUs are automatically attached by the machine type (omit
      --accelerator
      unless attaching GPUs to
      N1
      or using vWS). Standalone Flex-Start GPU creation requires
      --reservation-affinity=none
      ,
      --max-run-duration
      (
      10m
      to
      7d
      ),
      --instance-termination-action=DELETE
      (or
      STOP
      ),
      --maintenance-policy=TERMINATE
      , and
      --request-valid-for-duration
      (
      90s
      to
      2h
      to queue in a zone, or
      0s
      for immediate-only allocation).
    2. For multi-instance GPU jobs or longer/multi-zone queueing, use a MIG with a Flex-Start instance template and resize requests (see Create resize requests in a MIG). The MIG requires autoscaling disabled,
      --default-action-on-vm-failure=do-nothing
      , and (on a Regional MIG)
      --target-distribution-shape=any-single-zone
      with
      --instance-redistribution-type=none
      .
sh
undefined
当稀缺 GPU shape(
a4-highgpu-8g
、
a3-highgpu-8g
、
a2-highgpu-1g
或
g2-standard-4
)因
ZONE_RESOURCE_POOL_EXHAUSTED
而无法立即创建时,使用 Dynamic Workload Scheduler(DWS)Flex-Start 对容量进行排队,并在硬件释放后自动预配资源(参见关于 Dynamic Workload Scheduler和创建独立 Flex-Start VM):
  • DWS Flex-Start 独立 VM 的强制 CLI 规则:
    1. Machine Family Accelerator 与维护规则:对于
      G2
      (
      nvidia-l4
      )、
      G4
      、
      A2
      (
      nvidia-tesla-a100
      )、
      A3
      和
      A4
      machine series,GPU 会由 machine type 自动附加(除非为
      N1
      附加 GPU 或使用 vWS,否则省略
      --accelerator
      )。创建独立 Flex-Start GPU 需要
      --reservation-affinity=none
      、
      --max-run-duration
      (
      10m
      到
      7d
      )、
      --instance-termination-action=DELETE
      (或
      STOP
      )、
      --maintenance-policy=TERMINATE
      和
      --request-valid-for-duration
      (使用
      90s
      到
      2h
      在某个可用区排队,或使用
      0s
      仅立即分配)。
    2. 对于多实例 GPU 作业或需要更长时间/跨可用区排队的工作负载,使用带有 Flex-Start instance template 和 resize request 的 MIG(参见在 MIG 中创建 resize request)。该 MIG 要求禁用 autoscaling,设置
      --default-action-on-vm-failure=do-nothing
      ;对于 Regional MIG,还要求设置
      --target-distribution-shape=any-single-zone
      和
      --instance-redistribution-type=none
      。
sh
undefined

Option A: Standalone VM with DWS Flex-Start (queues up to 2h via --request-valid-for-duration):

Option A: Standalone VM with DWS Flex-Start (queues up to 2h via --request-valid-for-duration):

gcloud compute instances create {vm_name}
--project={project_id} --zone={zone}
--machine-type=g2-standard-4
--provisioning-model=FLEX_START
--reservation-affinity=none
--request-valid-for-duration=2h
--max-run-duration=7d
--instance-termination-action=DELETE
--maintenance-policy=TERMINATE
gcloud compute instances create {vm_name}
--project={project_id} --zone={zone}
--machine-type=g2-standard-4
--provisioning-model=FLEX_START
--reservation-affinity=none
--request-valid-for-duration=2h
--max-run-duration=7d
--instance-termination-action=DELETE
--maintenance-policy=TERMINATE

Option B: Instance Template + Zonal or Regional MIG Resize Request (longer queueing):

Option B: Instance Template + Zonal or Regional MIG Resize Request (longer queueing):

gcloud compute instance-templates create {template_name}
--project={project_id} --machine-type=g2-standard-4
--provisioning-model=FLEX_START
--reservation-affinity=none
--max-run-duration=7d
--instance-termination-action=DELETE
--maintenance-policy=TERMINATE
gcloud compute instance-templates create {template_name}
--project={project_id} --machine-type=g2-standard-4
--provisioning-model=FLEX_START
--reservation-affinity=none
--max-run-duration=7d
--instance-termination-action=DELETE
--maintenance-policy=TERMINATE

Queue capacity on a MIG configured per bullet 2 (--zone={zone} or --region={region}):

Queue capacity on a MIG configured per bullet 2 (--zone={zone} or --region={region}):

gcloud compute instance-groups managed resize-requests create {mig_name}
--project={project_id} --region={region}
--resize-request={request_name} --resize-by={vm_count}

---
gcloud compute instance-groups managed resize-requests create {mig_name}
--project={project_id} --region={region}
--resize-request={request_name} --resize-by={vm_count}

---

Step 3: Long-Term Architectural Prevention

第 3 步:长期架构预防

To permanently eliminate single-zone stockouts for production services, implement one or more of the following patterns:
要彻底消除生产服务的单可用区库存耗尽问题,应实施以下一种或多种模式:

A. Configure Regional MIGs with
balanced
or
any
Distribution Shape

A. 使用
balanced
或
any
Distribution Shape 配置 Regional MIG

A Regional MIG spreads instances across zones in a region. Configure its Target Distribution Shape (see Regional MIG distribution shape documentation):
  • balanced
    (
    BALANCED
    )
    : Prioritizes even zonal distribution, but automatically spills over to healthy sibling zones during a stockout.
  • any
    (
    ANY
    )
    : Provisions capacity in whichever zone has available hardware fastest, maximizing obtainability under regional pressure.
  • any-single-zone
    (
    ANY_SINGLE_ZONE
    )
    : Colocates all VMs in one healthy zone with sufficient capacity (ideal for low-latency inter-node workloads).
Critical Operational Rule (
--instance-redistribution-type
)
:
  • Regional MIGs default to
    even
    (
    EVEN
    ) +
    proactive
    (
    PROACTIVE
    ), which periodically deletes running VMs in over-represented zones and recreates them in under-represented or recovering zones to enforce strict zonal balance—disrupting stateful workloads and failing during stockouts.
  • balanced
    (
    BALANCED
    ),
    any
    (
    ANY
    ), and
    any-single-zone
    (
    ANY_SINGLE_ZONE
    ) all require
    --instance-redistribution-type=none
    . Compute Engine supports
    proactive
    redistribution only with
    even
    and rejects
    proactive
    with
    balanced
    ,
    any
    , or
    any-single-zone
    (see Set a target distribution shape).
  • Calliope Flag Case Rule: Unlike
    instances bulk create
    (uppercase),
    gcloud compute instance-groups managed update
    requires lowercase values for
    --target-distribution-shape
    (
    any
    ,
    any-single-zone
    ,
    balanced
    ,
    even
    ) and
    --instance-redistribution-type
    (
    none
    ,
    proactive
    ):
sh
undefined
Regional MIG 将实例分布到区域内的多个可用区。请配置其 Target Distribution Shape(参见Regional MIG distribution shape 文档):
  • balanced
    (
    BALANCED
    )
    :优先实现均匀的可用区分布,但发生库存耗尽时会自动溢出到健康的相邻可用区。
  • any
    (
    ANY
    )
    :在最快获得可用硬件的可用区中预配容量,从而在区域资源紧张时最大化资源可获取性。
  • any-single-zone
    (
    ANY_SINGLE_ZONE
    )
    :将所有 VM 共置在一个具有足够容量的健康可用区中,适用于对节点间延迟敏感的低延迟工作负载。
关键运维规则(
--instance-redistribution-type
)
:
  • Regional MIG 默认使用
    even
    (
    EVEN
    )+
    proactive
    (
    PROACTIVE
    ),会定期删除分布过多的可用区中正在运行的 VM,并将其重新创建在分布不足或正在恢复的可用区中,以维持严格的可用区均衡;这会中断有状态工作负载,并且可能在库存耗尽期间失败。
  • balanced
    (
    BALANCED
    )、
    any
    (
    ANY
    )和
    any-single-zone
    (
    ANY_SINGLE_ZONE
    )均要求设置
    --instance-redistribution-type=none
    。Compute Engine 仅支持将
    proactive
    redistribution 与
    even
    搭配使用,并会拒绝将
    proactive
    与
    balanced
    、
    any
    或
    any-single-zone
    搭配使用(参见设置 Target Distribution Shape)。
  • Calliope 标志大小写规则:与
    instances bulk create
    不同(使用大写),
    gcloud compute instance-groups managed update
    要求
    --target-distribution-shape
    (
    any
    、
    any-single-zone
    、
    balanced
    、
    even
    )和
    --instance-redistribution-type
    (
    none
    、
    proactive
    )使用小写值:
sh
undefined

Allow spillover to healthy zones without proactively deleting running VMs:

Allow spillover to healthy zones without proactively deleting running VMs:

gcloud compute instance-groups managed update {mig_name}
--project={project_id} --region={region}
--instance-redistribution-type=none --target-distribution-shape=balanced
gcloud compute instance-groups managed update {mig_name}
--project={project_id} --region={region}
--instance-redistribution-type=none --target-distribution-shape=balanced

Or maximize obtainability in any zone (requires --instance-redistribution-type=none):

Or maximize obtainability in any zone (requires --instance-redistribution-type=none):

gcloud compute instance-groups managed update {mig_name}
--project={project_id} --region={region}
--instance-redistribution-type=none --target-distribution-shape=any
undefined
gcloud compute instance-groups managed update {mig_name}
--project={project_id} --region={region}
--instance-redistribution-type=none --target-distribution-shape=any
undefined

B. Enable MIG Instance Flexibility (Multi-Family Machine Type Fallback)

B. 启用 MIG Instance Flexibility(Multi-Family Machine Type Fallback)

Configure the MIG with Instance Flexibility (
instanceFlexibilityPolicy
) so it can fall back across equivalent machine families (for example, rank 1
c3-standard-8
, rank 2
n2-standard-8
or
n2d-standard-8
) when the primary family is exhausted (see About instance flexibility and Add instance flexibility):
  • Constraints & Disk/Repair Rules:
    • Supported only on Regional MIGs (not Zonal MIGs) with
      --target-distribution-shape
      set to
      balanced
      ,
      any
      , or
      any-single-zone
      (default
      even
      is rejected) and
      --instance-redistribution-type=none
      . Configure with
      gcloud beta compute instance-groups managed update
      (
      --instance-selection
      /
      --remove-instance-selections
      ) or REST
      instanceFlexibilityPolicy.instanceSelections
      .
    • Falling back across series with incompatible boot disk support (such as Hyperdisk-only
      n4-standard-8
      to Persistent-Disk-only
      n2-standard-8
      ) requires per-selection disk overrides via
      --instance-flexibility-policy
      (or
      --flags-file
      ) rather than
      --instance-selection
      alone.
    • Enable
      --force-update-on-repair
      so MIG VM repairs can fall back to alternative machine types (and whenever using disk or CPU overrides).
sh
gcloud beta compute instance-groups managed update {mig_name} \
    --project={project_id} --region={region} \
    --target-distribution-shape=balanced \
    --instance-redistribution-type=none \
    --instance-selection="name=primary-tier,machine-type=c3-standard-8,rank=1" \
    --instance-selection="name=fallback-tier,machine-type=n2-standard-8,rank=2"
为 MIG 配置 Instance Flexibility(
instanceFlexibilityPolicy
),使其在主要 machine family 耗尽时,可以回退到等效 machine family(例如优先级 1 的
c3-standard-8
,优先级 2 的
n2-standard-8
或
n2d-standard-8
)(参见关于 Instance Flexibility和添加 Instance Flexibility):
  • 约束与磁盘/修复规则:
    • 仅支持在 Regional MIG 上使用(不支持 Zonal MIG),并且必须将
      --target-distribution-shape
      设置为
      balanced
      、
      any
      或
      any-single-zone
      (默认的
      even
      会被拒绝),同时设置
      --instance-redistribution-type=none
      。通过
      gcloud beta compute instance-groups managed update
      (
      --instance-selection
      /
      --remove-instance-selections
      )或 REST 中的
      instanceFlexibilityPolicy.instanceSelections
      进行配置。
    • 如果在不同 series 之间回退时存在不兼容的 boot disk 支持(例如仅支持 Hyperdisk 的
      n4-standard-8
      回退到仅支持 Persistent Disk 的
      n2-standard-8
      ),则必须通过
      --instance-flexibility-policy
      (或
      --flags-file
      )为每个 selection 设置磁盘覆盖项,不能只使用
      --instance-selection
      。
    • 启用
      --force-update-on-repair
      ,使 MIG VM 修复操作能够回退到替代 machine type;使用磁盘或 CPU 覆盖项时也应启用此选项。
sh
gcloud beta compute instance-groups managed update {mig_name} \
    --project={project_id} --region={region} \
    --target-distribution-shape=balanced \
    --instance-redistribution-type=none \
    --instance-selection="name=primary-tier,machine-type=c3-standard-8,rank=1" \
    --instance-selection="name=fallback-tier,machine-type=n2-standard-8,rank=2"

C. Purchase On-Demand Capacity Reservations

C. 购买 On-Demand Capacity Reservations

For mission-critical production workloads requiring 100% guaranteed zonal capacity assurance, create an On-Demand Zonal Reservation (see About reservations of zonal resources and Create a reservation for a single project):
  • Operational & IAM Rules:
    1. Requires
      roles/compute.admin
      (or custom
      compute.reservations.*
      ) and succeeds only when physical capacity is currently available in
      {zone}
      —if
      {zone}
      is actively stocked out, create the reservation in an available sibling zone or wait until capacity replenishes.
    2. For N1 with GPUs or Local SSDs,
      --accelerator
      and
      --local-ssd
      flags must exactly match the consuming VM's properties.
sh
gcloud compute reservations create {reservation_name} \
    --project={project_id} --zone={zone} \
    --vm-count={vm_count} --machine-type={machine_type}

对于需要 100% 保证可用区容量保障的关键生产工作负载,请创建 On-Demand Zonal Reservation(参见关于可用区资源预留和为单个项目创建预留):
  • 运维与 IAM 规则:
    1. 需要
      roles/compute.admin
      (或自定义
      compute.reservations.*
      ),并且仅当
      {zone}
      当前存在物理容量时才能成功
      ;如果
      {zone}
      正处于库存耗尽状态,请在有容量的相邻可用区创建预留,或等待容量补充。
    2. 对于带 GPU 的 N1 或使用 Local SSD 的实例,
      --accelerator
      和
      --local-ssd
      标志必须与使用该预留的 VM 属性完全一致。
sh
gcloud compute reservations create {reservation_name} \
    --project={project_id} --zone={zone} \
    --vm-count={vm_count} --machine-type={machine_type}

Safe Rollback, Teardown & Cost Governance

安全回滚、拆除与成本治理

  1. Restore Regional MIG Distribution Settings: To roll back from
    any
    to opportunistic zonal balance (
    balanced
    requires
    --instance-redistribution-type=none
    and converges on resizes), or to return a stateless MIG to strict
    even
    +
    proactive
    distribution (after manually rebalancing if zone counts differ by more than 1 VM):
    sh
    # Option A: Converge opportunistically on resizes without deleting VMs:
    gcloud compute instance-groups managed update {mig_name} \
        --project={project_id} --region={region} \
        --target-distribution-shape=balanced --instance-redistribution-type=none
    # Option B: Stateless strict even + proactive (rebalance manually first if >1 VM diff):
    gcloud compute instance-groups managed update {mig_name} \
        --project={project_id} --region={region} \
        --target-distribution-shape=even --instance-redistribution-type=none
    gcloud compute instance-groups managed update {mig_name} \
        --project={project_id} --region={region} \
        --instance-redistribution-type=proactive
  2. Remove MIG Instance Selections: To roll back fallback selections:
    sh
    gcloud beta compute instance-groups managed update {mig_name} \
        --project={project_id} --region={region} \
        --remove-instance-selections=fallback-tier
  3. Delete Unused Capacity Reservations & Temporary VMs:
    • Cost Warning: On-Demand Capacity Reservations are billed at standard Compute Engine rates from creation whether or not VMs run inside them (and count toward regional quota). Consult Google Cloud pricing and Compute Engine pricing for current rates rather than quoting hardcoded prices.
    • Omit
      --quiet
      on manual destructive commands so interactive confirmation prevents accidental deletion in production:
      sh
      gcloud compute reservations delete {reservation_name} \
          --project={project_id} --zone={zone}
      gcloud compute instances delete {vm_name} \
          --project={project_id} --zone={zone}
  1. 恢复 Regional MIG Distribution 设置:如需从
    any
    回滚到机会式可用区均衡(
    balanced
    要求
    --instance-redistribution-type=none
    ,并通过 resize 逐步收敛),或将无状态 MIG 恢复为严格的
    even
    +
    proactive
    分布(如果各可用区的 VM 数量相差超过 1 个,必须先执行手动重新平衡):
    sh
    # Option A: Converge opportunistically on resizes without deleting VMs:
    gcloud compute instance-groups managed update {mig_name} \
        --project={project_id} --region={region} \
        --target-distribution-shape=balanced --instance-redistribution-type=none
    # Option B: Stateless strict even + proactive (rebalance manually first if >1 VM diff):
    gcloud compute instance-groups managed update {mig_name} \
        --project={project_id} --region={region} \
        --target-distribution-shape=even --instance-redistribution-type=none
    gcloud compute instance-groups managed update {mig_name} \
        --project={project_id} --region={region} \
        --instance-redistribution-type=proactive
  2. 移除 MIG Instance Selections:如需回退 fallback selections:
    sh
    gcloud beta compute instance-groups managed update {mig_name} \
        --project={project_id} --region={region} \
        --remove-instance-selections=fallback-tier
  3. 删除未使用的 Capacity Reservations 和临时 VM:
    • 成本警告:On-Demand Capacity Reservations 从创建之时起即按标准 Compute Engine 费率计费,无论其内部是否运行 VM(并且会占用区域配额)。请查阅 Google Cloud pricing 和 Compute Engine pricing 获取当前费率,不要引用硬编码价格。
    • 手动执行破坏性命令时不要省略
      --quiet
      ,让交互式确认机制防止生产环境中的误删除:
      sh
      gcloud compute reservations delete {reservation_name} \
          --project={project_id} --zone={zone}
      gcloud compute instances delete {vm_name} \
          --project={project_id} --zone={zone}