compute-host-events-troubleshooting

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Compute Engine VM Host Events & Infrastructure Crash Troubleshooting

Compute Engine VM 宿主事件与基础设施崩溃故障排查

This skill diagnoses unexpected Google Compute Engine VM restarts, terminations, and unresponsiveness caused by infrastructure host events (
compute.instances.hostError
,
compute.instances.migrateOnHostMaintenance
,
compute.instances.terminateOnHostMaintenance
,
compute.instances.preempted
,
compute.instances.automaticRestart
) or guest-OS-initiated shutdowns (
compute.instances.guestTerminate
). It provides authoritative Cloud Audit Log queries, Log Analytics
_Required._AllLogs
SQL aggregations, upcoming maintenance window checks, and host scheduling remediations.
MCP Tools:
get_instance_basic_info
,
list_instances
,
list_instance_attached_disks
,
start_instance
,
stop_instance
,
reset_instance
,
delete_instance
,
get_zone_operation
本 skill 诊断由基础设施宿主事件 (
compute.instances.hostError
、
compute.instances.migrateOnHostMaintenance
、
compute.instances.terminateOnHostMaintenance
、
compute.instances.preempted
、
compute.instances.automaticRestart
) 或来宾操作系统发起的关闭 (
compute.instances.guestTerminate
) 导致的 Google Compute Engine VM 意外重启、 终止和失去响应。它提供权威的 Cloud Audit Log 查询、Log Analytics
_Required._AllLogs
SQL 聚合、即将进行的 维护窗口检查,以及宿主调度补救措施。
MCP 工具:
get_instance_basic_info
、
list_instances
、
list_instance_attached_disks
、
start_instance
、
stop_instance
、
reset_instance
、
delete_instance
、
get_zone_operation

Usage Attribution

使用归属说明

MUST READ: Before executing any
gcloud
, REST API, or MCP command, you MUST read and follow the instructions in Usage Attribution.

必须阅读:在执行任何
gcloud
、REST API 或 MCP 命令前,你 必须阅读并遵循 使用归属说明中的说明。

Prerequisites & Environment Setup

前置条件与环境设置

Any host or project modifications (installing or updating
gcloud
, configuring the Compute Engine MCP server, enabling APIs or Log Analytics, mutating VM scheduling/metadata, creating log-based metrics, or deleting resources) must be explicitly approved by the user before execution.
  1. Google Cloud SDK: Verify
    gcloud
    is installed (
    gcloud --version
    ). If missing or outdated, ask the user for approval before installing via Install the gcloud CLI.
  2. Compute Engine MCP Server & Capability Boundary:
    • The Compute Engine MCP server (Use the Compute Engine MCP server) exposes zonal VM discovery and lifecycle tools (
      get_instance_basic_info
      ,
      list_instances
      ,
      list_instance_attached_disks
      ,
      start_instance
      ,
      stop_instance
      ,
      reset_instance
      ,
      delete_instance
      ,
      get_zone_operation
      , requiring
      roles/mcp.toolUser
      ).
    • Critical Routing Rule:
      get_instance_basic_info
      returns a reduced
      InstanceBasicInfo
      projection (
      name
      ,
      id
      ,
      status
      ,
      machine_type
      ,
      create_time
      ,
      guest_accelerators
      ) that omits
      scheduling
      and
      resourceStatus.upcomingMaintenance
      , and Compute Engine MCP does not expose tools for Cloud Logging queries, serial console output,
      set-scheduling
      ,
      perform-maintenance
      , or instance metadata updates. Always route full scheduling inspection, upcoming maintenance checks, serial port output, audit log queries, and scheduling/metadata mutations to
      gcloud
      (or Compute Engine / Cloud Logging REST APIs).
  3. Authentication:
    • Local CLI:
      gcloud auth login
    • Application Default Credentials:
      gcloud auth application-default login
    • Service Accounts: Set
      GOOGLE_APPLICATION_CREDENTIALS
      to your service account credentials path.
  4. Project, Region & Billing:
    sh
    gcloud config set project {project_id}
    gcloud config set compute/region {region}
    gcloud config set compute/zone {zone}
    Note: An active billing account attachment is required (even when evaluating Free Tier eligible resources). If no region is specified, default to
    us-central1
    (
    us-central1-a
    ), and check Google Cloud locations for regional hardware availability.
  5. API Enablement:
    sh
    gcloud services enable compute.googleapis.com logging.googleapis.com monitoring.googleapis.com
  6. Required IAM Roles:
    • Read-Only Diagnostics:
      roles/compute.viewer
      (instance scheduling and serial console output),
      roles/logging.viewer
      (or
      roles/logging.privateLogViewer
      to read
      system_event
      and
      activity
      audit logs), and
      roles/logging.viewAccessor
      for Log Analytics queries in Cloud Logging (plus
      roles/bigquery.jobUser
      and
      roles/bigquery.dataViewer
      when querying a linked Log Analytics dataset from BigQuery Studio or
      bq
      ).
    • Remediation & MCP:
      roles/compute.instanceAdmin.v1
      (to run
      set-scheduling
      ,
      perform-maintenance
      ,
      add-metadata
      , or
      remove-metadata
      ),
      roles/logging.configWriter
      (to create log-based metrics), and
      roles/mcp.toolUser
      (for Compute Engine MCP calls).

任何宿主或项目层面的修改(安装或更新
gcloud
、配置 Compute Engine MCP server、启用 API 或 Log Analytics、修改 VM 调度/元数据、创建基于日志的指标,或删除资源)都必须 在执行前获得用户明确批准。
  1. Google Cloud SDK:确认已安装
    gcloud
    (
    gcloud --version
    )。如果 缺失或版本过旧,请在通过 安装 gcloud CLI安装前征得用户批准。
  2. Compute Engine MCP server 与能力边界:
    • Compute Engine MCP server (使用 Compute Engine MCP server) 暴露可用区 VM 发现与生命周期工具(
      get_instance_basic_info
      、
      list_instances
      、
      list_instance_attached_disks
      、
      start_instance
      、
      stop_instance
      、
      reset_instance
      、
      delete_instance
      、
      get_zone_operation
      ,需要
      roles/mcp.toolUser
      )。
    • 关键路由规则:
      get_instance_basic_info
      返回精简后的
      InstanceBasicInfo
      投影(
      name
      、
      id
      、
      status
      、
      machine_type
      、
      create_time
      、
      guest_accelerators
      ),其中不包含
      scheduling
      和
      resourceStatus.upcomingMaintenance
      ;Compute Engine MCP 也不提供 Cloud Logging 查询、串口控制台输出、
      set-scheduling
      、
      perform-maintenance
      或实例元数据更新工具。 始终将完整的调度检查、即将进行的维护检查、串口输出、 审计日志查询以及调度/元数据变更路由至
      gcloud
      (或 Compute Engine / Cloud Logging REST API)。
  3. 身份验证:
    • 本地 CLI:
      gcloud auth login
    • Application Default Credentials:
      gcloud auth application-default login
    • Service Accounts:将
      GOOGLE_APPLICATION_CREDENTIALS
      设置为你的 Service Account 凭据路径。
  4. 项目、区域与结算:
    sh
    gcloud config set project {project_id}
    gcloud config set compute/region {region}
    gcloud config set compute/zone {zone}
    注意:必须关联有效的结算账号(即使是在评估符合 Free Tier 条件的资源时)。如果未指定区域,请默认使用
    us-central1
    (
    us-central1-a
    ),并查看 Google Cloud 区域以了解区域硬件可用性。
  5. 启用 API:
    sh
    gcloud services enable compute.googleapis.com logging.googleapis.com monitoring.googleapis.com
  6. 必需的 IAM 角色:
    • 只读诊断:
      roles/compute.viewer
      (实例调度和串口控制台输出)、
      roles/logging.viewer
      (或使用
      roles/logging.privateLogViewer
      读取
      system_event
      和
      activity
      审计日志),以及用于在 Cloud Logging 中执行 Log Analytics 查询的
      roles/logging.viewAccessor
      (从 BigQuery Studio 或
      bq
      查询关联的 Log Analytics 数据集时,还需要
      roles/bigquery.jobUser
      和
      roles/bigquery.dataViewer
      )。
    • 补救与 MCP:
      roles/compute.instanceAdmin.v1
      (用于运行
      set-scheduling
      、
      perform-maintenance
      、
      add-metadata
      或
      remove-metadata
      )、
      roles/logging.configWriter
      (用于创建基于日志的指标), 以及
      roles/mcp.toolUser
      (用于 Compute Engine MCP 调用)。

Diagnostic Workflow: Classify the Termination Event

诊断工作流:终止事件分类

When a user reports an unexpected VM reboot, stop, or crash, run targeted Cloud Logging queries against
cloudaudit.googleapis.com/system_event
(Compute Engine infrastructure events) and
cloudaudit.googleapis.com/activity
(caller-initiated API stops/resets). Always pass
--freshness=7d
(or a window covering the incident timestamp) because
gcloud logging read
defaults
--freshness
to
1d
and silently returns zero rows for incidents older than 24 hours. See Troubleshooting VM shutdowns and reboots.
当用户报告 VM 意外重启、停止或崩溃时,针对
cloudaudit.googleapis.com/system_event
(Compute Engine 基础设施事件)和
cloudaudit.googleapis.com/activity
(调用方发起的 API 停止/重置)运行定向 Cloud Logging 查询。 始终传入
--freshness=7d
(或一个涵盖事件时间戳的时间窗口), 因为
gcloud logging read
默认将
--freshness
设为
1d
, 对于超过 24 小时的事件会静默返回零行。请参阅 排查 VM 关闭和重启问题。

Key Distinction:
hostError
vs. Planned Maintenance vs.
preempted

关键区别:
hostError
vs. 计划内维护 vs.
preempted

  • compute.instances.hostError
    (Unplanned Hardware/Hypervisor Crash)
    : Indicates an unexpected hardware or hypervisor failure on the physical server hosting the VM. If
    scheduling.automaticRestart
    is
    true
    (
    --restart-on-failure
    ), Compute Engine automatically restarts the VM on a healthy physical host (typically within 3 to 5.5 minutes from detection on standard VMs, logging
    compute.instances.automaticRestart
    ). On VMs with attached Local SSDs, Compute Engine logs
    compute.instances.hostError
    with
    operation.first="true"
    when Local SSD recovery begins and
    operation.last="true"
    when recovery completes or reaches
    scheduling.localSsdRecoveryTimeout
    .
  • compute.instances.migrateOnHostMaintenance
    /
    compute.instances.terminateOnHostMaintenance
    (Planned Infrastructure Maintenance)
    : Compute Engine logs these exact
    system_event
    method names (along with
    compute.instances.upcomingMaintenance
    notifications) during scheduled hardware or hypervisor updates. Depending on
    onHostMaintenance
    , the VM either transparently live-migrates (
    MIGRATE
    ->
    compute.instances.migrateOnHostMaintenance
    ) or terminates (
    TERMINATE
    ->
    compute.instances.terminateOnHostMaintenance
    , followed by
    compute.instances.automaticRestart
    if
    automaticRestart: true
    ).
  • compute.instances.preempted
    (Spot / Preemptible VM Reclaimed)
    : Indicates Compute Engine reclaimed capacity from a Spot VM or Preemptible VM.
  • compute.instances.hostError
    (计划外硬件/Hypervisor 崩溃)
    : 表示承载 VM 的物理服务器发生意外硬件或 Hypervisor 故障。 如果
    scheduling.automaticRestart
    为
    true
    (
    --restart-on-failure
    ),Compute Engine 会自动将 VM 重启到 健康的物理宿主上(对于标准 VM,通常在检测后的 3 到 5.5 分钟内完成, 并记录
    compute.instances.automaticRestart
    )。对于挂载 Local SSD 的 VM, 当 Local SSD 恢复开始时,Compute Engine 会记录
    compute.instances.hostError
    ,且
    operation.first="true"
    ; 当恢复完成或达到
    scheduling.localSsdRecoveryTimeout
    时,
    operation.last="true"
    。
  • compute.instances.migrateOnHostMaintenance
    /
    compute.instances.terminateOnHostMaintenance
    (计划内基础设施维护)
    : 在计划执行的硬件或 Hypervisor 更新期间,Compute Engine 会记录这些确切的
    system_event
    方法名称(同时还会发出
    compute.instances.upcomingMaintenance
    通知)。根据
    onHostMaintenance
    的配置,VM 可能会透明地实时迁移(
    MIGRATE
    ->
    compute.instances.migrateOnHostMaintenance
    ),也可能会被终止 (
    TERMINATE
    ->
    compute.instances.terminateOnHostMaintenance
    , 如果
    automaticRestart: true
    ,随后会记录
    compute.instances.automaticRestart
    )。
  • compute.instances.preempted
    (Spot / Preemptible VM 被回收)
    : 表示 Compute Engine 从 Spot VM 或 Preemptible VM 回收了容量。

1. Query Infrastructure Host Events & Caller Stops in Cloud Logging

1. 在 Cloud Logging 中查询基础设施宿主事件与调用方停止操作

Run this
gcloud logging read
command to determine whether
{vm_name}
experienced a physical host error, planned maintenance, Spot preemption, MIG recreation, or an external caller-initiated API stop/reset:
sh
gcloud logging read 'resource.type="gce_instance" AND (log_id("cloudaudit.googleapis.com/system_event") OR log_id("cloudaudit.googleapis.com/activity")) AND protoPayload.methodName=~"(compute\.instances\.(hostError|migrateOnHostMaintenance|terminateOnHostMaintenance|preempted|automaticRestart|upcomingMaintenance|guestTerminate|repair.*)|v1\.compute\.instances\.(stop|reset|delete|performMaintenance|simulateMaintenanceEvent))" AND protoPayload.resourceName:"{vm_name}"' \
    --project={project_id} \
    --freshness=7d \
    --limit=25 \
    --format="table(timestamp, logName.basename(), protoPayload.methodName, protoPayload.authenticationInfo.principalEmail, operation.first, operation.last, protoPayload.status.message)"
运行以下
gcloud logging read
命令,以判断
{vm_name}
是否发生了物理宿主错误、计划内维护、Spot 抢占、MIG 重建,或外部调用方发起的 API 停止/重置:
sh
gcloud logging read 'resource.type="gce_instance" AND (log_id("cloudaudit.googleapis.com/system_event") OR log_id("cloudaudit.googleapis.com/activity")) AND protoPayload.methodName=~"(compute\.instances\.(hostError|migrateOnHostMaintenance|terminateOnHostMaintenance|preempted|automaticRestart|upcomingMaintenance|guestTerminate|repair.*)|v1\.compute\.instances\.(stop|reset|delete|performMaintenance|simulateMaintenanceEvent))" AND protoPayload.resourceName:"{vm_name}"' \
    --project={project_id} \
    --freshness=7d \
    --limit=25 \
    --format="table(timestamp, logName.basename(), protoPayload.methodName, protoPayload.authenticationInfo.principalEmail, operation.first, operation.last, protoPayload.status.message)"

2. Check Specifically for Guest OS Initiated Terminations (
guestTerminate
)

2. 专门检查来宾操作系统发起的终止(
guestTerminate
)

If the VM stopped without a
hostError
, maintenance termination, preemption, or user-initiated
v1.compute.instances.stop
Admin Activity log, check whether the shutdown was initiated from inside the Guest OS (
sudo shutdown -h now
,
poweroff
,
systemctl poweroff
, or an internal watchdog script):
sh
gcloud logging read 'resource.type="gce_instance" AND log_id("cloudaudit.googleapis.com/system_event") AND protoPayload.methodName="compute.instances.guestTerminate" AND protoPayload.resourceName:"{vm_name}"' \
    --project={project_id} \
    --freshness=7d \
    --limit=20 \
    --format="yaml(timestamp, protoPayload.methodName, protoPayload.authenticationInfo.principalEmail, protoPayload.status)"
  • Critical
    guestTerminate
    Attribution Nuance
    : Because
    compute.instances.guestTerminate
    is a
    system_event
    generated when the guest ACPI power-off reaches the hypervisor, Compute Engine always records its generic built-in system principal in
    protoPayload.authenticationInfo.principalEmail
    — it never identifies which Linux/Windows user, cron job, or script inside the VM ran
    shutdown
    /
    poweroff
    (whereas external caller-initiated
    v1.compute.instances.stop
    ,
    reset
    , or
    delete
    entries in
    cloudaudit.googleapis.com/activity
    record the external IAM user or service account in
    principalEmail
    ).
  • How to Find the In-Guest Culprit: Inspect the VM's serial console output and guest OS logs from the previous boot (
    journalctl -b -1
    ,
    /var/log/syslog
    or
    /var/log/messages
    ,
    /var/log/auth.log
    , and
    last -x shutdown reboot
    ):
    sh
    gcloud compute instances get-serial-port-output {vm_name} --project={project_id} --zone={zone} --port=1
如果 VM 在没有
hostError
、维护终止、抢占或用户发起的
v1.compute.instances.stop
Admin Activity 日志的情况下停止,请检查 关闭操作是否从 Guest OS 内部发起(
sudo shutdown -h now
、
poweroff
、
systemctl poweroff
或内部 watchdog 脚本):
sh
gcloud logging read 'resource.type="gce_instance" AND log_id("cloudaudit.googleapis.com/system_event") AND protoPayload.methodName="compute.instances.guestTerminate" AND protoPayload.resourceName:"{vm_name}"' \
    --project={project_id} \
    --freshness=7d \
    --limit=20 \
    --format="yaml(timestamp, protoPayload.methodName, protoPayload.authenticationInfo.principalEmail, protoPayload.status)"
  • guestTerminate
    归属判定的重要细节
    :由于
    compute.instances.guestTerminate
    是在来宾操作系统的 ACPI 关机信号到达 Hypervisor 时生成的
    system_event
    ,因此 Compute Engine 始终只会在
    protoPayload.authenticationInfo.principalEmail
    中记录其 通用的内置系统主体
    ,而绝不会标识是 VM 内的哪个 Linux/Windows 用户、cron 任务或脚本执行了
    shutdown
    /
    poweroff
    ;相比之下,由外部调用方发起的
    v1.compute.instances.stop
    、
    reset
    或
    delete
    记录会出现在
    cloudaudit.googleapis.com/activity
    中,并在
    principalEmail
    中记录外部 IAM 用户或 Service Account。
  • 如何查找 Guest OS 内部的原因:检查 VM 的串口控制台输出和上一次启动期间的 Guest OS 日志(
    journalctl -b -1
    、
    /var/log/syslog
    或
    /var/log/messages
    、
    /var/log/auth.log
    ,以及
    last -x shutdown reboot
    ):
    sh
    gcloud compute instances get-serial-port-output {vm_name} --project={project_id} --zone={zone} --port=1

3. Log Analytics SQL Query: Aggregate Host Events Across the Project

3. Log Analytics SQL 查询:聚合整个项目中的宿主事件

Compute Engine host events (
hostError
,
migrateOnHostMaintenance
,
terminateOnHostMaintenance
,
preempted
,
automaticRestart
,
guestTerminate
) are
cloudaudit.googleapis.com/system_event
logs. By default, Cloud Logging's
_Default
sink excludes
system_event
logs and routes them exclusively to the
_Required
log bucket.
Prerequisites for
_Required._AllLogs
SQL
:
  1. Upgrade the
    _Required
    bucket to Log Analytics (seek user approval first):
    sh
    gcloud logging buckets update _Required --project={project_id} --location=global --enable-analytics
  2. In Log Analytics
    _AllLogs
    schemas,
    resource.labels
    has BigQuery type
    JSON
    (not
    STRUCT
    ). Always extract fields with
    JSON_VALUE(resource.labels.zone)
    and
    JSON_VALUE(resource.labels.instance_id)
    — grouping directly on raw
    JSON
    expressions fails with
    Grouping by expressions of type JSON is not allowed
    .
sql
SELECT
  JSON_VALUE(resource.labels.zone) AS zone,
  JSON_VALUE(resource.labels.instance_id) AS instance_id,
  REGEXP_EXTRACT(proto_payload.audit_log.resource_name, r'[^/]+$') AS vm_name,
  proto_payload.audit_log.method_name AS event_type,
  COUNT(*) AS event_count
FROM `{project_id}.global._Required._AllLogs`
WHERE
  resource.type = 'gce_instance'
  AND timestamp >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)
  AND proto_payload.audit_log.method_name IN (
    'compute.instances.hostError',
    'compute.instances.migrateOnHostMaintenance',
    'compute.instances.terminateOnHostMaintenance',
    'compute.instances.preempted',
    'compute.instances.automaticRestart',
    'compute.instances.guestTerminate'
  )
GROUP BY zone, instance_id, vm_name, event_type
ORDER BY event_count DESC;
Compute Engine 宿主事件(
hostError
、
migrateOnHostMaintenance
、
terminateOnHostMaintenance
、
preempted
、
automaticRestart
、
guestTerminate
)属于
cloudaudit.googleapis.com/system_event
日志。 默认情况下,Cloud Logging 的
_Default
sink 会排除
system_event
日志,并将其仅路由到
_Required
日志存储桶。
_Required._AllLogs
SQL 的前置条件
:
  1. 将
    _Required
    存储桶升级为 Log Analytics(先征得用户批准):
    sh
    gcloud logging buckets update _Required --project={project_id} --location=global --enable-analytics
  2. 在 Log Analytics
    _AllLogs
    Schema 中,
    resource.labels
    的 BigQuery 类型为
    JSON
    (而不是
    STRUCT
    )。始终使用
    JSON_VALUE(resource.labels.zone)
    和
    JSON_VALUE(resource.labels.instance_id)
    提取字段——直接按原始
    JSON
    表达式分组会失败并提示
    Grouping by expressions of type JSON is not allowed
    。
sql
SELECT
  JSON_VALUE(resource.labels.zone) AS zone,
  JSON_VALUE(resource.labels.instance_id) AS instance_id,
  REGEXP_EXTRACT(proto_payload.audit_log.resource_name, r'[^/]+$') AS vm_name,
  proto_payload.audit_log.method_name AS event_type,
  COUNT(*) AS event_count
FROM `{project_id}.global._Required._AllLogs`
WHERE
  resource.type = 'gce_instance'
  AND timestamp >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)
  AND proto_payload.audit_log.method_name IN (
    'compute.instances.hostError',
    'compute.instances.migrateOnHostMaintenance',
    'compute.instances.terminateOnHostMaintenance',
    'compute.instances.preempted',
    'compute.instances.automaticRestart',
    'compute.instances.guestTerminate'
  )
GROUP BY zone, instance_id, vm_name, event_type
ORDER BY event_count DESC;

4. Inspect Upcoming Scheduled Host Maintenance & Trigger Early Maintenance

4. 检查即将执行的计划宿主维护并提前触发维护

For instances that emit advance maintenance notifications (such as
A4X Max
,
A4X
,
A4
,
A3
,
A2
,
G4
,
G2
GPU instances, bare metal
-metal
instances,
H4D
/ Cloud RDMA instances,
X4
instances, storage-optimized
Z3
/
Z4D
/
Z4M
Local SSD instances, or Cloud TPU nodes), inspect upcoming maintenance via
gcloud
or the in-guest metadata server. See Monitor and plan for a host maintenance event and Get live migration notices from the metadata server:
sh
undefined
对于会提前发出维护通知的实例(例如
A4X Max
、
A4X
、
A4
、
A3
、
A2
、
G4
、
G2
GPU 实例、裸金属
-metal
实例、
H4D
/ Cloud RDMA 实例、
X4
实例、面向存储优化的
Z3
/
Z4D
/
Z4M
Local SSD 实例,或 Cloud TPU 节点),请通过
gcloud
或 Guest OS 内的元数据服务器检查即将进行的维护。 请参阅 监视并规划宿主维护事件和 从元数据服务器获取实时迁移通知:
sh
undefined

1. Check a single instance for scheduling policy and upcoming maintenance (Compute Engine API):

1. Check a single instance for scheduling policy and upcoming maintenance (Compute Engine API):

gcloud compute instances describe {vm_name} --project={project_id} --zone={zone}
--format="yaml(name, scheduling, resourceStatus.upcomingMaintenance)"
gcloud compute instances describe {vm_name} --project={project_id} --zone={zone}
--format="yaml(name, scheduling, resourceStatus.upcomingMaintenance)"

2. List upcoming maintenance windows across all instances in the project:

2. List upcoming maintenance windows across all instances in the project:

gcloud compute instances list --project={project_id}
--format="table(name, zone.basename(), resourceStatus.upcomingMaintenance.type, resourceStatus.upcomingMaintenance.windowStartTime, resourceStatus.upcomingMaintenance.windowEndTime, resourceStatus.upcomingMaintenance.canReschedule, resourceStatus.upcomingMaintenance.maintenanceStatus)"
gcloud compute instances list --project={project_id}
--format="table(name, zone.basename(), resourceStatus.upcomingMaintenance.type, resourceStatus.upcomingMaintenance.windowStartTime, resourceStatus.upcomingMaintenance.windowEndTime, resourceStatus.upcomingMaintenance.canReschedule, resourceStatus.upcomingMaintenance.maintenanceStatus)"

3. Query from inside the Guest OS via the Metadata Server:

3. Query from inside the Guest OS via the Metadata Server:


- **API vs. Metadata Server JSON Field Name Difference**:
  - In `gcloud` / Compute Engine REST (`resourceStatus.upcomingMaintenance`) and
    Cloud Logging, the maintenance classification field is named **`type`**
    (`SCHEDULED` or `UNSCHEDULED`), alongside `maintenanceStatus` (`PENDING` or
    `ONGOING`), `canReschedule` (`true` or `false`), `windowStartTime`,
    `windowEndTime`, and `latestWindowStartTime`.
  - When querying `/computeMetadata/v1/instance/upcoming-maintenance?alt=json`
    inside the guest OS, returns `"NONE"` when no maintenance is scheduled, or a
    JSON object where the classification property is named **`maintenanceType`**
    (e.g., `"maintenanceType": "SCHEDULED"`).
- **Manually Trigger Host Maintenance Early (`perform-maintenance`)**:
  - **Preconditions**: Check `canReschedule` and `maintenanceStatus` first. You
    can run `perform-maintenance` when `canReschedule` is `true` (and
    `maintenanceStatus` is `PENDING` when an active notice exists, or on
    supported machine series before a notice is posted to reset the maintenance
    cycle). If an active notice reports `canReschedule: false` or
    `maintenanceStatus: ONGOING`, manual triggering is rejected by the API.
  - **Immediate Impact**: Confirm with the user before running —
    `perform-maintenance` starts host maintenance **immediately** (terminating
    and optionally restarting `TERMINATE`-policy instances right away). See
    [Simulate or trigger a host maintenance event](https://docs.cloud.google.com/compute/docs/instances/trigger-host-maintenance-event.md.txt):

```sh
gcloud compute instances perform-maintenance {vm_name} --project={project_id} --zone={zone}


- **API 与元数据服务器之间的 JSON 字段名差异**:
  - 在 `gcloud` / Compute Engine REST(`resourceStatus.upcomingMaintenance`)和
    Cloud Logging 中,维护分类字段名为 **`type`**
    (`SCHEDULED` 或 `UNSCHEDULED`),并且还包含
    `maintenanceStatus`(`PENDING` 或 `ONGOING`)、`canReschedule`
    (`true` 或 `false`)、`windowStartTime`、
    `windowEndTime` 和 `latestWindowStartTime`。
  - 在 Guest OS 内查询
    `/computeMetadata/v1/instance/upcoming-maintenance?alt=json` 时,
    如果没有计划维护,则返回 `"NONE"`;否则返回一个 JSON 对象,
    其分类属性名为 **`maintenanceType`**
    (例如 `"maintenanceType": "SCHEDULED"`)。
- **手动提前触发宿主维护(`perform-maintenance`)**:
  - **前置条件**:先检查 `canReschedule` 和 `maintenanceStatus`。
    当 `canReschedule` 为 `true` 时可以运行 `perform-maintenance`
    (如果存在有效通知,`maintenanceStatus` 应为 `PENDING`;
    或者对于受支持的机器系列,可以在通知发布前运行,以重置维护周期)。
    如果有效通知显示 `canReschedule: false` 或
    `maintenanceStatus: ONGOING`,API 将拒绝手动触发。
  - **即时影响**:运行前必须与用户确认——
    `perform-maintenance` 会**立即**开始宿主维护(立即终止采用 `TERMINATE`
    策略的实例,并可选地重启这些实例)。请参阅
    [模拟或触发宿主维护事件](https://docs.cloud.google.com/compute/docs/instances/trigger-host-maintenance-event.md.txt):

```sh
gcloud compute instances perform-maintenance {vm_name} --project={project_id} --zone={zone}

Actionable Remediation & Hardening Rules

可执行的补救与加固规则

Required IAM Role: Modifying instance scheduling or metadata requires
roles/compute.instanceAdmin.v1
. Always obtain user confirmation before executing mutating commands.
所需 IAM 角色:修改实例调度或元数据需要
roles/compute.instanceAdmin.v1
。执行变更命令前始终必须获得用户确认。

Case A: Preventing Downtime During Planned Maintenance (
MIGRATE
Policy)

案例 A:在计划内维护期间防止停机(
MIGRATE
策略)

If a live-migration-capable VM (such as an
N2
,
N2D
,
C3
,
C3D
,
C4
, or
N4
instance; note that standard
E2
VMs only support
MIGRATE
during host maintenance events and disallow
TERMINATE
) logged
compute.instances.terminateOnHostMaintenance
during routine host maintenance, its
scheduling.onHostMaintenance
policy was set to
TERMINATE
instead of
MIGRATE
.
  • Fix: Enable Live Migration (
    --maintenance-policy=MIGRATE
    ) and Automatic Restart (
    --restart-on-failure
    , which sets REST
    scheduling.automaticRestart: true
    ; use
    --no-restart-on-failure
    to disable). Per Set VM host maintenance policy,
    gcloud compute instances set-scheduling
    updates
    onHostMaintenance
    ,
    automaticRestart
    (except on running CSEK-encrypted VMs),
    hostErrorTimeoutSeconds
    , and
    localSsdRecoveryTimeout
    in-place on a running VM without stopping or rebooting it (note: although the
    gcloud compute instances set-scheduling --help
    / REST
    instances.setScheduling
    reference docstring contains a legacy note stating the method requires a stopped
    TERMINATED
    instance, stopping the VM first is only required when changing
    --provisioning-model
    /
    --preemptible
    or updating
    automaticRestart
    on CSEK-encrypted VMs)
    :
sh
gcloud compute instances set-scheduling {vm_name} \
    --project={project_id} \
    --zone={zone} \
    --maintenance-policy=MIGRATE \
    --restart-on-failure
  • VM Families & Configurations That Do Not Support Live Migration (Always Clarify): Whenever recommending
    --maintenance-policy=MIGRATE
    , explicitly note which VM configurations cannot live-migrate and instead require
    --maintenance-policy=TERMINATE
    paired with
    --restart-on-failure
    (except Spot/Preemptible VMs, which require
    TERMINATE
    with
    --no-restart-on-failure
    ), and note that standard
    E2
    VMs only support
    MIGRATE
    :
    • Spot VMs (
      provisioningModel: SPOT
      ) and legacy Preemptible VMs
    • GPU-attached VMs (
      A4X Max
      ,
      A4X
      ,
      A4
      ,
      A3
      ,
      A2
      ,
      G4
      ,
      G2
      , and
      N1
      with attached GPUs; note that
      N2
      does not support attached GPUs)
    • Cloud TPU VMs and Bare Metal Instances (
      *-metal
      )
    • Instances with a Cloud RDMA network interface (such as
      H4D
      and
      Z4M
      with Cloud RDMA)
    • X4
      and
      X5
      memory-optimized instances
    • Large Local SSD Configurations (
      Z3
      instances with > 18,000 GiB of Local SSD, and
      Z4D
      /
      Z4M
      instances with > 42,000 GiB of Local SSD)
如果支持实时迁移的 VM(例如
N2
、
N2D
、
C3
、
C3D
、
C4
或
N4
实例;请注意,标准
E2
VM 仅支持宿主维护事件期间的
MIGRATE
,不允许使用
TERMINATE
)在例行宿主维护期间记录了
compute.instances.terminateOnHostMaintenance
,则说明其
scheduling.onHostMaintenance
策略被设置为
TERMINATE
,而不是
MIGRATE
。
  • 修复:启用 Live Migration(
    --maintenance-policy=MIGRATE
    )和 Automatic Restart(
    --restart-on-failure
    ,这会将 REST
    scheduling.automaticRestart
    设置为
    true
    ;使用
    --no-restart-on-failure
    可将其禁用)。根据 设置 VM 宿主维护策略,
    gcloud compute instances set-scheduling
    可以更新
    onHostMaintenance
    、
    automaticRestart
    (运行中的 CSEK 加密 VM 除外)、
    hostErrorTimeoutSeconds
    和
    localSsdRecoveryTimeout
    , 无需停止或重启正在运行的 VM (请注意,虽然
    gcloud compute instances set-scheduling --help
    / REST
    instances.setScheduling
    参考文档字符串中包含一条旧版说明,称该方法要求实例 处于已停止的
    TERMINATED
    状态,但只有更改
    --provisioning-model
    /
    --preemptible
    ,或更新 CSEK 加密 VM 的
    automaticRestart
    时,才需要先停止 VM)
    :
sh
gcloud compute instances set-scheduling {vm_name} \
    --project={project_id} \
    --zone={zone} \
    --maintenance-policy=MIGRATE \
    --restart-on-failure
  • 不支持 Live Migration 的 VM 系列与配置(始终说明) 每当建议使用
    --maintenance-policy=MIGRATE
    时,必须明确说明哪些 VM 配置无法实时迁移、因而必须使用
    --maintenance-policy=TERMINATE
    搭配
    --restart-on-failure
    (Spot/Preemptible VM 除外,它们必须使用
    TERMINATE
    搭配
    --no-restart-on-failure
    ),并说明标准
    E2
    VM 仅支持
    MIGRATE
    :
    • Spot VM(
      provisioningModel: SPOT
      )和旧版 Preemptible VM
    • 挂载 GPU 的 VM(
      A4X Max
      、
      A4X
      、
      A4
      、
      A3
      、
      A2
      、
      G4
      、
      G2
      , 以及挂载 GPU 的
      N1
      ;请注意,
      N2
      不支持挂载 GPU)
    • Cloud TPU VM 和裸金属实例(
      *-metal
      )
    • 使用 Cloud RDMA 网络接口的实例(例如使用 Cloud RDMA 的
      H4D
      和
      Z4M
      )
    • X4
      和
      X5
      内存优化实例
    • 大型 Local SSD 配置(Local SSD 超过 18,000 GiB 的
      Z3
      实例, 以及 Local SSD 超过 42,000 GiB 的
      Z4D
      /
      Z4M
      实例)

Case B: Faster Recovery from Unresponsive Hosts (
--host-error-timeout-seconds
)

案例 B:加快无响应宿主的恢复速度(
--host-error-timeout-seconds
)

If an application experiences prolonged unresponsiveness before Compute Engine's default host watchdog declares a
hostError
, shorten the detection timeout via
--host-error-timeout-seconds
:
  • Constraints: Must be specified in 30-second increments. Valid range is
    90
    to
    300
    seconds in
    gcloud
    (
    90
    to
    330
    seconds via Compute Engine REST
    instances.setScheduling
    ). Unsupported on Sole-Tenant VMs and Spot / Preemptible VMs. Updates dynamically on a running VM (see Set VM host maintenance policy):
sh
gcloud compute instances set-scheduling {vm_name} --project={project_id} --zone={zone} --host-error-timeout-seconds=90
如果应用在 Compute Engine 的默认宿主 watchdog 判定
hostError
之前长时间无响应,请通过
--host-error-timeout-seconds
缩短检测超时:
  • 约束:必须以 30 秒为增量指定。在
    gcloud
    中的有效范围是
    90
    到
    300
    秒
    (通过 Compute Engine REST
    instances.setScheduling
    设置时为
    90
    到
    330
    秒)。Sole-Tenant VM 和 Spot / Preemptible VM 不支持此设置。可在运行中的 VM 上动态更新 (请参阅 设置 VM 宿主维护策略):
sh
gcloud compute instances set-scheduling {vm_name} --project={project_id} --zone={zone} --host-error-timeout-seconds=90

Case C: Customizing Local SSD Recovery Timeout (
--local-ssd-recovery-timeout
)

案例 C:自定义 Local SSD 恢复超时(
--local-ssd-recovery-timeout
)

When a
hostError
strikes a VM with attached Local SSDs, Compute Engine attempts to recover data from the failed host's NVMe drives while the VM stays offline, logging
compute.instances.hostError
with
operation.first="true"
when recovery begins and
operation.last="true"
when recovery completes or reaches
scheduling.localSsdRecoveryTimeout
.
  • Valid Range, Defaults & Trade-off: Valid values are
    0
    to
    168
    hours (up to 7 days)
    in 1-hour increments (see About host maintenance):
    • Defaults by Machine Series:
      • 168
        hours (7 days)
        :
        Z4M
        bare metal and
        Z3
        bare metal (
        z3-highmem-192-highlssd-metal
        ) instances
      • 6
        hours
        :
        Z4D
        ,
        Z4M
        , and
        Z3
        VMs
      • 1
        hour
        : All other instance types
    • Stateless Scratch / Cache Local SSDs: Set
      --local-ssd-recovery-timeout=0
      to skip NVMe data recovery and restart the VM immediately with blank Local SSDs.
    • Stateful Local SSD Data: Increase
      --local-ssd-recovery-timeout
      (up to
      168
      hours, e.g.,
      --local-ssd-recovery-timeout=24
      ) to give Compute Engine more time to recover attached Local SSD data before restarting.
sh
undefined
当挂载了 Local SSD 的 VM 遭遇
hostError
时,Compute Engine 会在 VM 保持离线期间尝试从故障宿主机的 NVMe 驱动器恢复数据; 恢复开始时记录
compute.instances.hostError
,且
operation.first="true"
; 恢复完成或达到
scheduling.localSsdRecoveryTimeout
时,
operation.last="true"
。
  • 有效范围、默认值与权衡:有效值为
    0
    到
    168
    小时 (最多 7 天)
    ,且必须以 1 小时为增量(请参阅 关于宿主维护):
    • 按机器系列划分的默认值:
      • 168
        小时(7 天)
        :
        Z4M
        裸金属和
        Z3
        裸金属 (
        z3-highmem-192-highlssd-metal
        )实例
      • 6
        小时
        :
        Z4D
        、
        Z4M
        和
        Z3
        VM
      • 1
        小时
        :其他所有实例类型
    • 无状态 Scratch / Cache Local SSD:设置
      --local-ssd-recovery-timeout=0
      ,以跳过 NVMe 数据恢复并立即重启 VM,且 Local SSD 为空白状态。
    • 有状态 Local SSD 数据:增大
      --local-ssd-recovery-timeout
      (最高为
      168
      小时,例如
      --local-ssd-recovery-timeout=24
      ), 让 Compute Engine 在重启前有更多时间恢复已挂载的 Local SSD 数据。
sh
undefined

Fast restart (0 hours wait) for stateless cache/scratch Local SSD workloads:

Fast restart (0 hours wait) for stateless cache/scratch Local SSD workloads:

gcloud compute instances set-scheduling {vm_name} --project={project_id} --zone={zone} --local-ssd-recovery-timeout=0
undefined
gcloud compute instances set-scheduling {vm_name} --project={project_id} --zone={zone} --local-ssd-recovery-timeout=0
undefined

Case D: Handling Spot / Preemptible VM Preemptions (
compute.instances.preempted
)

案例 D:处理 Spot / Preemptible VM 抢占(
compute.instances.preempted
)

When
compute.instances.preempted
appears in Cloud Logging, Compute Engine reclaimed a Spot VM or Preemptible VM by sending an ACPI G2 Soft Off signal.
  1. Best-Effort Preemption Window: The ACPI shutdown window provides up to 30 seconds on a best-effort basis (not guaranteed under severe physical capacity emergencies), so keep checkpointing scripts fast and idempotent and persist critical state continuously. See Spot VMs.
  2. Attach a
    shutdown-script
    & Check the Metadata Server
    : Query
    http://metadata.google.internal/computeMetadata/v1/instance/preempted
    with header
    Metadata-Flavor: Google
    (
    TRUE
    during preemption vs.
    FALSE
    on normal shutdown):
sh
gcloud compute instances add-metadata {vm_name} \
    --project={project_id} \
    --zone={zone} \
    --metadata=shutdown-script='#!/bin/bash
PREEMPTED=$(curl -s "http://metadata.google.internal/computeMetadata/v1/instance/preempted" -H "Metadata-Flavor: Google")
if [ "$PREEMPTED" = "TRUE" ]; then
  echo "Spot VM preemption detected; flushing checkpoints..."
  # Insert fast (<20s), idempotent checkpoint or drain command here
fi'
  1. Safe Rollback of Test Shutdown Metadata (Always Include):
    sh
    gcloud compute instances remove-metadata {vm_name} --project={project_id} --zone={zone} --keys=shutdown-script
当 Cloud Logging 中出现
compute.instances.preempted
时, 说明 Compute Engine 已通过发送 ACPI G2 Soft Off 信号, 回收了 Spot VM 或 Preemptible VM。
  1. 尽力而为的抢占窗口:ACPI 关闭窗口最多可提供 30 秒 (在严重的物理容量紧急情况下不保证提供),因此应确保检查点脚本快速且幂等, 并持续持久化关键状态。请参阅 Spot VM。
  2. 附加
    shutdown-script
    并检查元数据服务器
    :查询
    http://metadata.google.internal/computeMetadata/v1/instance/preempted
    , 并使用标头
    Metadata-Flavor: Google
    (抢占期间为
    TRUE
    ,正常关闭时为
    FALSE
    ):
sh
gcloud compute instances add-metadata {vm_name} \
    --project={project_id} \
    --zone={zone} \
    --metadata=shutdown-script='#!/bin/bash
PREEMPTED=$(curl -s "http://metadata.google.internal/computeMetadata/v1/instance/preempted" -H "Metadata-Flavor: Google")
if [ "$PREEMPTED" = "TRUE" ]; then
  echo "Spot VM preemption detected; flushing checkpoints..."
  # Insert fast (<20s), idempotent checkpoint or drain command here
fi'
  1. 安全回滚测试用关闭元数据(始终包含):
    sh
    gcloud compute instances remove-metadata {vm_name} --project={project_id} --zone={zone} --keys=shutdown-script

Case E: VM Lifecycle Events Dashboard & High-Availability SAP Workloads

案例 E:VM 生命周期事件仪表板与高可用 SAP 工作负载

  1. GCE VM Lifecycle Events Monitoring
    Dashboard (
    vm-lifecycle-events
    Log-Based Metric)
    : To populate the pre-generated
    GCE VM Lifecycle Events Monitoring
    dashboard in Cloud Monitoring (Monitoring > Dashboards > Dashboard List), create the
    vm-lifecycle-events
    counter metric using
    --config-from-file
    so the required
    method
    STRING
    label extractor on
    protoPayload.methodName
    is included (
    gcloud logging metrics create --log-filter
    alone omits label extractors):
    yaml
    # Save as vm-lifecycle-events-metric.yaml:
    name: vm-lifecycle-events
    description: "Count of Compute Engine VM lifecycle and host events by method"
    filter: >-
      resource.type="gce_instance" AND
      (log_id("cloudaudit.googleapis.com/activity") OR log_id("cloudaudit.googleapis.com/system_event")) AND
      operation.first="true"
    metricDescriptor:
      metricKind: DELTA
      valueType: INT64
      unit: "1"
      labels:
        - key: method
          valueType: STRING
          description: "Compute Engine lifecycle or host event method"
    labelExtractors:
      method: >-
        REGEXP_EXTRACT(protoPayload.methodName, "(recreateInstance|hostError|automaticRestart|guestTerminate|terminateOnHostMaintenance|preempted|insert|stop|delete|reset|start)")
    sh
    gcloud logging metrics create vm-lifecycle-events --project={project_id} --config-from-file=vm-lifecycle-events-metric.yaml
  2. SAP Workloads (HANA / NetWeaver): If the impacted VM runs SAP HANA or SAP NetWeaver, verify automatic Pacemaker cluster failover to the secondary node in Cloud Monitoring using the SAP HANA Availability (
    google_agent_for_sap.agent-for-sap-hana-availability-monitoring
    ) or SAP NetWeaver Availability (
    google_agent_for_sap.agent-for-sap-netweaver-availability-monitoring
    ) dashboards.

  1. GCE VM 生命周期事件监控仪表板(
    vm-lifecycle-events
    基于日志的指标)
    : 要在 Cloud Monitoring(Monitoring > Dashboards > Dashboard List)中填充 预生成的 GCE VM Lifecycle Events Monitoring 仪表板, 请使用
    --config-from-file
    创建
    vm-lifecycle-events
    计数器指标, 从而确保包含基于
    protoPayload.methodName
    的必需
    method
    STRING
    标签提取器(单独使用
    gcloud logging metrics create --log-filter
    会省略标签提取器):
    yaml
    # Save as vm-lifecycle-events-metric.yaml:
    name: vm-lifecycle-events
    description: "Count of Compute Engine VM lifecycle and host events by method"
    filter: >-
      resource.type="gce_instance" AND
      (log_id("cloudaudit.googleapis.com/activity") OR log_id("cloudaudit.googleapis.com/system_event")) AND
      operation.first="true"
    metricDescriptor:
      metricKind: DELTA
      valueType: INT64
      unit: "1"
      labels:
        - key: method
          valueType: STRING
          description: "Compute Engine lifecycle or host event method"
    labelExtractors:
      method: >-
        REGEXP_EXTRACT(protoPayload.methodName, "(recreateInstance|hostError|automaticRestart|guestTerminate|terminateOnHostMaintenance|preempted|insert|stop|delete|reset|start)")
    sh
    gcloud logging metrics create vm-lifecycle-events --project={project_id} --config-from-file=vm-lifecycle-events-metric.yaml
  2. SAP 工作负载(HANA / NetWeaver): 如果受影响的 VM 运行 SAP HANA 或 SAP NetWeaver, 请通过 Cloud Monitoring 中的 SAP HANA Availability (
    google_agent_for_sap.agent-for-sap-hana-availability-monitoring
    )或 SAP NetWeaver Availability (
    google_agent_for_sap.agent-for-sap-netweaver-availability-monitoring
    ) 仪表板,验证 Pacemaker 集群是否自动故障转移到辅助节点。

Verification, Rollback & Cleanup

验证、回滚与清理

  1. Verify Updated Scheduling Policy:
    sh
    gcloud compute instances describe {vm_name} --project={project_id} --zone={zone} \
        --format="yaml(scheduling, resourceStatus.upcomingMaintenance)"
  2. Remove Test Shutdown Metadata:
    sh
    gcloud compute instances remove-metadata {vm_name} --project={project_id} --zone={zone} --keys=shutdown-script
  3. Teardown Temporary Test VMs: Delete any temporary test VMs when finished (omitting
    --quiet
    on manual commands so interactive confirmation prevents accidental disk loss):
    sh
    gcloud compute instances delete {vm_name} --project={project_id} --zone={zone} --delete-disks=all
  4. Pricing Reference: Read-only troubleshooting queries create no billable VM resources (user-defined log-based metrics and Log Analytics queries on linked BigQuery datasets follow standard Cloud Monitoring and BigQuery pricing). Consult Google Cloud pricing and Compute Engine pricing for current Standard vs. Spot regional rates rather than hardcoded dollar figures.
  1. 验证更新后的调度策略:
    sh
    gcloud compute instances describe {vm_name} --project={project_id} --zone={zone} \
        --format="yaml(scheduling, resourceStatus.upcomingMaintenance)"
  2. 移除测试用关闭元数据:
    sh
    gcloud compute instances remove-metadata {vm_name} --project={project_id} --zone={zone} --keys=shutdown-script
  3. 拆除临时测试 VM:完成后删除所有临时测试 VM (手动命令中省略
    --quiet
    ,让交互式确认防止意外丢失磁盘):
    sh
    gcloud compute instances delete {vm_name} --project={project_id} --zone={zone} --delete-disks=all
  4. 定价参考:只读故障排查查询不会创建计费 VM 资源 (用户定义的基于日志的指标,以及在关联 BigQuery 数据集上执行的 Log Analytics 查询,遵循 Cloud Monitoring 和 BigQuery 的标准定价)。 请查阅 Google Cloud 定价和 Compute Engine 定价, 以获取当前 Standard 与 Spot 区域费率,而不是使用硬编码的美元金额。