compute-host-events-troubleshooting
Compare original and translation side by side
🇺🇸
Original
English🇨🇳
Translation
ChineseCompute Engine VM Host Events & Infrastructure Crash Troubleshooting
Compute Engine VM 宿主事件与基础设施崩溃故障排查
This skill diagnoses unexpected Google Compute Engine VM restarts,
terminations, and unresponsiveness caused by infrastructure host events
(, ,
, ,
) or guest-OS-initiated shutdowns
(). It provides authoritative Cloud Audit Log
queries, Log Analytics SQL aggregations, upcoming
maintenance window checks, and host scheduling remediations.
compute.instances.hostErrorcompute.instances.migrateOnHostMaintenancecompute.instances.terminateOnHostMaintenancecompute.instances.preemptedcompute.instances.automaticRestartcompute.instances.guestTerminate_Required._AllLogsMCP Tools:,get_instance_basic_info,list_instances,list_instance_attached_disks,start_instance,stop_instance,reset_instance,delete_instanceget_zone_operation
本 skill 诊断由基础设施宿主事件
(、、
、、
) 或来宾操作系统发起的关闭
() 导致的 Google Compute Engine VM 意外重启、
终止和失去响应。它提供权威的 Cloud Audit Log
查询、Log Analytics SQL 聚合、即将进行的
维护窗口检查,以及宿主调度补救措施。
compute.instances.hostErrorcompute.instances.migrateOnHostMaintenancecompute.instances.terminateOnHostMaintenancecompute.instances.preemptedcompute.instances.automaticRestartcompute.instances.guestTerminate_Required._AllLogsMCP 工具:、get_instance_basic_info、list_instances、list_instance_attached_disks、start_instance、stop_instance、reset_instance、delete_instanceget_zone_operation
Usage Attribution
使用归属说明
MUST READ: Before executing any , REST API, or MCP command, you
MUST read and follow the instructions in
Usage Attribution.
gcloud必须阅读:在执行任何 、REST API 或 MCP 命令前,你
必须阅读并遵循
使用归属说明中的说明。
gcloudPrerequisites & Environment Setup
前置条件与环境设置
Any host or project modifications (installing or updating , configuring
the Compute Engine MCP server, enabling APIs or Log Analytics, mutating VM
scheduling/metadata, creating log-based metrics, or deleting resources) must be
explicitly approved by the user before execution.
gcloud- Google Cloud SDK: Verify is installed (
gcloud). If missing or outdated, ask the user for approval before installing via Install the gcloud CLI.gcloud --version - Compute Engine MCP Server & Capability Boundary:
- The Compute Engine MCP server
(Use the Compute Engine MCP server)
exposes zonal VM discovery and lifecycle tools (,
get_instance_basic_info,list_instances,list_instance_attached_disks,start_instance,stop_instance,reset_instance,delete_instance, requiringget_zone_operation).roles/mcp.toolUser - Critical Routing Rule: returns a reduced
get_instance_basic_infoprojection (InstanceBasicInfo,name,id,status,machine_type,create_time) that omitsguest_acceleratorsandscheduling, and Compute Engine MCP does not expose tools for Cloud Logging queries, serial console output,resourceStatus.upcomingMaintenance,set-scheduling, or instance metadata updates. Always route full scheduling inspection, upcoming maintenance checks, serial port output, audit log queries, and scheduling/metadata mutations toperform-maintenance(or Compute Engine / Cloud Logging REST APIs).gcloud
- The Compute Engine MCP server
(Use the Compute Engine MCP server)
exposes zonal VM discovery and lifecycle tools (
- Authentication:
- Local CLI:
gcloud auth login - Application Default Credentials:
gcloud auth application-default login - Service Accounts: Set to your service account credentials path.
GOOGLE_APPLICATION_CREDENTIALS
- Local CLI:
- Project, Region & Billing:
Note: An active billing account attachment is required (even when evaluating Free Tier eligible resources). If no region is specified, default tosh
gcloud config set project {project_id} gcloud config set compute/region {region} gcloud config set compute/zone {zone}(us-central1), and check Google Cloud locations for regional hardware availability.us-central1-a - API Enablement:
sh
gcloud services enable compute.googleapis.com logging.googleapis.com monitoring.googleapis.com - Required IAM Roles:
- Read-Only Diagnostics: (instance scheduling and serial console output),
roles/compute.viewer(orroles/logging.viewerto readroles/logging.privateLogViewerandsystem_eventaudit logs), andactivityfor Log Analytics queries in Cloud Logging (plusroles/logging.viewAccessorandroles/bigquery.jobUserwhen querying a linked Log Analytics dataset from BigQuery Studio orroles/bigquery.dataViewer).bq - Remediation & MCP: (to run
roles/compute.instanceAdmin.v1,set-scheduling,perform-maintenance, oradd-metadata),remove-metadata(to create log-based metrics), androles/logging.configWriter(for Compute Engine MCP calls).roles/mcp.toolUser
- Read-Only Diagnostics:
任何宿主或项目层面的修改(安装或更新 、配置
Compute Engine MCP server、启用 API 或 Log Analytics、修改 VM
调度/元数据、创建基于日志的指标,或删除资源)都必须
在执行前获得用户明确批准。
gcloud- Google Cloud SDK:确认已安装 (
gcloud)。如果 缺失或版本过旧,请在通过 安装 gcloud CLI安装前征得用户批准。gcloud --version - Compute Engine MCP server 与能力边界:
- Compute Engine MCP server
(使用 Compute Engine MCP server)
暴露可用区 VM 发现与生命周期工具(、
get_instance_basic_info、list_instances、list_instance_attached_disks、start_instance、stop_instance、reset_instance、delete_instance,需要get_zone_operation)。roles/mcp.toolUser - 关键路由规则:返回精简后的
get_instance_basic_info投影(InstanceBasicInfo、name、id、status、machine_type、create_time),其中不包含guest_accelerators和scheduling;Compute Engine MCP 也不提供 Cloud Logging 查询、串口控制台输出、resourceStatus.upcomingMaintenance、set-scheduling或实例元数据更新工具。 始终将完整的调度检查、即将进行的维护检查、串口输出、 审计日志查询以及调度/元数据变更路由至perform-maintenance(或 Compute Engine / Cloud Logging REST API)。gcloud
- Compute Engine MCP server
(使用 Compute Engine MCP server)
暴露可用区 VM 发现与生命周期工具(
- 身份验证:
- 本地 CLI:
gcloud auth login - Application Default Credentials:
gcloud auth application-default login - Service Accounts:将 设置为你的 Service Account 凭据路径。
GOOGLE_APPLICATION_CREDENTIALS
- 本地 CLI:
- 项目、区域与结算:
注意:必须关联有效的结算账号(即使是在评估符合 Free Tier 条件的资源时)。如果未指定区域,请默认使用sh
gcloud config set project {project_id} gcloud config set compute/region {region} gcloud config set compute/zone {zone}(us-central1),并查看 Google Cloud 区域以了解区域硬件可用性。us-central1-a - 启用 API:
sh
gcloud services enable compute.googleapis.com logging.googleapis.com monitoring.googleapis.com - 必需的 IAM 角色:
- 只读诊断:(实例调度和串口控制台输出)、
roles/compute.viewer(或使用roles/logging.viewer读取roles/logging.privateLogViewer和system_event审计日志),以及用于在 Cloud Logging 中执行 Log Analytics 查询的activity(从 BigQuery Studio 或roles/logging.viewAccessor查询关联的 Log Analytics 数据集时,还需要bq和roles/bigquery.jobUser)。roles/bigquery.dataViewer - 补救与 MCP:(用于运行
roles/compute.instanceAdmin.v1、set-scheduling、perform-maintenance或add-metadata)、remove-metadata(用于创建基于日志的指标), 以及roles/logging.configWriter(用于 Compute Engine MCP 调用)。roles/mcp.toolUser
- 只读诊断:
Diagnostic Workflow: Classify the Termination Event
诊断工作流:终止事件分类
When a user reports an unexpected VM reboot, stop, or crash, run targeted Cloud
Logging queries against (Compute Engine
infrastructure events) and
(caller-initiated API stops/resets). Always pass (or a window
covering the incident timestamp) because defaults
to and silently returns zero rows for incidents older than 24
hours. See
Troubleshooting VM shutdowns and reboots.
cloudaudit.googleapis.com/system_eventcloudaudit.googleapis.com/activity--freshness=7dgcloud logging read--freshness1d当用户报告 VM 意外重启、停止或崩溃时,针对
(Compute Engine
基础设施事件)和
(调用方发起的 API 停止/重置)运行定向 Cloud Logging 查询。
始终传入 (或一个涵盖事件时间戳的时间窗口),
因为 默认将 设为 ,
对于超过 24 小时的事件会静默返回零行。请参阅
排查 VM 关闭和重启问题。
cloudaudit.googleapis.com/system_eventcloudaudit.googleapis.com/activity--freshness=7dgcloud logging read--freshness1dKey Distinction: hostError
vs. Planned Maintenance vs. preempted
hostErrorpreempted关键区别:hostError
vs. 计划内维护 vs. preempted
hostErrorpreempted- (Unplanned Hardware/Hypervisor Crash): Indicates an unexpected hardware or hypervisor failure on the physical server hosting the VM. If
compute.instances.hostErrorisscheduling.automaticRestart(true), Compute Engine automatically restarts the VM on a healthy physical host (typically within 3 to 5.5 minutes from detection on standard VMs, logging--restart-on-failure). On VMs with attached Local SSDs, Compute Engine logscompute.instances.automaticRestartwithcompute.instances.hostErrorwhen Local SSD recovery begins andoperation.first="true"when recovery completes or reachesoperation.last="true".scheduling.localSsdRecoveryTimeout - /
compute.instances.migrateOnHostMaintenance(Planned Infrastructure Maintenance): Compute Engine logs these exactcompute.instances.terminateOnHostMaintenancemethod names (along withsystem_eventnotifications) during scheduled hardware or hypervisor updates. Depending oncompute.instances.upcomingMaintenance, the VM either transparently live-migrates (onHostMaintenance->MIGRATE) or terminates (compute.instances.migrateOnHostMaintenance->TERMINATE, followed bycompute.instances.terminateOnHostMaintenanceifcompute.instances.automaticRestart).automaticRestart: true - (Spot / Preemptible VM Reclaimed): Indicates Compute Engine reclaimed capacity from a Spot VM or Preemptible VM.
compute.instances.preempted
- (计划外硬件/Hypervisor 崩溃): 表示承载 VM 的物理服务器发生意外硬件或 Hypervisor 故障。 如果
compute.instances.hostError为scheduling.automaticRestart(true),Compute Engine 会自动将 VM 重启到 健康的物理宿主上(对于标准 VM,通常在检测后的 3 到 5.5 分钟内完成, 并记录--restart-on-failure)。对于挂载 Local SSD 的 VM, 当 Local SSD 恢复开始时,Compute Engine 会记录compute.instances.automaticRestart,且compute.instances.hostError; 当恢复完成或达到operation.first="true"时,scheduling.localSsdRecoveryTimeout。operation.last="true" - /
compute.instances.migrateOnHostMaintenance(计划内基础设施维护): 在计划执行的硬件或 Hypervisor 更新期间,Compute Engine 会记录这些确切的compute.instances.terminateOnHostMaintenance方法名称(同时还会发出system_event通知)。根据compute.instances.upcomingMaintenance的配置,VM 可能会透明地实时迁移(onHostMaintenance->MIGRATE),也可能会被终止 (compute.instances.migrateOnHostMaintenance->TERMINATE, 如果compute.instances.terminateOnHostMaintenance,随后会记录automaticRestart: true)。compute.instances.automaticRestart - (Spot / Preemptible VM 被回收): 表示 Compute Engine 从 Spot VM 或 Preemptible VM 回收了容量。
compute.instances.preempted
1. Query Infrastructure Host Events & Caller Stops in Cloud Logging
1. 在 Cloud Logging 中查询基础设施宿主事件与调用方停止操作
Run this command to determine whether
experienced a physical host error, planned maintenance, Spot preemption, MIG
recreation, or an external caller-initiated API stop/reset:
gcloud logging read{vm_name}sh
gcloud logging read 'resource.type="gce_instance" AND (log_id("cloudaudit.googleapis.com/system_event") OR log_id("cloudaudit.googleapis.com/activity")) AND protoPayload.methodName=~"(compute\.instances\.(hostError|migrateOnHostMaintenance|terminateOnHostMaintenance|preempted|automaticRestart|upcomingMaintenance|guestTerminate|repair.*)|v1\.compute\.instances\.(stop|reset|delete|performMaintenance|simulateMaintenanceEvent))" AND protoPayload.resourceName:"{vm_name}"' \
--project={project_id} \
--freshness=7d \
--limit=25 \
--format="table(timestamp, logName.basename(), protoPayload.methodName, protoPayload.authenticationInfo.principalEmail, operation.first, operation.last, protoPayload.status.message)"运行以下 命令,以判断
是否发生了物理宿主错误、计划内维护、Spot 抢占、MIG
重建,或外部调用方发起的 API 停止/重置:
gcloud logging read{vm_name}sh
gcloud logging read 'resource.type="gce_instance" AND (log_id("cloudaudit.googleapis.com/system_event") OR log_id("cloudaudit.googleapis.com/activity")) AND protoPayload.methodName=~"(compute\.instances\.(hostError|migrateOnHostMaintenance|terminateOnHostMaintenance|preempted|automaticRestart|upcomingMaintenance|guestTerminate|repair.*)|v1\.compute\.instances\.(stop|reset|delete|performMaintenance|simulateMaintenanceEvent))" AND protoPayload.resourceName:"{vm_name}"' \
--project={project_id} \
--freshness=7d \
--limit=25 \
--format="table(timestamp, logName.basename(), protoPayload.methodName, protoPayload.authenticationInfo.principalEmail, operation.first, operation.last, protoPayload.status.message)"2. Check Specifically for Guest OS Initiated Terminations (guestTerminate
)
guestTerminate2. 专门检查来宾操作系统发起的终止(guestTerminate
)
guestTerminateIf the VM stopped without a , maintenance termination, preemption, or
user-initiated Admin Activity log, check whether the
shutdown was initiated from inside the Guest OS (,
, , or an internal watchdog script):
hostErrorv1.compute.instances.stopsudo shutdown -h nowpoweroffsystemctl poweroffsh
gcloud logging read 'resource.type="gce_instance" AND log_id("cloudaudit.googleapis.com/system_event") AND protoPayload.methodName="compute.instances.guestTerminate" AND protoPayload.resourceName:"{vm_name}"' \
--project={project_id} \
--freshness=7d \
--limit=20 \
--format="yaml(timestamp, protoPayload.methodName, protoPayload.authenticationInfo.principalEmail, protoPayload.status)"- Critical Attribution Nuance: Because
guestTerminateis acompute.instances.guestTerminategenerated when the guest ACPI power-off reaches the hypervisor, Compute Engine always records its generic built-in system principal insystem_event— it never identifies which Linux/Windows user, cron job, or script inside the VM ranprotoPayload.authenticationInfo.principalEmail/shutdown(whereas external caller-initiatedpoweroff,v1.compute.instances.stop, orresetentries indeleterecord the external IAM user or service account incloudaudit.googleapis.com/activity).principalEmail - How to Find the In-Guest Culprit: Inspect the VM's serial console output
and guest OS logs from the previous boot (,
journalctl -b -1or/var/log/syslog,/var/log/messages, and/var/log/auth.log):last -x shutdown rebootshgcloud compute instances get-serial-port-output {vm_name} --project={project_id} --zone={zone} --port=1
如果 VM 在没有 、维护终止、抢占或用户发起的
Admin Activity 日志的情况下停止,请检查
关闭操作是否从 Guest OS 内部发起(、
、 或内部 watchdog 脚本):
hostErrorv1.compute.instances.stopsudo shutdown -h nowpoweroffsystemctl poweroffsh
gcloud logging read 'resource.type="gce_instance" AND log_id("cloudaudit.googleapis.com/system_event") AND protoPayload.methodName="compute.instances.guestTerminate" AND protoPayload.resourceName:"{vm_name}"' \
--project={project_id} \
--freshness=7d \
--limit=20 \
--format="yaml(timestamp, protoPayload.methodName, protoPayload.authenticationInfo.principalEmail, protoPayload.status)"- 归属判定的重要细节:由于
guestTerminate是在来宾操作系统的 ACPI 关机信号到达 Hypervisor 时生成的compute.instances.guestTerminate,因此 Compute Engine 始终只会在system_event中记录其 通用的内置系统主体,而绝不会标识是 VM 内的哪个 Linux/Windows 用户、cron 任务或脚本执行了protoPayload.authenticationInfo.principalEmail/shutdown;相比之下,由外部调用方发起的poweroff、v1.compute.instances.stop或reset记录会出现在delete中,并在cloudaudit.googleapis.com/activity中记录外部 IAM 用户或 Service Account。principalEmail - 如何查找 Guest OS 内部的原因:检查 VM 的串口控制台输出和上一次启动期间的
Guest OS 日志(、
journalctl -b -1或/var/log/syslog、/var/log/messages,以及/var/log/auth.log):last -x shutdown rebootshgcloud compute instances get-serial-port-output {vm_name} --project={project_id} --zone={zone} --port=1
3. Log Analytics SQL Query: Aggregate Host Events Across the Project
3. Log Analytics SQL 查询:聚合整个项目中的宿主事件
Compute Engine host events (, ,
, , ,
) are logs. By default,
Cloud Logging's sink excludes logs and routes them
exclusively to the log bucket.
hostErrormigrateOnHostMaintenanceterminateOnHostMaintenancepreemptedautomaticRestartguestTerminatecloudaudit.googleapis.com/system_event_Defaultsystem_event_RequiredPrerequisites forSQL:_Required._AllLogs
- Upgrade the
bucket to Log Analytics (seek user approval first):_Requiredshgcloud logging buckets update _Required --project={project_id} --location=global --enable-analytics- In Log Analytics
schemas,_AllLogshas BigQuery typeresource.labels(notJSON). Always extract fields withSTRUCTandJSON_VALUE(resource.labels.zone)— grouping directly on rawJSON_VALUE(resource.labels.instance_id)expressions fails withJSON.Grouping by expressions of type JSON is not allowed
sql
SELECT
JSON_VALUE(resource.labels.zone) AS zone,
JSON_VALUE(resource.labels.instance_id) AS instance_id,
REGEXP_EXTRACT(proto_payload.audit_log.resource_name, r'[^/]+$') AS vm_name,
proto_payload.audit_log.method_name AS event_type,
COUNT(*) AS event_count
FROM `{project_id}.global._Required._AllLogs`
WHERE
resource.type = 'gce_instance'
AND timestamp >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)
AND proto_payload.audit_log.method_name IN (
'compute.instances.hostError',
'compute.instances.migrateOnHostMaintenance',
'compute.instances.terminateOnHostMaintenance',
'compute.instances.preempted',
'compute.instances.automaticRestart',
'compute.instances.guestTerminate'
)
GROUP BY zone, instance_id, vm_name, event_type
ORDER BY event_count DESC;Compute Engine 宿主事件(、、
、、、
)属于 日志。
默认情况下,Cloud Logging 的 sink 会排除
日志,并将其仅路由到 日志存储桶。
hostErrormigrateOnHostMaintenanceterminateOnHostMaintenancepreemptedautomaticRestartguestTerminatecloudaudit.googleapis.com/system_event_Defaultsystem_event_RequiredSQL 的前置条件:_Required._AllLogs
- 将
存储桶升级为 Log Analytics(先征得用户批准):_Requiredshgcloud logging buckets update _Required --project={project_id} --location=global --enable-analytics- 在 Log Analytics
Schema 中,_AllLogs的 BigQuery 类型为resource.labels(而不是JSON)。始终使用STRUCT和JSON_VALUE(resource.labels.zone)提取字段——直接按原始JSON_VALUE(resource.labels.instance_id)表达式分组会失败并提示JSON。Grouping by expressions of type JSON is not allowed
sql
SELECT
JSON_VALUE(resource.labels.zone) AS zone,
JSON_VALUE(resource.labels.instance_id) AS instance_id,
REGEXP_EXTRACT(proto_payload.audit_log.resource_name, r'[^/]+$') AS vm_name,
proto_payload.audit_log.method_name AS event_type,
COUNT(*) AS event_count
FROM `{project_id}.global._Required._AllLogs`
WHERE
resource.type = 'gce_instance'
AND timestamp >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)
AND proto_payload.audit_log.method_name IN (
'compute.instances.hostError',
'compute.instances.migrateOnHostMaintenance',
'compute.instances.terminateOnHostMaintenance',
'compute.instances.preempted',
'compute.instances.automaticRestart',
'compute.instances.guestTerminate'
)
GROUP BY zone, instance_id, vm_name, event_type
ORDER BY event_count DESC;4. Inspect Upcoming Scheduled Host Maintenance & Trigger Early Maintenance
4. 检查即将执行的计划宿主维护并提前触发维护
For instances that emit advance maintenance notifications (such as ,
, , , , , GPU instances, bare metal
instances, / Cloud RDMA instances, instances, storage-optimized
/ / Local SSD instances, or Cloud TPU nodes), inspect upcoming
maintenance via or the in-guest metadata server. See
Monitor and plan for a host maintenance event
and
Get live migration notices from the metadata server:
A4X MaxA4XA4A3A2G4G2-metalH4DX4Z3Z4DZ4Mgcloudsh
undefined对于会提前发出维护通知的实例(例如 、
、、、、、 GPU 实例、裸金属
实例、 / Cloud RDMA 实例、 实例、面向存储优化的
/ / Local SSD 实例,或 Cloud TPU 节点),请通过
或 Guest OS 内的元数据服务器检查即将进行的维护。
请参阅
监视并规划宿主维护事件和
从元数据服务器获取实时迁移通知:
A4X MaxA4XA4A3A2G4G2-metalH4DX4Z3Z4DZ4Mgcloudsh
undefined1. Check a single instance for scheduling policy and upcoming maintenance (Compute Engine API):
1. Check a single instance for scheduling policy and upcoming maintenance (Compute Engine API):
gcloud compute instances describe {vm_name} --project={project_id} --zone={zone}
--format="yaml(name, scheduling, resourceStatus.upcomingMaintenance)"
--format="yaml(name, scheduling, resourceStatus.upcomingMaintenance)"
gcloud compute instances describe {vm_name} --project={project_id} --zone={zone}
--format="yaml(name, scheduling, resourceStatus.upcomingMaintenance)"
--format="yaml(name, scheduling, resourceStatus.upcomingMaintenance)"
2. List upcoming maintenance windows across all instances in the project:
2. List upcoming maintenance windows across all instances in the project:
gcloud compute instances list --project={project_id}
--format="table(name, zone.basename(), resourceStatus.upcomingMaintenance.type, resourceStatus.upcomingMaintenance.windowStartTime, resourceStatus.upcomingMaintenance.windowEndTime, resourceStatus.upcomingMaintenance.canReschedule, resourceStatus.upcomingMaintenance.maintenanceStatus)"
--format="table(name, zone.basename(), resourceStatus.upcomingMaintenance.type, resourceStatus.upcomingMaintenance.windowStartTime, resourceStatus.upcomingMaintenance.windowEndTime, resourceStatus.upcomingMaintenance.canReschedule, resourceStatus.upcomingMaintenance.maintenanceStatus)"
gcloud compute instances list --project={project_id}
--format="table(name, zone.basename(), resourceStatus.upcomingMaintenance.type, resourceStatus.upcomingMaintenance.windowStartTime, resourceStatus.upcomingMaintenance.windowEndTime, resourceStatus.upcomingMaintenance.canReschedule, resourceStatus.upcomingMaintenance.maintenanceStatus)"
--format="table(name, zone.basename(), resourceStatus.upcomingMaintenance.type, resourceStatus.upcomingMaintenance.windowStartTime, resourceStatus.upcomingMaintenance.windowEndTime, resourceStatus.upcomingMaintenance.canReschedule, resourceStatus.upcomingMaintenance.maintenanceStatus)"
3. Query from inside the Guest OS via the Metadata Server:
3. Query from inside the Guest OS via the Metadata Server:
curl -s "http://metadata.google.internal/computeMetadata/v1/instance/upcoming-maintenance?alt=json"
-H "Metadata-Flavor: Google"
-H "Metadata-Flavor: Google"
- **API vs. Metadata Server JSON Field Name Difference**:
- In `gcloud` / Compute Engine REST (`resourceStatus.upcomingMaintenance`) and
Cloud Logging, the maintenance classification field is named **`type`**
(`SCHEDULED` or `UNSCHEDULED`), alongside `maintenanceStatus` (`PENDING` or
`ONGOING`), `canReschedule` (`true` or `false`), `windowStartTime`,
`windowEndTime`, and `latestWindowStartTime`.
- When querying `/computeMetadata/v1/instance/upcoming-maintenance?alt=json`
inside the guest OS, returns `"NONE"` when no maintenance is scheduled, or a
JSON object where the classification property is named **`maintenanceType`**
(e.g., `"maintenanceType": "SCHEDULED"`).
- **Manually Trigger Host Maintenance Early (`perform-maintenance`)**:
- **Preconditions**: Check `canReschedule` and `maintenanceStatus` first. You
can run `perform-maintenance` when `canReschedule` is `true` (and
`maintenanceStatus` is `PENDING` when an active notice exists, or on
supported machine series before a notice is posted to reset the maintenance
cycle). If an active notice reports `canReschedule: false` or
`maintenanceStatus: ONGOING`, manual triggering is rejected by the API.
- **Immediate Impact**: Confirm with the user before running —
`perform-maintenance` starts host maintenance **immediately** (terminating
and optionally restarting `TERMINATE`-policy instances right away). See
[Simulate or trigger a host maintenance event](https://docs.cloud.google.com/compute/docs/instances/trigger-host-maintenance-event.md.txt):
```sh
gcloud compute instances perform-maintenance {vm_name} --project={project_id} --zone={zone}curl -s "http://metadata.google.internal/computeMetadata/v1/instance/upcoming-maintenance?alt=json"
-H "Metadata-Flavor: Google"
-H "Metadata-Flavor: Google"
- **API 与元数据服务器之间的 JSON 字段名差异**:
- 在 `gcloud` / Compute Engine REST(`resourceStatus.upcomingMaintenance`)和
Cloud Logging 中,维护分类字段名为 **`type`**
(`SCHEDULED` 或 `UNSCHEDULED`),并且还包含
`maintenanceStatus`(`PENDING` 或 `ONGOING`)、`canReschedule`
(`true` 或 `false`)、`windowStartTime`、
`windowEndTime` 和 `latestWindowStartTime`。
- 在 Guest OS 内查询
`/computeMetadata/v1/instance/upcoming-maintenance?alt=json` 时,
如果没有计划维护,则返回 `"NONE"`;否则返回一个 JSON 对象,
其分类属性名为 **`maintenanceType`**
(例如 `"maintenanceType": "SCHEDULED"`)。
- **手动提前触发宿主维护(`perform-maintenance`)**:
- **前置条件**:先检查 `canReschedule` 和 `maintenanceStatus`。
当 `canReschedule` 为 `true` 时可以运行 `perform-maintenance`
(如果存在有效通知,`maintenanceStatus` 应为 `PENDING`;
或者对于受支持的机器系列,可以在通知发布前运行,以重置维护周期)。
如果有效通知显示 `canReschedule: false` 或
`maintenanceStatus: ONGOING`,API 将拒绝手动触发。
- **即时影响**:运行前必须与用户确认——
`perform-maintenance` 会**立即**开始宿主维护(立即终止采用 `TERMINATE`
策略的实例,并可选地重启这些实例)。请参阅
[模拟或触发宿主维护事件](https://docs.cloud.google.com/compute/docs/instances/trigger-host-maintenance-event.md.txt):
```sh
gcloud compute instances perform-maintenance {vm_name} --project={project_id} --zone={zone}Actionable Remediation & Hardening Rules
可执行的补救与加固规则
Required IAM Role: Modifying instance scheduling or metadata requires
. Always obtain user confirmation before
executing mutating commands.
roles/compute.instanceAdmin.v1所需 IAM 角色:修改实例调度或元数据需要
。执行变更命令前始终必须获得用户确认。
roles/compute.instanceAdmin.v1Case A: Preventing Downtime During Planned Maintenance (MIGRATE
Policy)
MIGRATE案例 A:在计划内维护期间防止停机(MIGRATE
策略)
MIGRATEIf a live-migration-capable VM (such as an , , , , , or
instance; note that standard VMs only support during
host maintenance events and disallow ) logged
during routine host maintenance,
its policy was set to instead of
.
N2N2DC3C3DC4N4E2MIGRATETERMINATEcompute.instances.terminateOnHostMaintenancescheduling.onHostMaintenanceTERMINATEMIGRATE- Fix: Enable Live Migration () and Automatic Restart (
--maintenance-policy=MIGRATE, which sets REST--restart-on-failure; usescheduling.automaticRestart: trueto disable). Per Set VM host maintenance policy,--no-restart-on-failureupdatesgcloud compute instances set-scheduling,onHostMaintenance(except on running CSEK-encrypted VMs),automaticRestart, andhostErrorTimeoutSecondsin-place on a running VM without stopping or rebooting it (note: although thelocalSsdRecoveryTimeout/ RESTgcloud compute instances set-scheduling --helpreference docstring contains a legacy note stating the method requires a stoppedinstances.setSchedulinginstance, stopping the VM first is only required when changingTERMINATED/--provisioning-modelor updating--preemptibleon CSEK-encrypted VMs):automaticRestart
sh
gcloud compute instances set-scheduling {vm_name} \
--project={project_id} \
--zone={zone} \
--maintenance-policy=MIGRATE \
--restart-on-failure- VM Families & Configurations That Do Not Support Live Migration (Always
Clarify):
Whenever recommending , explicitly note which VM configurations cannot live-migrate and instead require
--maintenance-policy=MIGRATEpaired with--maintenance-policy=TERMINATE(except Spot/Preemptible VMs, which require--restart-on-failurewithTERMINATE), and note that standard--no-restart-on-failureVMs only supportE2:MIGRATE- Spot VMs () and legacy Preemptible VMs
provisioningModel: SPOT - GPU-attached VMs (,
A4X Max,A4X,A4,A3,A2,G4, andG2with attached GPUs; note thatN1does not support attached GPUs)N2 - Cloud TPU VMs and Bare Metal Instances ()
*-metal - Instances with a Cloud RDMA network interface (such as and
H4Dwith Cloud RDMA)Z4M - and
X4memory-optimized instancesX5 - Large Local SSD Configurations (instances with > 18,000 GiB of Local SSD, and
Z3/Z4Dinstances with > 42,000 GiB of Local SSD)Z4M
- Spot VMs (
如果支持实时迁移的 VM(例如 、、、、 或
实例;请注意,标准 VM 仅支持宿主维护事件期间的
,不允许使用 )在例行宿主维护期间记录了
,则说明其
策略被设置为 ,而不是 。
N2N2DC3C3DC4N4E2MIGRATETERMINATEcompute.instances.terminateOnHostMaintenancescheduling.onHostMaintenanceTERMINATEMIGRATE- 修复:启用 Live Migration()和 Automatic Restart(
--maintenance-policy=MIGRATE,这会将 REST--restart-on-failure设置为scheduling.automaticRestart;使用true可将其禁用)。根据 设置 VM 宿主维护策略,--no-restart-on-failure可以更新gcloud compute instances set-scheduling、onHostMaintenance(运行中的 CSEK 加密 VM 除外)、automaticRestart和hostErrorTimeoutSeconds, 无需停止或重启正在运行的 VM (请注意,虽然localSsdRecoveryTimeout/ RESTgcloud compute instances set-scheduling --help参考文档字符串中包含一条旧版说明,称该方法要求实例 处于已停止的instances.setScheduling状态,但只有更改TERMINATED/--provisioning-model,或更新 CSEK 加密 VM 的--preemptible时,才需要先停止 VM):automaticRestart
sh
gcloud compute instances set-scheduling {vm_name} \
--project={project_id} \
--zone={zone} \
--maintenance-policy=MIGRATE \
--restart-on-failure- 不支持 Live Migration 的 VM 系列与配置(始终说明)
每当建议使用 时,必须明确说明哪些 VM 配置无法实时迁移、因而必须使用
--maintenance-policy=MIGRATE搭配--maintenance-policy=TERMINATE(Spot/Preemptible VM 除外,它们必须使用--restart-on-failure搭配TERMINATE),并说明标准--no-restart-on-failureVM 仅支持E2:MIGRATE- Spot VM()和旧版 Preemptible VM
provisioningModel: SPOT - 挂载 GPU 的 VM(、
A4X Max、A4X、A4、A3、A2、G4, 以及挂载 GPU 的G2;请注意,N1不支持挂载 GPU)N2 - Cloud TPU VM 和裸金属实例()
*-metal - 使用 Cloud RDMA 网络接口的实例(例如使用 Cloud RDMA 的 和
H4D)Z4M - 和
X4内存优化实例X5 - 大型 Local SSD 配置(Local SSD 超过 18,000 GiB 的 实例, 以及 Local SSD 超过 42,000 GiB 的
Z3/Z4D实例)Z4M
- Spot VM(
Case B: Faster Recovery from Unresponsive Hosts (--host-error-timeout-seconds
)
--host-error-timeout-seconds案例 B:加快无响应宿主的恢复速度(--host-error-timeout-seconds
)
--host-error-timeout-secondsIf an application experiences prolonged unresponsiveness before Compute Engine's
default host watchdog declares a , shorten the detection timeout via
:
hostError--host-error-timeout-seconds- Constraints: Must be specified in 30-second increments. Valid range is
to
90seconds in300(gcloudto90seconds via Compute Engine REST330). Unsupported on Sole-Tenant VMs and Spot / Preemptible VMs. Updates dynamically on a running VM (see Set VM host maintenance policy):instances.setScheduling
sh
gcloud compute instances set-scheduling {vm_name} --project={project_id} --zone={zone} --host-error-timeout-seconds=90如果应用在 Compute Engine 的默认宿主 watchdog 判定
之前长时间无响应,请通过 缩短检测超时:
hostError--host-error-timeout-seconds- 约束:必须以 30 秒为增量指定。在 中的有效范围是
gcloud到90秒(通过 Compute Engine REST300设置时为instances.setScheduling到90秒)。Sole-Tenant VM 和 Spot / Preemptible VM 不支持此设置。可在运行中的 VM 上动态更新 (请参阅 设置 VM 宿主维护策略):330
sh
gcloud compute instances set-scheduling {vm_name} --project={project_id} --zone={zone} --host-error-timeout-seconds=90Case C: Customizing Local SSD Recovery Timeout (--local-ssd-recovery-timeout
)
--local-ssd-recovery-timeout案例 C:自定义 Local SSD 恢复超时(--local-ssd-recovery-timeout
)
--local-ssd-recovery-timeoutWhen a strikes a VM with attached Local SSDs, Compute Engine
attempts to recover data from the failed host's NVMe drives while the VM stays
offline, logging with
when recovery begins and when recovery completes or
reaches .
hostErrorcompute.instances.hostErroroperation.first="true"operation.last="true"scheduling.localSsdRecoveryTimeout- Valid Range, Defaults & Trade-off: Valid values are to
0hours (up to 7 days) in 1-hour increments (see About host maintenance):168- Defaults by Machine Series:
- hours (7 days):
168bare metal andZ4Mbare metal (Z3) instancesz3-highmem-192-highlssd-metal - hours:
6,Z4D, andZ4MVMsZ3 - hour: All other instance types
1
- Stateless Scratch / Cache Local SSDs: Set
to skip NVMe data recovery and restart the VM immediately with blank Local SSDs.
--local-ssd-recovery-timeout=0 - Stateful Local SSD Data: Increase (up to
--local-ssd-recovery-timeouthours, e.g.,168) to give Compute Engine more time to recover attached Local SSD data before restarting.--local-ssd-recovery-timeout=24
- Defaults by Machine Series:
sh
undefined当挂载了 Local SSD 的 VM 遭遇 时,Compute Engine
会在 VM 保持离线期间尝试从故障宿主机的 NVMe 驱动器恢复数据;
恢复开始时记录 ,且 ;
恢复完成或达到 时,
。
hostErrorcompute.instances.hostErroroperation.first="true"scheduling.localSsdRecoveryTimeoutoperation.last="true"- 有效范围、默认值与权衡:有效值为 到
0小时 (最多 7 天),且必须以 1 小时为增量(请参阅 关于宿主维护):168- 按机器系列划分的默认值:
- 小时(7 天):
168裸金属和Z4M裸金属 (Z3)实例z3-highmem-192-highlssd-metal - 小时:
6、Z4D和Z4MVMZ3 - 小时:其他所有实例类型
1
- 无状态 Scratch / Cache Local SSD:设置
,以跳过 NVMe 数据恢复并立即重启 VM,且 Local SSD 为空白状态。
--local-ssd-recovery-timeout=0 - 有状态 Local SSD 数据:增大 (最高为
--local-ssd-recovery-timeout小时,例如168), 让 Compute Engine 在重启前有更多时间恢复已挂载的 Local SSD 数据。--local-ssd-recovery-timeout=24
- 按机器系列划分的默认值:
sh
undefinedFast restart (0 hours wait) for stateless cache/scratch Local SSD workloads:
Fast restart (0 hours wait) for stateless cache/scratch Local SSD workloads:
gcloud compute instances set-scheduling {vm_name} --project={project_id} --zone={zone} --local-ssd-recovery-timeout=0
undefinedgcloud compute instances set-scheduling {vm_name} --project={project_id} --zone={zone} --local-ssd-recovery-timeout=0
undefinedCase D: Handling Spot / Preemptible VM Preemptions (compute.instances.preempted
)
compute.instances.preempted案例 D:处理 Spot / Preemptible VM 抢占(compute.instances.preempted
)
compute.instances.preemptedWhen appears in Cloud Logging, Compute Engine
reclaimed a Spot VM or Preemptible VM by sending an ACPI G2 Soft Off
signal.
compute.instances.preempted- Best-Effort Preemption Window: The ACPI shutdown window provides up to 30 seconds on a best-effort basis (not guaranteed under severe physical capacity emergencies), so keep checkpointing scripts fast and idempotent and persist critical state continuously. See Spot VMs.
- Attach a & Check the Metadata Server: Query
shutdown-scriptwith headerhttp://metadata.google.internal/computeMetadata/v1/instance/preempted(Metadata-Flavor: Googleduring preemption vs.TRUEon normal shutdown):FALSE
sh
gcloud compute instances add-metadata {vm_name} \
--project={project_id} \
--zone={zone} \
--metadata=shutdown-script='#!/bin/bash
PREEMPTED=$(curl -s "http://metadata.google.internal/computeMetadata/v1/instance/preempted" -H "Metadata-Flavor: Google")
if [ "$PREEMPTED" = "TRUE" ]; then
echo "Spot VM preemption detected; flushing checkpoints..."
# Insert fast (<20s), idempotent checkpoint or drain command here
fi'- Safe Rollback of Test Shutdown Metadata (Always Include):
sh
gcloud compute instances remove-metadata {vm_name} --project={project_id} --zone={zone} --keys=shutdown-script
当 Cloud Logging 中出现 时,
说明 Compute Engine 已通过发送 ACPI G2 Soft Off 信号,
回收了 Spot VM 或 Preemptible VM。
compute.instances.preempted- 尽力而为的抢占窗口:ACPI 关闭窗口最多可提供 30 秒 (在严重的物理容量紧急情况下不保证提供),因此应确保检查点脚本快速且幂等, 并持续持久化关键状态。请参阅 Spot VM。
- 附加 并检查元数据服务器:查询
shutdown-script, 并使用标头http://metadata.google.internal/computeMetadata/v1/instance/preempted(抢占期间为Metadata-Flavor: Google,正常关闭时为TRUE):FALSE
sh
gcloud compute instances add-metadata {vm_name} \
--project={project_id} \
--zone={zone} \
--metadata=shutdown-script='#!/bin/bash
PREEMPTED=$(curl -s "http://metadata.google.internal/computeMetadata/v1/instance/preempted" -H "Metadata-Flavor: Google")
if [ "$PREEMPTED" = "TRUE" ]; then
echo "Spot VM preemption detected; flushing checkpoints..."
# Insert fast (<20s), idempotent checkpoint or drain command here
fi'- 安全回滚测试用关闭元数据(始终包含):
sh
gcloud compute instances remove-metadata {vm_name} --project={project_id} --zone={zone} --keys=shutdown-script
Case E: VM Lifecycle Events Dashboard & High-Availability SAP Workloads
案例 E:VM 生命周期事件仪表板与高可用 SAP 工作负载
- Dashboard (
GCE VM Lifecycle Events MonitoringLog-Based Metric): To populate the pre-generatedvm-lifecycle-eventsdashboard in Cloud Monitoring (Monitoring > Dashboards > Dashboard List), create theGCE VM Lifecycle Events Monitoringcounter metric usingvm-lifecycle-eventsso the required--config-from-filemethodlabel extractor onSTRINGis included (protoPayload.methodNamealone omits label extractors):gcloud logging metrics create --log-filteryaml# Save as vm-lifecycle-events-metric.yaml: name: vm-lifecycle-events description: "Count of Compute Engine VM lifecycle and host events by method" filter: >- resource.type="gce_instance" AND (log_id("cloudaudit.googleapis.com/activity") OR log_id("cloudaudit.googleapis.com/system_event")) AND operation.first="true" metricDescriptor: metricKind: DELTA valueType: INT64 unit: "1" labels: - key: method valueType: STRING description: "Compute Engine lifecycle or host event method" labelExtractors: method: >- REGEXP_EXTRACT(protoPayload.methodName, "(recreateInstance|hostError|automaticRestart|guestTerminate|terminateOnHostMaintenance|preempted|insert|stop|delete|reset|start)")shgcloud logging metrics create vm-lifecycle-events --project={project_id} --config-from-file=vm-lifecycle-events-metric.yaml - SAP Workloads (HANA / NetWeaver):
If the impacted VM runs SAP HANA or SAP NetWeaver, verify automatic
Pacemaker cluster failover to the secondary node in Cloud Monitoring using
the SAP HANA Availability
() or SAP NetWeaver Availability (
google_agent_for_sap.agent-for-sap-hana-availability-monitoring) dashboards.google_agent_for_sap.agent-for-sap-netweaver-availability-monitoring
- GCE VM 生命周期事件监控仪表板(基于日志的指标): 要在 Cloud Monitoring(Monitoring > Dashboards > Dashboard List)中填充 预生成的 GCE VM Lifecycle Events Monitoring 仪表板, 请使用
vm-lifecycle-events创建--config-from-file计数器指标, 从而确保包含基于vm-lifecycle-events的必需protoPayload.methodNamemethod标签提取器(单独使用STRING会省略标签提取器):gcloud logging metrics create --log-filteryaml# Save as vm-lifecycle-events-metric.yaml: name: vm-lifecycle-events description: "Count of Compute Engine VM lifecycle and host events by method" filter: >- resource.type="gce_instance" AND (log_id("cloudaudit.googleapis.com/activity") OR log_id("cloudaudit.googleapis.com/system_event")) AND operation.first="true" metricDescriptor: metricKind: DELTA valueType: INT64 unit: "1" labels: - key: method valueType: STRING description: "Compute Engine lifecycle or host event method" labelExtractors: method: >- REGEXP_EXTRACT(protoPayload.methodName, "(recreateInstance|hostError|automaticRestart|guestTerminate|terminateOnHostMaintenance|preempted|insert|stop|delete|reset|start)")shgcloud logging metrics create vm-lifecycle-events --project={project_id} --config-from-file=vm-lifecycle-events-metric.yaml - SAP 工作负载(HANA / NetWeaver):
如果受影响的 VM 运行 SAP HANA 或 SAP NetWeaver,
请通过 Cloud Monitoring 中的 SAP HANA Availability
()或 SAP NetWeaver Availability (
google_agent_for_sap.agent-for-sap-hana-availability-monitoring) 仪表板,验证 Pacemaker 集群是否自动故障转移到辅助节点。google_agent_for_sap.agent-for-sap-netweaver-availability-monitoring
Verification, Rollback & Cleanup
验证、回滚与清理
- Verify Updated Scheduling Policy:
sh
gcloud compute instances describe {vm_name} --project={project_id} --zone={zone} \ --format="yaml(scheduling, resourceStatus.upcomingMaintenance)" - Remove Test Shutdown Metadata:
sh
gcloud compute instances remove-metadata {vm_name} --project={project_id} --zone={zone} --keys=shutdown-script - Teardown Temporary Test VMs: Delete any temporary test VMs when finished
(omitting on manual commands so interactive confirmation prevents accidental disk loss):
--quietshgcloud compute instances delete {vm_name} --project={project_id} --zone={zone} --delete-disks=all - Pricing Reference: Read-only troubleshooting queries create no billable VM resources (user-defined log-based metrics and Log Analytics queries on linked BigQuery datasets follow standard Cloud Monitoring and BigQuery pricing). Consult Google Cloud pricing and Compute Engine pricing for current Standard vs. Spot regional rates rather than hardcoded dollar figures.
- 验证更新后的调度策略:
sh
gcloud compute instances describe {vm_name} --project={project_id} --zone={zone} \ --format="yaml(scheduling, resourceStatus.upcomingMaintenance)" - 移除测试用关闭元数据:
sh
gcloud compute instances remove-metadata {vm_name} --project={project_id} --zone={zone} --keys=shutdown-script - 拆除临时测试 VM:完成后删除所有临时测试 VM
(手动命令中省略 ,让交互式确认防止意外丢失磁盘):
--quietshgcloud compute instances delete {vm_name} --project={project_id} --zone={zone} --delete-disks=all - 定价参考:只读故障排查查询不会创建计费 VM 资源 (用户定义的基于日志的指标,以及在关联 BigQuery 数据集上执行的 Log Analytics 查询,遵循 Cloud Monitoring 和 BigQuery 的标准定价)。 请查阅 Google Cloud 定价和 Compute Engine 定价, 以获取当前 Standard 与 Spot 区域费率,而不是使用硬编码的美元金额。