google-cloud-solution-guided-gke-ai-migration

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Migrate AI Workloads to GKE Inference

将AI工作负载迁移到GKE推理环境

This skill guides agents through the end-to-end process of migrating existing AI inference workloads (e.g., from Cloud Run, Gemini API, Gemini Enterprise Agent Platform) to self-hosted inference on Google Kubernetes Engine (GKE). The agent will act as an interactive architect, using a structured 4-phase workflow to discover requirements, design a Google Cloud-native solution, execute the implementation using

gcloud

and

kubectl

, and validate the deployment.

本技能将引导代理完成将现有AI推理工作负载（例如来自Cloud Run、Gemini API、Gemini Enterprise Agent Platform）迁移到Google Kubernetes Engine（GKE）自托管推理环境的端到端流程。代理将作为交互式架构师，采用结构化的4阶段工作流来梳理需求、设计Google Cloud原生解决方案、使用

gcloud

和

kubectl

执行部署，并验证部署结果。

Gemini Cloud Assist MCP off-ramp

Gemini Cloud Assist MCP 分流规则

This skill covers manual, architect-guided migration only. Automated migration is the job of the Gemini Cloud Assist MCP server. Route between them as follows:

The user asks to USE Gemini Cloud Assist or MCP automation for this migration (e.g., "use the Cloud Assist MCP server to do this"): stop the manual workflow and respond with the 4 required points below.
The user mentions MCP only in passing, or explicitly declines it (e.g., "no MCP, let's do this manually"): proceed with the manual workflow. Do not stop and do not ask about MCP.
The user does not mention MCP at all: proceed directly to the active phase. In your first discovery response only, add one sentence noting that an automated alternative exists via the Gemini Cloud Assist MCP server and the user can switch to it at any time. Do not wait for an answer before beginning discovery.

When stopping for an MCP request, your response MUST include these 4 points:

STOP the manual workflow & clarify scope: State that
```
google-cloud-solution-guided-gke-ai-migration
```
is strictly intended for manual, architect-guided migration using native CLIs (
```
gcloud
```
and
```
kubectl
```
), and that this manual skill workflow is being stopped.
Explain MCP capabilities: Explain that the Gemini Cloud Assist MCP server assists in automated infrastructure analysis (
```
gemini_cloud_assist:ask_cloud_assist
```
) or direct Google Cloud resource mutation (
```
gemini_cloud_assist:invoke_operation
```
).
Link to MCP Documentation: Provide a valid hyperlink to the Gemini Cloud Assist MCP Documentation.
Link to Intent to Infrastructure Codelab: Provide a valid hyperlink to the Intent to Infrastructure Codelab for guidance on setting up the MCP server.

本技能仅涵盖人工、架构师指导的迁移流程。自动化迁移由Gemini Cloud Assist MCP服务器负责。请按以下规则进行路由：

用户要求使用Gemini Cloud Assist或MCP自动化完成迁移（例如："使用Cloud Assist MCP服务器来完成这项工作"）：终止人工工作流，并按以下4点要求回应。
用户仅顺带提及MCP，或明确拒绝使用（例如："不要用MCP，我们手动来做"）：继续执行人工工作流。无需终止，也无需询问是否使用MCP。
用户完全未提及MCP：直接进入当前活动阶段。仅在首次梳理需求的回应中添加一句话，说明可通过Gemini Cloud Assist MCP服务器实现自动化替代方案，用户可随时切换。无需等待用户回复即可开始需求梳理。

当因MCP请求终止工作流时，回应必须包含以下4点：

终止人工工作流并明确范围：说明
```
google-cloud-solution-guided-gke-ai-migration
```
仅适用于使用原生CLI（
```
gcloud
```
和
```
kubectl
```
）进行的人工、架构师指导迁移，当前人工技能工作流将终止。
说明MCP功能：解释Gemini Cloud Assist MCP服务器可协助完成自动化基础设施分析（
```
gemini_cloud_assist:ask_cloud_assist
```
）或直接操作Google Cloud资源（
```
gemini_cloud_assist:invoke_operation
```
）。
提供MCP文档链接：提供有效的Gemini Cloud Assist MCP文档超链接。
提供Intent to Infrastructure代码实验室链接：提供有效的Intent to Infrastructure代码实验室超链接，以指导用户设置MCP服务器。

Scope Check: New Deployments vs. Migrations

范围检查：新部署 vs 迁移

This skill is specifically intended for migrating existing AI workloads (from Cloud Run, Gemini API, Agent Platform, or other platforms) to GKE.

If the user wants to deploy a new AI model server from scratch on GKE (and does NOT have an existing deployment to migrate), STOP and recommend using the gke-inference
skill instead. Explain that

google-cloud-solution-guided-gke-ai-migration

focuses on migration workflows (discovering existing Cloud Run/Agent Platform configurations, traffic cutover, etc.), while

gke-inference

is optimized for fresh GKE AI model server deployments using AI Profiles and golden path manifests.

本技能专门用于将现有AI工作负载（来自Cloud Run、Gemini API、Agent Platform或其他平台）迁移到GKE。

如果用户希望在GKE上从零开始部署新的AI模型服务器（且无现有部署可迁移），请终止当前流程，并推荐使用**

gke-inference

**技能。解释

google-cloud-solution-guided-gke-ai-migration

专注于迁移工作流（梳理现有Cloud Run/Agent Platform配置、流量切换等），而

gke-inference

则针对使用AI配置文件和标准清单的全新GKE AI模型服务器部署进行了优化。

Core Architectural Principles (The "Golden Path")

核心架构原则（"标准路径"）

When designing the solution, always default to the latest GKE AI best practices:

Execution:
- Execution policy (who runs commands):
  - Phase 1 (Discovery): after the user grants permission, execute read-only
```
gcloud
```
    inspection commands (
```
list
```
    ,
```
describe
```
    ) directly and summarize the results.
  - Phases 2-4: write manifests to disk, then present the exact
```
gcloud
```
    and
```
kubectl
```
    commands for the user to run. Do not execute mutating commands (
```
apply
```
    ,
```
create
```
    ,
```
delete
```
    , cluster or IAM changes) unless the user explicitly asks you to run them, in which case execute them and report each command's actual output.
  - Informational and troubleshooting questions: answer with markdown guidance, manifests, and recommended commands only; execute nothing.
- Favor raw Kubernetes manifests, native CLIs (
```
gcloud
```
  for infrastructure,
```
kubectl
```
  for workloads), and opinionated templates.
- Save YAML files to the user's current directory and apply them using
```
kubectl
```
  .
- Write ad-hoc scripts (e.g., for VRAM calculation) only if absolutely necessary.
Node Provisioning:
- Utilize Custom Compute Classes (CCC) to maximize accelerator obtainability (e.g., dynamically choosing spot vs. on-demand or specific GPU profiles).
- Use GKE's managed GPU driver installations.
- Select appropriate node topologies: use a single node in a static pool for a simple job, or multiple nodes with LWS/CCC for larger jobs.
Inference Stack & Versioning:
- Default to vLLM (
```
vllm/vllm-openai
```
  ) as the standard LLM serving engine. If migrating from Vertex AI, the user may opt to retain the Vertex AI Model Garden image (e.g.,
```
pytorch-vllm-serve
```
  ), which is permissible.
- Explicit Entrypoint Override: Regardless of the chosen image, the vLLM Deployment MUST explicitly set
```
command: ["python", "-m", "vllm.entrypoints.openai.api_server"]
```
  to bypass potentially problematic entrypoint scripts (like
```
gcs_download_launcher.sh
```
  in Vertex AI images) that crash when passed standard vLLM arguments.
- Always pin an explicit, stable vLLM image tag, never
```
:latest
```
  . Resolve the current stable release at design time (check the vLLM releases page, or take the tag from
```
gcloud container ai profiles manifests create
```
  output) and record it in
```
migration-state.md
```
  ; do not reuse a tag remembered from a previous migration or from documentation examples.
- Expose the service through the GKE Gateway API. Default to a regional internal Application Load Balancer (
```
gatewayClassName: gke-l7-rilb
```
  ) with an HTTPRoute that sends
```
/v1
```
  requests to the vLLM ClusterIP service (
```
{workload_name}-vllm-svc
```
  ) on port 8000, based on
```
assets/gke-inference-gateway.yaml.tmpl
```
  .
- If the user needs LLM-aware load balancing (routing on KV-cache utilization, queue depth, or LoRA adapter placement), offer the GKE Inference Gateway as an upgrade: it requires an
```
InferencePool
```
  resource as the HTTPRoute backend instead of a Service, and it is only supported on the
```
gke-l7-rilb
```
  and
```
gke-l7-regional-external-managed
```
  GatewayClasses. Fetch About GKE Inference Gateway before generating InferencePool manifests; do not improvise them from memory.
- For multi-node models, use LeaderWorkerSet (LWS) with vLLM.
Security & Access:
- Always use GKE Workload Identity for Google Cloud API access.
- Gated Model Secret Security: For gated models (e.g., Llama 3, Gemma) requiring Hugging Face tokens (
```
HF_TOKEN
```
  ):
  - NEVER write literal token values into Deployment or Pod specifications; reference the secret securely using
```
env.valueFrom.secretKeyRef
```
    (e.g., pointing to
```
hf-secret
```
    ).
  - Always warn the user about the security risks of exposing sensitive API tokens in plain text prompts or manifest files.
  - NEVER write a Kubernetes
```
Secret
```
    manifest to disk, and do NOT include it in templates. Instead, explicitly instruct the user to create the Secret directly via CLI before applying any other manifests:
```
kubectl create secret generic hf-secret --namespace={namespace} --from-literal=hf_api_token=<YOUR_HF_TOKEN>
```
    with the user substituting the real value themselves.
  - If the user has already pasted a token into the conversation, treat that token as exposed: use a
```
<YOUR_HF_TOKEN>
```
    placeholder in every command and manifest you produce, and advise the user to revoke and reissue the token at https://huggingface.co/settings/tokens once the migration is complete.
- Endpoint exposure: Default the Gateway to the internal class (
```
gke-l7-rilb
```
  ). The vLLM OpenAI-compatible endpoint has no built-in authentication; if the user requires external exposure, warn them explicitly that an unauthenticated external listener is an open inference API on their GPU bill, and require an explicit decision plus a fronting control (IAP, an authenticating API gateway, or strict client allowlisting) before generating an externally-exposed Gateway manifest.
Model Storage & Cold Starts:
- Stage the chosen model in a Cloud Storage bucket to save on downloading more than once.
- Execute staging via a Job on the cluster: Explain to the user that staging weights via a cluster Job avoids downloading heavy weights to their local workstation and avoids re-downloading on every container restart. Always save the staging manifest as
```
model-staging-job.yaml
```
  and instruct the user to run
```
kubectl apply -f model-staging-job.yaml
```
  .
- Staging Logic based on Source/Target:
  - Source is Hugging Face ( gCS FUSE or Lustre target): Use the staging Job to download weights directly to the PVC.
  - Source is GCS (GCS FUSE target): The staging Job is OPTIONAL. If the PVC mounts the GCS bucket directly via FUSE, the weights are already accessible and no staging step is needed.
  - Source is GCS (Lustre target): Use the staging Job to copy weights from the source GCS bucket to the Lustre PVC (e.g., using
```
gcloud storage cp
```
    ).
- Favor Cloud Storage staging via Cloud Storage FUSE for most workloads, or Managed Lustre for ultra-low latency, PiB-scale needs.
- Every pod that mounts a Cloud Storage FUSE volume MUST carry the pod annotation
```
gke-gcsfuse/volumes: "true"
```
  (this injects the FUSE sidecar) and MUST run as the Kubernetes ServiceAccount bound to a Google service account with
```
roles/storage.objectUser
```
  on the model bucket via Workload Identity. A pod missing either one will fail to mount or fail to read; check both before troubleshooting anything else storage-related.
Observability:
- Default to Google Cloud Managed Service for Prometheus with DCGM metrics for deep GPU visibility.
Autoscaling:
- Do NOT assume the user wants Horizontal Pod Autoscaling (HPA); you MUST ask them during discovery.
- If HPA is declined, omit all autoscaling manifests.
- If HPA is desired, warn the user about the LLM Autoscaling Trap: standard CPU, Memory, and GPU Memory utilization metrics are unreliable because vLLM preallocates VRAM for KV caching, appearing highly utilized constantly.
- Recommend scaling based on custom server metrics reflecting actual concurrency or queue depth (e.g.,
```
vllm:num_requests_waiting
```
  or batch size).
- Include a reasonable default in the form of Queue Size unless the user specifically mentions a different metric.
- Implementation can use either GKE Custom Metrics (Stackdriver Adapter) or KEDA.

设计解决方案时，请始终遵循最新的GKE AI最佳实践：

执行规则：
- 执行策略（谁来运行命令）：
  - **阶段1（需求梳理）：**获得用户许可后，直接执行只读
```
gcloud
```
    检查命令（
```
list
```
    、
```
describe
```
    ）并汇总结果。
  - **阶段2-4：**将清单写入磁盘，然后向用户展示精确的
```
gcloud
```
    和
```
kubectl
```
    命令供其运行。除非用户明确要求执行变更命令（
```
apply
```
    、
```
create
```
    、
```
delete
```
    、集群或IAM变更），否则请勿执行此类命令；若用户要求，需执行命令并报告每条命令的实际输出。
  - **信息查询与故障排查问题：**仅使用Markdown指南、清单和推荐命令作答；不执行任何命令。
- 优先使用原生Kubernetes清单、原生CLI（
```
gcloud
```
  用于基础设施，
```
kubectl
```
  用于工作负载）和标准化模板。
- 将YAML文件保存到用户当前目录，并使用
```
kubectl
```
  应用这些文件。
- 仅在绝对必要时编写临时脚本（例如用于VRAM计算的脚本）。
节点配置：
- 利用**自定义计算类（CCC）**最大化加速器获取能力（例如动态选择抢占式 vs 按需实例，或特定GPU配置）。
- 使用GKE的托管GPU驱动安装。
- 选择合适的节点拓扑：简单任务使用静态池中的单个节点，大型任务则使用多节点搭配LWS/CCC。
推理栈与版本控制：
- 默认使用vLLM（
```
vllm/vllm-openai
```
  ）作为标准LLM服务引擎。如果从Vertex AI迁移，用户可选择保留Vertex AI Model Garden镜像（例如
```
pytorch-vllm-serve
```
  ），此操作是允许的。
- **显式入口点覆盖：**无论选择哪种镜像，vLLM部署必须显式设置
```
command: ["python", "-m", "vllm.entrypoints.openai.api_server"]
```
  ，以绕过可能存在问题的入口点脚本（例如Vertex AI镜像中的
```
gcs_download_launcher.sh
```
  ），这些脚本在传入标准vLLM参数时可能崩溃。
- 始终固定使用明确、稳定的vLLM镜像标签，绝不要使用
```
:latest
```
  。在设计阶段确定当前稳定版本（查看vLLM发布页面，或从
```
gcloud container ai profiles manifests create
```
  输出中获取标签），并将其记录在
```
migration-state.md
```
  中；请勿复用之前迁移或文档示例中记住的标签。
- 通过GKE Gateway API暴露服务。默认使用区域内部应用负载均衡器（
```
gatewayClassName: gke-l7-rilb
```
  ），并配置HTTPRoute将
```
/v1
```
  请求转发到端口8000上的vLLM ClusterIP服务（
```
{workload_name}-vllm-svc
```
  ），基于
```
assets/gke-inference-gateway.yaml.tmpl
```
  模板。
- 如果用户需要LLM感知负载均衡（基于KV缓存利用率、队列深度或LoRA适配器位置进行路由），可提供GKE推理网关作为升级方案：它需要
```
InferencePool
```
  资源作为HTTPRoute后端，而非Service，且仅支持
```
gke-l7-rilb
```
  和
```
gke-l7-regional-external-managed
```
  GatewayClasses。生成InferencePool清单前，请先查阅GKE推理网关介绍；请勿凭记忆自行编写。
- 对于多节点模型，搭配vLLM使用LeaderWorkerSet (LWS)。
安全与访问：
- 始终使用**GKE工作负载身份（Workload Identity）**访问Google Cloud API。
- **受限模型密钥安全：**对于需要Hugging Face令牌（
```
HF_TOKEN
```
  ）的受限模型（例如Llama 3、Gemma）：
  - 绝不要将令牌明文写入Deployment或Pod规格；请使用
```
env.valueFrom.secretKeyRef
```
    安全引用密钥（例如指向
```
hf-secret
```
    ）。
  - 始终提醒用户在纯文本提示或清单文件中暴露敏感API令牌的安全风险。
  - 绝不要将Kubernetes
```
Secret
```
    清单写入磁盘，也不要将其包含在模板中。相反，需明确指示用户在应用其他任何清单之前，通过CLI直接创建Secret：
```
kubectl create secret generic hf-secret --namespace={namespace} --from-literal=hf_api_token=<YOUR_HF_TOKEN>
```
    ，由用户自行替换实际值。
  - 如果用户已将令牌粘贴到对话中，需将该令牌视为已暴露：在所有命令和清单中使用
```
<YOUR_HF_TOKEN>
```
    占位符，并建议用户迁移完成后在https://huggingface.co/settings/tokens撤销并重新生成令牌。
- **端点暴露：**默认将网关设置为内部类（
```
gke-l7-rilb
```
  ）。vLLM的OpenAI兼容端点无内置身份验证；如果用户需要外部暴露，需明确警告用户：未认证的外部监听器相当于在GPU账单上开放了一个推理API，必须获得用户明确决定，并提供前置控制（IAP、带认证的API网关或严格的客户端白名单）后，再生成外部暴露的网关清单。
模型存储与冷启动：
- 将选定模型暂存到Cloud Storage存储桶中，避免重复下载。
- 通过集群上的Job执行暂存：向用户解释，通过集群Job暂存权重可避免将大权重下载到本地工作站，也避免每次容器重启时重新下载。请始终将暂存清单保存为
```
model-staging-job.yaml
```
  ，并指示用户运行
```
kubectl apply -f model-staging-job.yaml
```
  。
- 基于源/目标的暂存逻辑：
  - **源为Hugging Face（目标为GCS FUSE或Lustre）：**使用暂存Job直接将权重下载到PVC。
  - 源为GCS（目标为GCS FUSE）：暂存Job为可选。如果PVC通过FUSE直接挂载GCS存储桶，权重已可访问，无需暂存步骤。
  - **源为GCS（目标为Lustre）：**使用暂存Job将权重从源GCS存储桶复制到Lustre PVC（例如使用
```
gcloud storage cp
```
    ）。
- 对于大多数工作负载，优先通过Cloud Storage FUSE进行Cloud Storage暂存；对于超低延迟、PiB级需求，使用Managed Lustre。
- 每个挂载Cloud Storage FUSE卷的Pod必须携带Pod注解
```
gke-gcsfuse/volumes: "true"
```
  （这会注入FUSE sidecar），且必须以绑定到具有模型存储桶
```
roles/storage.objectUser
```
  权限的Google服务账号的Kubernetes ServiceAccount运行。缺少其中任何一项，Pod都将无法挂载或读取；排查存储相关问题前，请先检查这两项。
可观测性：
- 默认使用Google Cloud托管Prometheus服务搭配DCGM指标，实现深度GPU可见性。
自动扩缩容：
- 不要假设用户需要水平Pod自动扩缩容（HPA）；必须在需求梳理阶段询问用户。
- 如果用户拒绝HPA，请省略所有自动扩缩容清单。
- 如果用户需要HPA，需提醒用户注意LLM自动扩缩容陷阱：标准CPU、内存和GPU内存利用率指标不可靠，因为vLLM会为KV缓存预分配VRAM，导致利用率始终显示为高。
- 建议基于反映实际并发量或队列深度的自定义服务器指标进行扩缩容（例如
```
vllm:num_requests_waiting
```
  或批处理大小）。
- 除非用户明确提及其他指标，否则默认使用队列大小作为合理指标。
- 可使用GKE自定义指标（Stackdriver Adapter）或KEDA实现。

Workflow

工作流

The solution design and implementation workflow consists of the following 4 phases:

Phase 1: Discovery: Inspect existing infrastructure via
```
gcloud
```
and gather model/traffic requirements.
Phase 2: Solution Design: Calculate VRAM requirements, select hardware/storage, and generate Kubernetes manifests.
Phase 3: Implementation: Provision resources, stage model weights via an ephemeral pod, and apply workload manifests using
```
kubectl
```
.
Phase 4: Validation and Cutover: Verify pod health and endpoint inference, then provide traffic migration guidance.

At the start of every architectural response, print a simple visual progress indicator line to keep both the user and model aligned on the current phase:

markdown

**Migration Progress:** [● Discovery] ➔ [○ Solution Design] ➔ [○ Implementation] ➔ [○ Validation]

(Update
●
to mark the current active phase, e.g.,
[● Solution Design]
during Phase 2).

解决方案设计与实现工作流包含以下4个阶段：

阶段1：需求梳理：通过
```
gcloud
```
检查现有基础设施，收集模型/流量需求。
阶段2：解决方案设计：计算VRAM需求，选择硬件/存储，生成Kubernetes清单。
阶段3：部署实施：配置资源，通过临时Pod暂存模型权重，使用
```
kubectl
```
应用工作负载清单。
阶段4：验证与流量切换：验证Pod健康状态和端点推理能力，提供流量迁移指导。

在每个架构相关回应的开头，请打印一条简单的可视化进度指示线，让用户和模型都能清晰了解当前阶段：

markdown

**迁移进度：** [● 需求梳理] ➔ [○ 解决方案设计] ➔ [○ 部署实施] ➔ [○ 验证]

(更新
●
标记当前活动阶段，例如阶段2时标记为
[● 解决方案设计]
)

Phase Routing Rules

阶段路由规则

Determine the active phase based on the user's prompt context:

migration-state.md

exists in the current directory, read it before anything else and resume from the recorded phase with the recorded values; only re-ask a discovery question if its value is missing from the file or contradicted by the user's prompt.

Phase 1 (Discovery): Use for new migration requests without prior discovery.
Phase 2 (Solution Design): Use when the prompt indicates discovery is complete or asks for architecture design / YAML manifest generation.
Phase 3 (Implementation): Use when the prompt indicates model weights are staged or asks directly for deployment steps/commands. Progress indicator:
```
**Migration Progress:** [○ Discovery] ➔ [○ Solution Design] ➔ [● Implementation] ➔ [○ Validation]
```
.
Phase 4 (Validation): Use when the prompt indicates the workload is running or asks for health checks/testing steps.

根据用户提示上下文确定活动阶段：

如果当前目录中存在

migration-state.md

，请先读取该文件，然后从记录的阶段和值恢复流程；仅当文件中缺少某个需求梳理问题的值，或与用户提示矛盾时，才重新询问该问题。

阶段1（需求梳理）：适用于无前期需求梳理的新迁移请求。
阶段2（解决方案设计）：适用于提示表明需求梳理已完成，或用户询问架构设计/YAML清单生成的场景。
阶段3（部署实施）：适用于提示表明模型权重已暂存，或用户直接询问部署步骤/命令的场景。进度指示：
```
**迁移进度：** [○ 需求梳理] ➔ [○ 解决方案设计] ➔ [● 部署实施] ➔ [○ 验证]
```
。
阶段4（验证）：适用于提示表明工作负载已运行，或用户询问健康检查/测试步骤的场景。

Phase 1: Discovery

阶段1：需求梳理

The goal of Phase 1 is to discover all workload specifications necessary to design and build the target GKE inference infrastructure.

Phase 1 Response Requirements: Every response during Phase 1 MUST begin with the visual progress indicator:

**Migration Progress:** [● Discovery] ➔ [○ Solution Design] ➔ [○ Implementation] ➔ [○ Validation]

阶段1的目标是梳理设计和构建目标GKE推理基础设施所需的所有工作负载规格。

**阶段1回应要求：**阶段1的每个回应必须以可视化进度指示开头：

**迁移进度：** [● 需求梳理] ➔ [○ 解决方案设计] ➔ [○ 部署实施] ➔ [○ 验证]

。

Discovery Checklist (Attributes to Identify)

需求梳理清单（需确认的属性）

The agent must discover or confirm the following 6 core attribute categories:

Source Platform & Service Configuration:
- Source platform (Cloud Run, Gemini API, Gemini Enterprise Agent Platform, custom VM).
- Container image tag, environment variables, CPU/RAM allocations, and secret bindings.
Model Specifications:
- Model name and parameter size (e.g., Gemma 2 9B, Llama 3 70B).
- Target precision & quantization tolerance (FP16/BF16 vs INT8/INT4 AWQ). Must inquire about quantization tolerance during discovery.
- Gated model status (whether Hugging Face token
```
HF_TOKEN
```
  access is required).
- Model-Specific Architecture Requirements: Explicitly check the model card (e.g., on Hugging Face) or
```
config.json
```
  for custom configuration requirements. For example, determine if the architecture requires
```
--trust-remote-code
```
  (like Qwen models), specific rope scaling arguments, or other custom flags.
Target GKE & Hardware Infrastructure:
- Target GKE cluster name and region (or confirm creating a new cluster).
- Accelerator preference (L4, A100, H100, TPU v5e) and provisioning model (Spot/CCC vs On-Demand).
- Regional quota availability for requested GPUs/TPUs.
Model Storage & Staging:
- Current location of model weights (GCS bucket, Hugging Face Hub, external URL).
- Preferred storage integration (Cloud Storage FUSE vs Managed Lustre).
Traffic Profile & Load Balancing:
- Expected traffic volume, concurrency, and request patterns (spiky vs consistent baseline).
- Load balancing needs (GKE Inference Gateway, standard Ingress/Service).
- Autoscaling Requirements: Ask the user if they need Horizontal Pod Autoscaling (HPA). Do not assume they do. If they do, identify target metrics (defaulting to Queue Size) and discuss the LLM Autoscaling Trap.
Model Equivalence (Gemini API / Agent Platform sources only):
- Which Gemini model and API features are in use (function calling, system instructions, context length, multimodal inputs).
- Which open-weights model will replace it, and how the user plans to evaluate output quality against the current system before cutover.
- Client impact: the vLLM endpoint is OpenAI-compatible, not Gemini-API-compatible; identify the client code and SDK calls that must change.

代理必须梳理或确认以下6个核心属性类别：

源平台与服务配置：
- 源平台（Cloud Run、Gemini API、Gemini Enterprise Agent Platform、自定义VM）。
- 容器镜像标签、环境变量、CPU/RAM分配和密钥绑定。
模型规格：
- 模型名称和参数规模（例如Gemma 2 9B、Llama 3 70B）。
- 目标精度与量化容忍度（FP16/BF16 vs INT8/INT4 AWQ）。必须在需求梳理阶段询问量化容忍度。
- 受限模型状态（是否需要Hugging Face令牌
```
HF_TOKEN
```
  访问）。
- **模型特定架构要求：**明确检查模型卡（例如在Hugging Face上）或
```
config.json
```
  中的自定义配置要求。例如，确定架构是否需要
```
--trust-remote-code
```
  （如Qwen模型）、特定的rope缩放参数或其他自定义标志。
目标GKE与硬件基础设施：
- 目标GKE集群名称和区域（或确认是否创建新集群）。
- 加速器偏好（L4、A100、H100、TPU v5e）和配置模式（抢占式/CCC vs 按需）。
- 目标区域中请求的GPU/TPU的配额可用性。
模型存储与暂存：
- 模型权重当前位置（GCS存储桶、Hugging Face Hub、外部URL）。
- 首选存储集成方式（Cloud Storage FUSE vs Managed Lustre）。
流量配置与负载均衡：
- 预期流量规模、并发量和请求模式（突发型 vs 稳定基线）。
- 负载均衡需求（GKE推理网关、标准Ingress/Service）。
- **自动扩缩容需求：**询问用户是否需要水平Pod自动扩缩容（HPA）。不要假设用户需要。如果需要，请确定目标指标（默认队列大小）并讨论LLM自动扩缩容陷阱。
模型等效性（仅适用于Gemini API / Agent Platform源）：
- 当前使用的Gemini模型和API功能（函数调用、系统指令、上下文长度、多模态输入）。
- 将替代它的开源权重模型，以及用户计划如何在切换前评估新系统与当前系统的输出质量。
- 客户端影响：vLLM端点兼容OpenAI API，而非Gemini API；需确定必须修改的客户端代码和SDK调用。

Discovery Execution (Automated & Interactive)

需求梳理执行（自动化与交互式）

Request Infrastructure Access & Permission First: You MUST explicitly request permission from the user to inspect existing environment resources via CLI commands (such as
```
gcloud run services list
```
or
```
gcloud run services describe
```
) before executing any discovery commands.
Inspection Upon Approval: Once the user grants permission, execute
```
gcloud
```
CLI commands, inspect environment variables, and review local workspace files to populate checklist items automatically:
- Cloud Run Workloads: Run
```
gcloud run services list --format="table(metadata.name,status.url,status.latestReadyRevisionName)"
```
  to enumerate services, then
```
gcloud run services describe {service_name} --format="yaml(spec.template.spec.containers,spec.template.metadata.annotations,spec.template.spec.serviceAccountName,spec.template.spec.containerConcurrency)"
```
  to extract only the container image, env vars, resource limits, concurrency, and secret bindings. Prefer
```
--format
```
  filters on all discovery commands; never pull a full unfiltered resource description into the conversation.
- Existing GKE Clusters: Run
```
gcloud container clusters list
```
  and
```
gcloud container clusters describe {cluster_name}
```
  to inspect active cluster config, Workload Identity setup, and available accelerator pools.
- Vertex AI / Storage: Run
```
gcloud ai endpoints list
```
  or
```
gcloud storage buckets list
```
  to locate model artifacts and storage buckets.
- Workspace & Environment: Inspect local configuration files or environment variables in the active workspace directory.
Summarize, Query Missing Items, and Confirm: Present a consolidated summary of all discovered configuration data and prompt the user for any remaining missing attributes. If any checklist item is unresolved or ambiguous, obtain explicit user confirmation before moving to Phase 2. If every checklist item was resolved without ambiguity, you MAY present the discovery summary and the Phase 2 solution design in the same response, under a single combined approval gate; the design approval then covers both.
Persist discovery state: After the user confirms the discovery summary, write it to
```
migration-state.md
```
in the current directory: one section per checklist category with the confirmed values, plus a final line
```
Current phase: <phase name>
```
. Update the
```
Current phase:
```
line every time the workflow advances a phase.

**先请求基础设施访问权限：**在执行任何需求梳理命令前，必须明确请求用户许可，通过CLI命令（如
```
gcloud run services list
```
或
```
gcloud run services describe
```
）检查现有环境资源。
**获得许可后检查：**获得用户许可后，执行
```
gcloud
```
CLI命令、检查环境变量并查看本地工作区文件，自动填充清单项：
- **Cloud Run工作负载：**运行
```
gcloud run services list --format="table(metadata.name,status.url,status.latestReadyRevisionName)"
```
  枚举服务，然后运行
```
gcloud run services describe {service_name} --format="yaml(spec.template.spec.containers,spec.template.metadata.annotations,spec.template.spec.serviceAccountName,spec.template.spec.containerConcurrency)"
```
  仅提取容器镜像、环境变量、资源限制、并发量和密钥绑定。所有需求梳理命令优先使用
```
--format
```
  过滤；绝不要将完整的未过滤资源描述引入对话。
- **现有GKE集群：**运行
```
gcloud container clusters list
```
  和
```
gcloud container clusters describe {cluster_name}
```
  检查活动集群配置、Workload Identity设置和可用加速器池。
- **Vertex AI / 存储：**运行
```
gcloud ai endpoints list
```
  或
```
gcloud storage buckets list
```
  查找模型制品和存储桶。
- **工作区与环境：**检查活动工作区目录中的本地配置文件或环境变量。
**汇总、询问缺失项并确认：**呈现所有已梳理配置数据的合并摘要，并提示用户提供任何剩余的缺失属性。如果任何清单项未解决或存在歧义，请在进入阶段2前获得用户明确确认。如果所有清单项均已明确解决，可在同一回应中呈现需求梳理摘要和阶段2解决方案设计，共用一个审批环节；设计审批将同时涵盖两者。
**持久化需求梳理状态：**用户确认需求梳理摘要后，将其写入当前目录的
```
migration-state.md
```
：每个清单类别对应一个章节，包含确认的值，最后添加一行
```
Current phase: <phase name>
```
。每次工作流进入下一阶段时，更新
```
Current phase:
```
行。

Phase 2: Solution Design

阶段2：解决方案设计

Based on the discovery phase, design the architecture and manifests needed to accomplish the migration.

Phase 2 Response Requirements: Every response during Phase 2 MUST begin with the visual progress indicator:

**Migration Progress:** [○ Discovery] ➔ [● Solution Design] ➔ [○ Implementation] ➔ [○ Validation]

Just-in-Time Context Loading: When evaluating specific architectural choices below (e.g., storage options, load balancing, or autoscaling), fetch and read the relevant reference documentation link from the Supporting links section as needed.

Map components and alternatives: For each major component (e.g., Load Balancing, Autoscaling, Single vs Multi-node), present your recommended "Golden Path" option along with alternatives. If the user requested HPA, ensure the design addresses the LLM Autoscaling Trap and recommends custom metrics (defaulting to Queue Size).
Accelerator Selection: Refer to GPU platforms or Plan your TPU configuration in GKE to recommend the best accelerator for the target model. (Note: the manifest templates in
assets/
are GPU-only. If the user selects Cloud TPU, state this explicitly, and base the serving manifests on the GKE TPU serving documentation instead of the templates in this skill.)
Storage Selection: Refer to GCP AI Storage Options to recommend the best storage solution for model staging and serving.
Hardware & VRAM Sizing: Accurately calculate VRAM using the Deterministic VRAM Sizing Formula below, based on parameters, context window, and quantization. Map to appropriate GPUs (L4, A100, H100).
Cluster & Node Setup: Design the appropriate cluster for the job. Add appropriate sized nodes for the task using Custom Compute Classes (CCC). Design Workload Identity if required. The vLLM Deployment MUST select nodes with
```
nodeSelector: cloud.google.com/compute-class: {compute_class_name}
```
so the workload actually schedules through the ComputeClass. If the user declines CCC and wants on-demand nodes only, replace this selector with
```
cloud.google.com/gke-accelerator: {accelerator_type}
```
and skip
```
ccc-profile.yaml
```
entirely; do not apply a ComputeClass that no workload references.
Model Staging Design: Determine if a staging Job is required based on the source location of the model weights and the target storage solution. If the source is Hugging Face, or if the target is Lustre and source is GCS, plan for a Job to download/copy weights. If the source is GCS and target is GCS FUSE, skip staging. Plan for using Cloud Storage FUSE or Lustre CSI to mount this storage to the workload.
Draft solution architecture and manifests: Read the manifest templates in
```
assets/
```
(
```
assets/vllm-deployment.yaml.tmpl
```
,
```
assets/ccc-profile.yaml.tmpl
```
,
```
assets/gke-inference-gateway.yaml.tmpl
```
,
```
assets/storage-config.yaml.tmpl
```
, and
```
assets/model-staging-job.yaml.tmpl
```
), substitute the parameters discovered in Phase 1, and save the resulting YAML manifests (
```
vllm-deployment.yaml
```
,
```
ccc-profile.yaml
```
,
```
gke-inference-gateway.yaml
```
,
```
storage-config.yaml
```
,
```
model-staging-job.yaml
```
) to disk in the current directory. When creating
```
vllm-deployment.yaml
```
, explicitly inject any required model-specific architecture flags discovered in Phase 1 (e.g.,
```
--trust-remote-code
```
) into the container
```
args
```
array. When creating
```
gke-inference-gateway.yaml
```
, base it on
```
assets/gke-inference-gateway.yaml.tmpl
```
: use
```
gatewayClassName: gke-l7-rilb
```
for the Gateway resource unless the user has explicitly chosen external exposure or the InferencePool-based GKE Inference Gateway, and route
```
/v1
```
traffic to the vLLM ClusterIP service (
```
{workload_name}-vllm-svc
```
on port 8000) in the HTTPRoute resource. If using a storage class other than
```
gcsfuse-csi
```
(e.g.,
```
lustre-csi
```
), remove the
```
gcsfuse.cloud.google.com
```
annotations from
```
storage-config.yaml
```
.
Request review and iterate: Present the generated solution architecture and diagram to the user and request their feedback. Iterate on the design until the user approves it before moving to Phase 3.

基于需求梳理阶段的结果，设计完成迁移所需的架构和清单。

**阶段2回应要求：**阶段2的每个回应必须以可视化进度指示开头：

**迁移进度：** [○ 需求梳理] ➔ [● 解决方案设计] ➔ [○ 部署实施] ➔ [○ 验证]

。

即时上下文加载：在评估以下特定架构选择时（例如存储选项、负载均衡或自动扩缩容），按需从支持链接部分获取并阅读相关参考文档链接。

**映射组件与替代方案：**针对每个主要组件（例如负载均衡、自动扩缩容、单节点 vs 多节点），提出推荐的"标准路径"选项及替代方案。如果用户请求HPA，确保设计解决LLM自动扩缩容陷阱并推荐自定义指标（默认队列大小）。
**加速器选择：**参考 GPU平台或在GKE中规划TPU配置为目标模型推荐最佳加速器。（注意：
assets/
中的清单模板仅适用于GPU。如果用户选择Cloud TPU，请明确说明，并基于GKE TPU服务文档而非本技能中的模板生成服务清单。）
**存储选择：**参考GCP AI存储选项推荐最佳的模型暂存和服务存储解决方案。
**硬件与VRAM选型：**根据参数、上下文窗口和量化方式，使用下面的确定性VRAM选型公式准确计算VRAM需求。映射到合适的GPU（L4、A100、H100）。
**集群与节点设置：**设计适合任务的集群。使用自定义计算类（CCC）添加合适规模的节点。如果需要，设计Workload Identity。vLLM部署必须通过
```
nodeSelector: cloud.google.com/compute-class: {compute_class_name}
```
选择节点，确保工作流通过ComputeClass调度。如果用户拒绝CCC，仅需要求按需节点，请将该选择器替换为
```
cloud.google.com/gke-accelerator: {accelerator_type}
```
，并完全跳过
```
ccc-profile.yaml
```
；不要应用无工作负载引用的ComputeClass。
**模型暂存设计：**根据模型权重的源位置和目标存储解决方案，确定是否需要暂存Job。如果源为Hugging Face，或目标为Lustre且源为GCS，则计划通过Job下载/复制权重。如果源为GCS且目标为GCS FUSE，则跳过暂存。计划使用Cloud Storage FUSE或Lustre CSI将此存储挂载到工作负载。
**草拟解决方案架构与清单：**读取
```
assets/
```
中的清单模板（
```
assets/vllm-deployment.yaml.tmpl
```
、
```
assets/ccc-profile.yaml.tmpl
```
、
```
assets/gke-inference-gateway.yaml.tmpl
```
、
```
assets/storage-config.yaml.tmpl
```
和
```
assets/model-staging-job.yaml.tmpl
```
），替换阶段1中梳理的参数，并将生成的YAML清单（
```
vllm-deployment.yaml
```
、
```
ccc-profile.yaml
```
、
```
gke-inference-gateway.yaml
```
、
```
storage-config.yaml
```
、
```
model-staging-job.yaml
```
）保存到当前目录的磁盘中。创建
```
vllm-deployment.yaml
```
时，需将阶段1中梳理的所有模型特定架构标志（例如
```
--trust-remote-code
```
）显式注入容器
```
args
```
数组。创建
```
gke-inference-gateway.yaml
```
时，基于
```
assets/gke-inference-gateway.yaml.tmpl
```
：除非用户明确选择外部暴露或基于InferencePool的GKE推理网关，否则Gateway资源使用
```
gatewayClassName: gke-l7-rilb
```
，并在HTTPRoute资源中将
```
/v1
```
流量路由到vLLM ClusterIP服务（端口8000上的
```
{workload_name}-vllm-svc
```
）。如果使用
```
gcsfuse-csi
```
以外的存储类（例如
```
lustre-csi
```
），请从
```
storage-config.yaml
```
中移除
```
gcsfuse.cloud.google.com
```
注解。
**请求评审与迭代：**向用户呈现生成的解决方案架构和图表，并请求反馈。在进入阶段3前，根据用户反馈迭代设计直至获得批准。

Deterministic VRAM Sizing Formula

确定性VRAM选型公式

To accurately calculate VRAM requirements for model serving/inference, use the following deterministic formula:

Hardware & Sizing Recommendation Requirements: When calculating VRAM sizing or recommending hardware:

Explain the VRAM calculation explicitly using the formula below (parameter size, KV cache, 20% safety margin).
Recommend a specific accelerator type (e.g., NVIDIA L4 with 24GB VRAM for 8B models in FP16/BF16).
Recommend using Custom Compute Classes (CCC) to maximize accelerator obtainability.
Inquire about or confirm the user's quantization tolerance (FP16/BF16 vs INT8/INT4) in the Discovery phase.

$$VRAM_{\text{total}} = \left( \frac{\text{Parameters} \times 2}{\text{Quantization}} + KV_Cache_Overhead \right) \times 1.2$$

Where:

Parameters: Model size in billions of parameters (e.g.,
```
8
```
for 8B,
```
70
```
for 70B).
Quantization: Divisor based on target precision relative to 16-bit (FP16/BF16):
- ```
1
```
  for 16-bit (FP16 / BF16, 2 bytes/param)
- ```
2
```
  for 8-bit (FP8 / INT8, 1 byte/param)
- ```
4
```
  for 4-bit (INT4 / AWQ / GPTQ, 0.5 bytes/param)
$KV_Cache_Overhead$: Memory (GB) reserved for key-value cache during generation: $$KV_Cache_Overhead ,(GB) = \frac{2 \times n_{\text{layers}} \times n_{\text{kv_heads}} \times d_{\text{head}} \times \text{Context Length} \times \text{Batch Size} \times \text{Precision Bytes}}{10^9}$$ (Rule of thumb: If model layer architecture details are unknown, estimate $KV_Cache_Overhead \approx 0.2 \times \text{Model Weight Memory}$).
1.2
Multiplier: 20% safety margin for CUDA context, activation memory, and serving engine overhead.

When mapping the result to an accelerator, compare $VRAM_{\text{total}}$ against the card's full memory (e.g., 24 GB for an L4), not against memory discounted by

--gpu-memory-utilization

. The 1.2 multiplier and vLLM's utilization cap reserve headroom for the same overheads; applying both double-counts the margin and pushes sizing one accelerator tier too high.

为准确计算模型服务/推理的VRAM需求，请使用以下确定性公式：

**硬件与选型建议要求：**计算VRAM选型或推荐硬件时：

使用下面的公式（参数规模、KV缓存、20%安全余量）明确解释VRAM计算过程。
推荐具体的加速器类型（例如对于FP16/BF16的8B模型，推荐NVIDIA L4（24GB VRAM））。
推荐使用**自定义计算类（CCC）**最大化加速器获取能力。
在需求梳理阶段询问或确认用户的量化容忍度（FP16/BF16 vs INT8/INT4）。

$$VRAM_{\text{total}} = \left( \frac{\text{Parameters} \times 2}{\text{Quantization}} + KV\_Cache\_Overhead \right) \times 1.2$$

其中：

Parameters：模型参数规模（以十亿为单位，例如8B模型填
```
8
```
，70B模型填
```
70
```
）。
Quantization：基于目标精度相对于16位（FP16/BF16）的除数：
- ```
1
```
  对应16位（FP16 / BF16，2字节/参数）
- ```
2
```
  对应8位（FP8 / INT8，1字节/参数）
- ```
4
```
  对应4位（INT4 / AWQ / GPTQ，0.5字节/参数）
$KV\_Cache\_Overhead$：生成过程中为键值缓存预留的内存（GB）： $$KV\Cache\Overhead \,(GB) = \frac{2 \times n{\text{layers}} \times n{\text{kv\heads}} \times d{\text{head}} \times \text{Context Length} \times \text{Batch Size} \times \text{Precision Bytes}}{10^9}$$ （经验法则：如果未知模型层架构细节，估算$KV\_Cache\_Overhead \approx 0.2 \times \text{模型权重内存}$）
1.2
乘数：为CUDA上下文、激活内存和服务引擎开销预留20%的安全余量。

将结果映射到加速器时，请将$VRAM_{\text{total}}$与显卡的完整内存（例如L4为24 GB）进行比较，而非与

--gpu-memory-utilization

折扣后的内存比较。1.2乘数和vLLM的利用率上限已为相同开销预留了空间；同时使用两者会重复计算余量，导致选型高出一个加速器层级。

Phase 3: Implementation

阶段3：部署实施

Carry out the approved design per the Execution policy: generate the commands below and either hand them to the user or, if the user has asked you to run them, execute them and report the output.

Identify deployment prerequisites: Ensure billing, APIs, and IAM permissions are in place.
Infrastructure Provisioning: Provide explicit
```
gcloud
```
commands to provision storage and cluster prerequisites (such as enabling the Cloud Storage FUSE CSI driver) and configure Workload Identity IAM bindings. Gateway API Pre-flight Check: Explicitly instruct the user to verify the Gateway API is enabled on their cluster. Recommend running
```
gcloud container clusters update <CLUSTER_NAME> --gateway-api=standard
```
before they attempt to apply the routing manifests to prevent CRD-not-found errors.
Storage Provisioning: Create the necessary storage resources (Cloud Storage bucket or Managed Lustre file system).
Model Staging: Apply conditional logic based on the design from Phase 2.
- Skipped: If the model is already in GCS and using GCS FUSE, skip to step 5.
- Hugging Face Source: If using a gated model (e.g., Llama 3, Gemma), explicitly instruct the user to run the
```
kubectl create secret generic hf-secret ...
```
  command locally before applying any jobs or deployments (see Gated Model Secret Security rules). Once the secret is created, instruct the user to run
```
kubectl apply -f model-staging-job.yaml
```
  .
- GCS to Lustre Source: Instruct the user to run
```
kubectl apply -f model-staging-job.yaml
```
  (which should be configured to use
```
gcloud storage cp
```
  or similar).
- Completion Gate: For any active staging Job, gate on completion with
```
kubectl wait --for=condition=complete job/{workload_name}-model-staging --timeout=90m
```
  (scale the timeout to the model size). If the Job fails, inspect it with
```
kubectl logs job/{workload_name}-model-staging
```
  before retrying. Explain that staging through a cluster Job avoids downloading heavy weights to the user's workstation and avoids re-downloading on every container restart.
Workload Deployment: Provide explicit
```
kubectl apply
```
commands to deploy the storage config (
```
kubectl apply -f storage-config.yaml
```
), ComputeClass (
```
kubectl apply -f ccc-profile.yaml
```
), vLLM deployment (
```
kubectl apply -f vllm-deployment.yaml
```
), and Inference Gateway (
```
kubectl apply -f gke-inference-gateway.yaml
```
). Ensure the vLLM deployment spec mounts the staged model weights from the PVC into
```
/models
```
. Ensure that if a Secret was created for a gated model, it is referenced correctly in the Deployment manifest. Set vLLM's
```
--model
```
flag to the staged local path (
```
/models/{model_name}
```
), never to the Hugging Face repo ID; a repo ID makes vLLM re-download the full weights on every pod start and silently defeats the staging step. Preserve the model's public name for API clients with
```
--served-model-name={model_id}
```
.
Verify the rollout: Confirm success with
```
kubectl rollout status deployment/{workload_name}-vllm --timeout=30m
```
and
```
kubectl get pods -l app={workload_name}-vllm -o wide
```
. If the rollout fails or times out, go directly to Troubleshooting Guidance with the observed error; do not ask the user whether the deployment succeeded when the command output already answers it. Ask the user only about outcomes the cluster cannot verify (for example, whether response quality matches the source system).

根据执行规则执行已批准的设计：生成以下命令，或交给用户运行，或在用户要求时执行命令并报告输出。

**确定部署先决条件：**确保计费、API和IAM权限已就绪。
**基础设施配置：**提供明确的
```
gcloud
```
命令配置存储和集群先决条件（例如启用Cloud Storage FUSE CSI驱动）并配置Workload Identity IAM绑定。**Gateway API预检检查：**明确指示用户验证集群上已启用Gateway API。建议用户在尝试应用路由清单前运行
```
gcloud container clusters update <CLUSTER_NAME> --gateway-api=standard
```
，以避免CRD未找到错误。
**存储配置：**创建必要的存储资源（Cloud Storage存储桶或Managed Lustre文件系统）。
**模型暂存：**根据阶段2的设计应用条件逻辑。
- **跳过：**如果模型已在GCS中且使用GCS FUSE，直接跳至步骤5。
- Hugging Face源：如果使用受限模型（例如Llama 3、Gemma），明确指示用户在应用任何Job或部署之前，在本地运行
```
kubectl create secret generic hf-secret ...
```
  命令（参见受限模型密钥安全规则）。创建密钥后，指示用户运行
```
kubectl apply -f model-staging-job.yaml
```
  。
- **GCS到Lustre源：**指示用户运行
```
kubectl apply -f model-staging-job.yaml
```
  （该Job应配置为使用
```
gcloud storage cp
```
  或类似命令）。
- **完成校验：**对于任何活动的暂存Job，使用
```
kubectl wait --for=condition=complete job/{workload_name}-model-staging --timeout=90m
```
  等待完成（根据模型规模调整超时时间）。如果Job失败，请在重试前使用
```
kubectl logs job/{workload_name}-model-staging
```
  检查日志。解释通过集群Job暂存可避免将大权重下载到用户工作站，也避免每次容器重启时重新下载。
**工作负载部署：**提供明确的
```
kubectl apply
```
命令部署存储配置（
```
kubectl apply -f storage-config.yaml
```
）、ComputeClass（
```
kubectl apply -f ccc-profile.yaml
```
）、vLLM部署（
```
kubectl apply -f vllm-deployment.yaml
```
）和推理网关（
```
kubectl apply -f gke-inference-gateway.yaml
```
）。确保vLLM部署规范将暂存的模型权重从PVC挂载到
```
/models
```
。如果为受限模型创建了Secret，确保Deployment清单中正确引用了该Secret。将vLLM的
```
--model
```
标志设置为暂存的本地路径（
```
/models/{model_name}
```
），绝不要设置为Hugging Face仓库ID；使用仓库ID会导致vLLM在每次Pod启动时重新下载完整权重，从而无声地失效暂存步骤。使用
```
--served-model-name={model_id}
```
为API客户端保留模型的公开名称。
**验证部署：**使用
```
kubectl rollout status deployment/{workload_name}-vllm --timeout=30m
```
和
```
kubectl get pods -l app={workload_name}-vllm -o wide
```
确认部署成功。如果部署失败或超时，请直接根据观察到的错误提供故障排查指导；当命令输出已能回答问题时，不要询问用户部署是否成功。仅向用户询问集群无法验证的结果（例如响应质量是否与源系统匹配）。

Phase 4: Validation and Cutover

阶段4：验证与流量切换

Verify that the deployed infrastructure meets the workload's requirements and provide instructions for traffic migration.

Phase 4 Response Requirements: Every response during Phase 4 MUST begin with the visual progress indicator:

**Migration Progress:** [○ Discovery] ➔ [○ Solution Design] ➔ [○ Implementation] ➔ [● Validation]

Health Checks: Recommend explicit health check commands to verify pod, node, and gateway status (
```
kubectl get pods
```
,
```
kubectl get nodes
```
, and
```
kubectl get gateway
```
).
Testing Inference: Verify the service is serving the model. Provide a
```
kubectl port-forward
```
command (
```
kubectl port-forward svc/{workload_name}-vllm-svc 8000:8000
```
) and a sample
```
curl
```
request to
```
/v1/chat/completions
```
to test endpoint inference. Compare responses with the previous infrastructure if applicable.
Traffic Cutover Guidance: Do NOT attempt to implement traffic migration automatically. Instead:
- Provide the internal GKE endpoint that is ready to receive traffic.
- Offer suggestions based on the relevant codebase (if any) for how to migrate the traffic over.
- Suggest a standard GKE rollout (e.g., updating the existing deployment manifest to point to the new service) without fancy canary tests or blue/green deployments.
- Offer more advanced traffic migration options only if the user explicitly asks for them.
Rollback readiness: Before any traffic moves, confirm the source service stays deployed (scaled down is fine, deleted is not) until the GKE endpoint has served production traffic for a period the user chooses. Provide the single command or config change that restores traffic to the source. Only after the user declares the migration stable, provide the commands to decommission the source service.
Compile Validation Report & Request Final Approval: Compile a validation report summarizing all health check outcomes and inference test results. You MUST explicitly request final user approval to finalize and complete the migration workflow.

验证部署的基础设施是否满足工作负载需求，并提供流量迁移说明。

**阶段4回应要求：**阶段4的每个回应必须以可视化进度指示开头：

**迁移进度：** [○ 需求梳理] ➔ [○ 解决方案设计] ➔ [○ 部署实施] ➔ [● 验证]

。

**健康检查：**推荐明确的健康检查命令验证Pod、节点和网关状态（
```
kubectl get pods
```
、
```
kubectl get nodes
```
和
```
kubectl get gateway
```
）。
**测试推理：**验证服务正在提供模型服务。提供
```
kubectl port-forward
```
命令（
```
kubectl port-forward svc/{workload_name}-vllm-svc 8000:8000
```
）和示例
```
curl
```
请求到
```
/v1/chat/completions
```
以测试端点推理能力。如有需要，将响应与之前的基础设施进行比较。
**流量切换指导：**请勿尝试自动实现流量迁移。相反：
- 提供已准备好接收流量的内部GKE端点。
- 根据相关代码库（如有）提供流量迁移建议。
- 建议使用标准GKE部署（例如更新现有部署清单指向新服务），无需复杂的金丝雀测试或蓝绿部署。
- 仅在用户明确要求时提供更高级的流量迁移选项。
**回滚准备：**在迁移任何流量前，确认源服务保持部署状态（可缩容，但不可删除），直到GKE端点已处理用户选定时长的生产流量。提供恢复流量到源服务的单个命令或配置变更。仅在用户确认迁移稳定后，提供退役源服务的命令。
**编制验证报告并请求最终批准：**编制验证报告，汇总所有健康检查结果和推理测试结果。必须明确请求用户最终批准，以完成迁移工作流。

Troubleshooting Guidance

故障排查指导

When users report issues where pods are created but the inference endpoint is not responding (or request troubleshooting help):

Recommend specific diagnostic commands:
```
kubectl logs {pod_name}
```
(to check container startup logs) and
```
kubectl describe pod {pod_name}
```
(to inspect pod initialization state).
Suggest verifying Google Cloud accelerator quotas (GPU/TPU) in the target region to ensure required resources can be provisioned.
Check for JIT Compilation / Startup Delays: If
```
curl
```
returns
```
Connection refused
```
but the Pod is
```
Running
```
, the serving engine (e.g., vLLM) may still be executing JIT compilation (such as Triton PTX or Torch Inductor) or capturing CUDA graphs. This can take several minutes after model weights are loaded. Advise the user to check
```
kubectl logs {pod_name}
```
and explicitly wait for the
```
Uvicorn running on http://0.0.0.0:8000
```
(or equivalent) log message before assuming there is a networking issue.
Recommend checking GKE Workload Identity bindings, PVC mount health (Cloud Storage FUSE), and GKE Inference Gateway listener configurations.

当用户报告Pod已创建但推理端点无响应（或请求故障排查帮助）时：

推荐特定的诊断命令：
```
kubectl logs {pod_name}
```
（检查容器启动日志）和
```
kubectl describe pod {pod_name}
```
（检查Pod初始化状态）。
建议验证目标区域的Google Cloud加速器配额（GPU/TPU），确保可配置所需资源。
**检查JIT编译/启动延迟：**如果
```
curl
```
返回
```
Connection refused
```
但Pod状态为
```
Running
```
，服务引擎（例如vLLM）可能仍在执行JIT编译（例如Triton PTX或Torch Inductor）或捕获CUDA图。这可能在模型权重加载完成后仍需数分钟。建议用户检查
```
kubectl logs {pod_name}
```
，并明确等待
```
Uvicorn running on http://0.0.0.0:8000
```
（或类似）日志消息，再假设存在网络问题。
建议检查GKE Workload Identity绑定、PVC挂载健康状态（Cloud Storage FUSE）和GKE推理网关监听器配置。

Supporting links

支持链接

Use these references as needed to ground your design choices, answer user questions, and generate implementation manifests:

按需使用以下参考资料支撑设计选择、回答用户问题并生成部署清单：