gke-batch-hpc

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

GKE Batch & HPC Workloads

GKE 批处理与HPC工作负载

This reference covers running batch processing and high-performance computing (HPC) workloads on GKE.

MCP Tools:

apply_k8s_manifest

get_k8s_resource

describe_k8s_resource

get_k8s_logs

delete_k8s_resource

list_k8s_events

本参考文档介绍如何在GKE上运行批处理和高性能计算（HPC）工作负载。

MCP工具：

apply_k8s_manifest

get_k8s_resource

describe_k8s_resource

get_k8s_logs

delete_k8s_resource

list_k8s_events

When to Use

适用场景

Running batch data processing pipelines
HPC simulations (CFD, molecular dynamics, financial modeling)
Large-scale parallel computation (MPI, MapReduce)
ML training jobs
CI/CD build farms

运行批处理数据处理流水线
HPC模拟（计算流体动力学CFD、分子动力学、金融建模）
大规模并行计算（MPI、MapReduce）
机器学习训练作业
CI/CD构建集群

Batch Processing on GKE

GKE上的批处理

Kubernetes Jobs

yaml

apiVersion: batch/v1
kind: Job
metadata:
  name: batch-job
spec:
  parallelism: 10
  completions: 100
  backoffLimit: 3
  template:
    spec:
      containers:
      - name: worker
        image: <IMAGE>
        resources:
          requests:
            cpu: "1"
            memory: "2Gi"
      restartPolicy: Never

yaml

apiVersion: batch/v1
kind: Job
metadata:
  name: batch-job
spec:
  parallelism: 10
  completions: 100
  backoffLimit: 3
  template:
    spec:
      containers:
      - name: worker
        image: <IMAGE>
        resources:
          requests:
            cpu: "1"
            memory: "2Gi"
      restartPolicy: Never

JobSet (for Complex Multi-Job Workflows)

JobSet（适用于复杂多作业工作流）

The golden path enables JobSet monitoring (

JOBSET

in monitoringConfig).

yaml

apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
  name: training-job
spec:
  replicatedJobs:
  - name: workers
    replicas: 4
    template:
      spec:
        parallelism: 1
        completions: 1
        template:
          spec:
            containers:
            - name: worker
              image: <IMAGE>
              resources:
                requests:
                  cpu: "4"
                  memory: "8Gi"

黄金路径支持JobSet监控（

monitoringConfig

中的

JOBSET

）。

yaml

apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
  name: training-job
spec:
  replicatedJobs:
  - name: workers
    replicas: 4
    template:
      spec:
        parallelism: 1
        completions: 1
        template:
          spec:
            containers:
            - name: worker
              image: <IMAGE>
              resources:
                requests:
                  cpu: "4"
                  memory: "8Gi"

Kueue (Job Queuing)

Kueue（作业队列）

Kueue manages job scheduling and resource allocation for batch workloads:

bash

undefined

Kueue用于管理批处理工作负载的作业调度和资源分配：

bash

undefined

Install Kueue

安装Kueue

kubectl apply --server-side -f https://github.com/kubernetes-sigs/kueue/releases/latest/download/manifests.yaml


```yaml

kubectl apply --server-side -f https://github.com/kubernetes-sigs/kueue/releases/latest/download/manifests.yaml


```yaml

Define a ClusterQueue

定义ClusterQueue

apiVersion: kueue.x-k8s.io/v1beta1 kind: ClusterQueue metadata: name: batch-queue spec: namespaceSelector: {} resourceGroups:

coveredResources: ["cpu", "memory"] flavors:
- name: default resources:
  - name: "cpu" nominalQuota: 100
  - name: "memory" nominalQuota: "200Gi"

apiVersion: kueue.x-k8s.io/v1beta1 kind: ClusterQueue metadata: name: batch-queue spec: namespaceSelector: {} resourceGroups:

coveredResources: ["cpu", "memory"] flavors:
- name: default resources:
  - name: "cpu" nominalQuota: 100
  - name: "memory" nominalQuota: "200Gi"

Allow a namespace to use the queue

允许命名空间使用该队列

apiVersion: kueue.x-k8s.io/v1beta1 kind: LocalQueue metadata: name: batch-local namespace: batch-jobs spec: clusterQueue: batch-queue

undefined

apiVersion: kueue.x-k8s.io/v1beta1 kind: LocalQueue metadata: name: batch-local namespace: batch-jobs spec: clusterQueue: batch-queue

undefined

HPC on GKE

GKE上的HPC

Compact Placement (Low-Latency Networking)

紧凑放置（低延迟网络）

For tightly-coupled HPC workloads that need low-latency inter-node communication:

bash

undefined

对于需要低延迟节点间通信的紧密耦合HPC工作负载：

bash

undefined

Standard clusters: create node pool with compact placement

标准集群：创建带有紧凑放置策略的节点池

gcloud container node-pools create hpc-pool
--cluster <CLUSTER_NAME> --region <REGION>
--machine-type c3-standard-44
--placement-type COMPACT
--num-nodes 8
--enable-autoscaling --min-nodes 0 --max-nodes 16
--quiet

undefined

undefined

MPI Workloads

MPI工作负载

Use the MPI Operator for MPI-based HPC applications:

bash

undefined

使用MPI Operator运行基于MPI的HPC应用：

bash

undefined

Install MPI Operator

安装MPI Operator

kubectl apply -f https://raw.githubusercontent.com/kubeflow/mpi-operator/master/deploy/v2beta1/mpi-operator.yaml


```yaml
apiVersion: kubeflow.org/v2beta1
kind: MPIJob
metadata:
  name: hpc-simulation
spec:
  slotsPerWorker: 4
  mpiReplicaSpecs:
    Launcher:
      replicas: 1
      template:
        spec:
          containers:
          - name: launcher
            image: <MPI_IMAGE>
            command: ["mpirun", "-np", "32", "./simulation"]
            resources:
              requests:
                cpu: "1"
                memory: "2Gi"
              limits:
                cpu: "2"
                memory: "4Gi"
    Worker:
      replicas: 8
      template:
        spec:
          containers:
          - name: worker
            image: <MPI_IMAGE>
            resources:
              requests:
                cpu: "4"
                memory: "8Gi"
              limits:
                cpu: "8"
                memory: "16Gi"

kubectl apply -f https://raw.githubusercontent.com/kubeflow/mpi-operator/master/deploy/v2beta1/mpi-operator.yaml


```yaml
apiVersion: kubeflow.org/v2beta1
kind: MPIJob
metadata:
  name: hpc-simulation
spec:
  slotsPerWorker: 4
  mpiReplicaSpecs:
    Launcher:
      replicas: 1
      template:
        spec:
          containers:
          - name: launcher
            image: <MPI_IMAGE>
            command: ["mpirun", "-np", "32", "./simulation"]
            resources:
              requests:
                cpu: "1"
                memory: "2Gi"
              limits:
                cpu: "2"
                memory: "4Gi"
    Worker:
      replicas: 8
      template:
        spec:
          containers:
          - name: worker
            image: <MPI_IMAGE>
            resources:
              requests:
                cpu: "4"
                memory: "8Gi"
              limits:
                cpu: "8"
                memory: "16Gi"

Cost Optimization for Batch/HPC

批处理/HPC的成本优化

Spot VMs for Batch

使用Spot VM运行批处理

Batch workloads are ideal Spot VM candidates (interruptible, can checkpoint). Use a ComputeClass with Spot-first priority and

activeMigration

to return to Spot when available. See the

gke-compute-classes

skill for the Spot-with-fallback pattern.

批处理工作负载非常适合使用Spot VM（可中断，支持检查点）。使用优先选择Spot的ComputeClass，并通过

activeMigration

在可用时切换回Spot。有关Spot回退模式，请参阅

gke-compute-classes

技能。

Scale-to-Zero

缩容至零

For batch clusters, allow node pools to scale to zero when no jobs are running:

Autopilot (golden path): Automatic, nodes scale to zero when no pods are scheduled
Standard: Set
```
--min-nodes 0
```
on batch node pools

对于批处理集群，允许节点池在无作业运行时缩容至零：

Autopilot（黄金路径）：自动缩容，当无Pod调度时节点缩容至零
标准集群：在批处理节点池上设置
```
--min-nodes 0
```

Best Practices & Production Guidelines

最佳实践与生产指南

Resource Quotas: Always specify resource requests and limits (CPU, memory, and optionally GPU/TPU) for all batch/HPC manifests. This is critical for Kueue admission, autoscaling, and preventing resource starvation in the cluster.
TPU/Spot Cluster Maintenance: For long-running AI training runs on Spot VMs/TPUs, advise using GKE maintenance exclusions to block automatic cluster upgrades/reboots during the active training window to minimize unnecessary preemption.
MPI Workloads: Use the Kubeflow Training Operator to orchestrate distributed MPI applications via the
```
MPIJob
```
custom resource.
Kueue & JobSet: Use Kueue for multi-tenant job queueing and fair sharing; use JobSet for multi-component tightly coupled workloads.
Resilience: Always set a
```
backoffLimit
```
on Jobs, and implement application-level checkpointing (e.g., using Orbax or PyTorch checkpointing) to survive Spot VM preemption.

资源配额：始终为所有批处理/HPC清单指定资源请求和限制（CPU、内存，可选GPU/TPU）。这对Kueue准入控制、自动扩缩容以及避免集群资源耗尽至关重要。
TPU/Spot集群维护：对于在Spot VM/TPU上运行的长时间AI训练任务，建议使用GKE维护排除功能，在活跃训练窗口期间阻止自动集群升级/重启，以最大限度减少不必要的抢占。
MPI工作负载：使用Kubeflow训练算子通过
```
MPIJob
```
自定义资源编排分布式MPI应用。
Kueue与JobSet：使用Kueue实现多租户作业排队和公平共享；使用JobSet处理多组件紧密耦合工作负载。
弹性：始终为Jobs设置
```
backoffLimit
```
，并实现应用级检查点（例如使用Orbax或PyTorch检查点），以应对Spot VM抢占。