nemo-automodel-launcher-config

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Launcher Configuration

启动器配置

NeMo AutoModel supports three launch methods: interactive (torchrun), Slurm (HPC clusters), and SkyPilot (cloud-agnostic).

NeMo AutoModel支持三种启动方式：交互式（torchrun）、Slurm（HPC集群）和SkyPilot（云无关）。

Instructions

说明

For launcher questions, answer directly from this skill without inspecting the repository unless the user asks you to edit files. Keep the answer focused on the relevant launch YAML, required fields, and the expected runtime behavior.

Use these compact answer patterns for common questions:

Slurm multi-node: show a

slurm:

YAML block with

job_name

nodes

ntasks_per_node

time

account

partition

container_image

hf_home

, optional

extra_mounts

env_vars

, and

master_port

; explain that the launcher derives

WORLD_SIZE = nodes * ntasks_per_node

and sets

MASTER_ADDR

and

MASTER_PORT

SkyPilot spot: show a

skypilot:

YAML block with

cloud

accelerators

num_nodes

use_spot: true

disk_size

region

setup

, and

env_vars

; warn that spot instances can be preempted, set a short

step_scheduler.checkpoint_interval

, and resume with

restore_from.path

Nsight Systems on Slurm: show
```
slurm.nsys_enabled: true
```
alongside normal Slurm fields, say the launcher wraps the training command with
```
nsys profile
```
, and state that it produces a
```
.nsys-rep
```
report file. Treat profiling as diagnostic-only: use short profiling runs and disable it for normal production training because it adds overhead and large artifacts.

For Slurm answers, start with this minimal template and then adjust only the fields the user asked about:

yaml

slurm:
  job_name: llm_finetune
  nodes: 2
  ntasks_per_node: 8
  time: "04:00:00"
  account: my_account
  partition: batch
  container_image: nvcr.io/nvidia/nemo:dev
  hf_home: ~/.cache/huggingface
  master_port: 13742
  env_vars:
    HF_TOKEN: "${HF_TOKEN}"

For Slurm-only questions, do not discuss SkyPilot or profiling unless the user asks. For profiling questions, say the

.nsys-rep

report is written in the Slurm job working or output directory, using the launcher's Nsys output setting when one is configured.

针对启动器相关问题，请直接基于本技能内容作答，无需查看代码仓库，除非用户要求编辑文件。回答需聚焦于相关启动YAML配置、必填字段以及预期运行时行为。

对于常见问题，使用以下简洁回答模板：

Slurm多节点：展示包含

job_name

、

nodes

、

ntasks_per_node

、

time

、

account

或

partition

、

container_image

、

hf_home

、可选

extra_mounts

、

env_vars

和

master_port

的

slurm:

YAML块；说明启动器会自动计算

WORLD_SIZE = nodes * ntasks_per_node

并设置

MASTER_ADDR

和

MASTER_PORT

。

SkyPilot抢占式实例：展示包含
```
cloud
```
、
```
accelerators
```
、
```
num_nodes
```
、
```
use_spot: true
```
、
```
disk_size
```
、
```
region
```
、
```
setup
```
和
```
env_vars
```
的
```
skypilot:
```
YAML块；提醒抢占式实例可能会被收回，需设置较短的
```
step_scheduler.checkpoint_interval
```
，并通过
```
restore_from.path
```
恢复训练。
Slurm上的Nsight Systems：在常规Slurm字段旁添加
```
slurm.nsys_enabled: true
```
，说明启动器会用
```
nsys profile
```
包裹训练命令，并生成
```
.nsys-rep
```
报告文件。仅将性能分析用于诊断场景：使用短时间的分析运行，在常规生产训练中禁用该功能，因为它会带来额外开销并生成大型文件。

针对Slurm相关问题，从以下最小模板开始，仅调整用户询问的字段：

yaml

slurm:
  job_name: llm_finetune
  nodes: 2
  ntasks_per_node: 8
  time: "04:00:00"
  account: my_account
  partition: batch
  container_image: nvcr.io/nvidia/nemo:dev
  hf_home: ~/.cache/huggingface
  master_port: 13742
  env_vars:
    HF_TOKEN: "${HF_TOKEN}"

仅针对Slurm的问题，除非用户询问，否则不要讨论SkyPilot或性能分析。针对性能分析问题，说明

.nsys-rep

报告将写入Slurm作业的工作目录或输出目录，若配置了启动器的Nsys输出设置则遵循该设置。

Routing Boundary

路由边界

Use this skill only for launch mechanics: interactive execution, Slurm, SkyPilot, containers, mounts, environment variables, rendezvous settings, and profiling.

Do not use this skill for implementing or registering new model architectures, Hugging Face state-dict adapters, model files, or capability flags. Those are model onboarding tasks, not launcher configuration tasks.

本技能仅用于启动机制相关内容：交互式执行、Slurm、SkyPilot、容器、挂载、环境变量、rendezvous设置以及性能分析。

请勿将本技能用于实现或注册新模型架构、Hugging Face状态字典适配器、模型文件或功能标志。这些属于模型接入任务，而非启动器配置任务。

Launch Methods

启动方式

Interactive (default): runs torchrun on the current node. Suitable for single-node development and debugging.
Slurm: submits a batch job to an HPC cluster scheduler. Handles multi-node setup, container management, and environment configuration.
SkyPilot: cloud-agnostic job submission to AWS, GCP, Azure, Lambda, or Kubernetes. Supports spot instances.

交互式（默认）：在当前节点运行torchrun。适用于单节点开发和调试。
Slurm：向HPC集群调度器提交批处理作业。处理多节点设置、容器管理和环境配置。
SkyPilot：与云无关的作业提交，支持AWS、GCP、Azure、Lambda或Kubernetes。支持抢占式实例。

Interactive Launch

交互式启动

bash

undefined

bash

undefined

Single GPU

单GPU

automodel finetune llm -c config.yaml

Multi-GPU (all GPUs on current node)

多GPU（当前节点所有GPU）

torchrun --nproc_per_node=8 -m nemo_automodel._cli.app finetune llm -c config.yaml


No additional YAML section is needed for interactive mode. The CLI routes to torchrun automatically when no `slurm:` or `skypilot:` section is present in the config.

torchrun --nproc_per_node=8 -m nemo_automodel._cli.app finetune llm -c config.yaml


交互式模式无需额外的YAML配置段。当配置文件中不存在`slurm:`或`skypilot:`段时，CLI会自动路由到torchrun。

Slurm Configuration

Slurm配置

The

SlurmConfig

dataclass generates an SBATCH script from a template.

SlurmConfig

数据类会根据模板生成SBATCH脚本。

YAML Example

YAML示例

yaml

slurm:
  job_name: llm_finetune
  nodes: 2
  ntasks_per_node: 8
  time: "04:00:00"
  account: my_account
  partition: batch
  container_image: nvcr.io/nvidia/nemo:dev
  hf_home: ~/.cache/huggingface
  extra_mounts:
    - source: /data
      dest: /data
  env_vars:
    WANDB_API_KEY: "${WANDB_API_KEY}"
    HF_TOKEN: "${HF_TOKEN}"

yaml

slurm:
  job_name: llm_finetune
  nodes: 2
  ntasks_per_node: 8
  time: "04:00:00"
  account: my_account
  partition: batch
  container_image: nvcr.io/nvidia/nemo:dev
  hf_home: ~/.cache/huggingface
  extra_mounts:
    - source: /data
      dest: /data
  env_vars:
    WANDB_API_KEY: "${WANDB_API_KEY}"
    HF_TOKEN: "${HF_TOKEN}"

Key Fields

关键字段

```
job_name
```
: Slurm job identifier
```
nodes
```
: number of nodes to request
```
ntasks_per_node
```
: number of tasks (GPUs) per node
```
time
```
: wall-time limit in HH:MM:SS format
```
account
```
,
```
partition
```
: Slurm scheduling parameters
```
container_image
```
: Enroot/Pyxis container image path
```
nemo_mount
```
: mount point for NeMo AutoModel source inside the container
```
hf_home
```
: HuggingFace cache directory path
```
extra_mounts
```
: list of
```
VolumeMapping(source, dest)
```
for additional container bind mounts
```
master_port
```
: port for distributed communication (default 13742)
```
env_vars
```
: environment variables passed into the job
```
nsys_enabled
```
: when true, wraps the training command with
```
nsys profile
```
for Nsight Systems profiling

```
job_name
```
: Slurm作业标识符
```
nodes
```
: 请求的节点数量
```
ntasks_per_node
```
: 每个节点的任务（GPU）数量
```
time
```
: 运行时间限制，格式为HH:MM:SS
```
account
```
,
```
partition
```
: Slurm调度参数
```
container_image
```
: Enroot/Pyxis容器镜像路径
```
nemo_mount
```
: NeMo AutoModel源码在容器内的挂载点
```
hf_home
```
: HuggingFace缓存目录路径
```
extra_mounts
```
: 额外容器绑定挂载的
```
VolumeMapping(source, dest)
```
列表
```
master_port
```
: 分布式通信端口（默认13742）
```
env_vars
```
: 传递给作业的环境变量
```
nsys_enabled
```
: 设为true时，用
```
nsys profile
```
包裹训练命令以进行Nsight Systems性能分析

SkyPilot Configuration

SkyPilot配置

The

SkyPilotConfig

dataclass defines cloud job parameters.

SkyPilotConfig

数据类定义云作业参数。

YAML Example

YAML示例

yaml

skypilot:
  cloud: aws
  accelerators: "H100:8"
  num_nodes: 2
  use_spot: true
  disk_size: 200
  region: us-east-1
  setup: "pip install nemo-automodel"
  env_vars:
    HF_TOKEN: "${HF_TOKEN}"

yaml

skypilot:
  cloud: aws
  accelerators: "H100:8"
  num_nodes: 2
  use_spot: true
  disk_size: 200
  region: us-east-1
  setup: "pip install nemo-automodel"
  env_vars:
    HF_TOKEN: "${HF_TOKEN}"

Key Fields

关键字段

```
cloud
```
: target cloud provider (
```
aws
```
,
```
gcp
```
,
```
azure
```
,
```
lambda
```
,
```
kubernetes
```
)
```
accelerators
```
: GPU type and count (e.g.,
```
"H100:8"
```
,
```
"A100-80GB:4"
```
)
```
num_nodes
```
: number of cloud instances
```
use_spot
```
: use preemptible/spot instances for cost savings
```
disk_size
```
: disk size in GB per node
```
region
```
: cloud region for instance placement
```
setup
```
: shell commands to run before the training job (e.g., install dependencies)
```
env_vars
```
: environment variables for the job

```
cloud
```
: 目标云提供商（
```
aws
```
、
```
gcp
```
、
```
azure
```
、
```
lambda
```
、
```
kubernetes
```
）
```
accelerators
```
: GPU类型和数量（例如
```
"H100:8"
```
、
```
"A100-80GB:4"
```
）
```
num_nodes
```
: 云实例数量
```
use_spot
```
: 使用可抢占/抢占式实例以节省成本
```
disk_size
```
: 每个节点的磁盘大小（GB）
```
region
```
: 实例部署的云区域
```
setup
```
: 训练作业前运行的shell命令（例如安装依赖）
```
env_vars
```
: 作业的环境变量

SkyPilot spot checklist

SkyPilot抢占式实例检查清单

When using spot or preemptible instances:

Set
```
use_spot: true
```
in the
```
skypilot:
```
section.

Include

accelerators

num_nodes

disk_size

region

setup

, and required

env_vars

Use short checkpoint intervals in the recipe, for example
```
step_scheduler.checkpoint_interval
```
, because spot instances can be preempted.
Resume from the most recent checkpoint after preemption with the recipe's
```
restore_from
```
setting.

Minimal spot-resume recipe keys:

yaml

step_scheduler:
  checkpoint_interval: 100

restore_from:
  path: /checkpoints/latest

使用抢占式或可抢占实例时：

在
```
skypilot:
```
段中设置
```
use_spot: true
```
。

包含

accelerators

、

num_nodes

、

disk_size

、

region

、

setup

以及必填的

env_vars

。

在训练配置中设置较短的检查点间隔，例如
```
step_scheduler.checkpoint_interval
```
，因为抢占式实例可能会被收回。
抢占后通过训练配置的
```
restore_from
```
设置从最近的检查点恢复训练。

最小化的抢占恢复训练配置项：

yaml

step_scheduler:
  checkpoint_interval: 100

restore_from:
  path: /checkpoints/latest

Multi-Node Environment

多节点环境

For multi-node training (both Slurm and SkyPilot), the launcher automatically configures:

```
MASTER_ADDR
```
: hostname of the first node
```
MASTER_PORT
```
: port for rendezvous (default 13742)
```
WORLD_SIZE
```
: total number of processes (
```
nodes * ntasks_per_node
```
)
NCCL environment variables for optimized collective communication

对于多节点训练（Slurm和SkyPilot均适用），启动器会自动配置：

```
MASTER_ADDR
```
: 第一个节点的主机名
```
MASTER_PORT
```
: rendezvous端口（默认13742）
```
WORLD_SIZE
```
: 总进程数（
```
nodes * ntasks_per_node
```
）
用于优化集合通信的NCCL环境变量

Nsys Profiling

Nsys性能分析

Enable Nsight Systems profiling in Slurm jobs:

yaml

slurm:
  job_name: llm_profile
  nodes: 1
  ntasks_per_node: 8
  time: "00:30:00"
  account: my_account
  partition: batch
  container_image: nvcr.io/nvidia/nemo:dev
  nsys_enabled: true

This is a Slurm launcher setting. Normal Slurm fields such as

job_name

nodes

ntasks_per_node

time

account

partition

, and

container_image

still apply.

When

nsys_enabled: true

, the launcher wraps the training command with

nsys profile

and writes a

.nsys-rep

report file for performance analysis in the Slurm job working or output directory. Profiling is diagnostic-only: run it for a short investigation, expect overhead and large artifacts, and turn it off for normal production training.

在Slurm作业中启用Nsight Systems性能分析：

yaml

slurm:
  job_name: llm_profile
  nodes: 1
  ntasks_per_node: 8
  time: "00:30:00"
  account: my_account
  partition: batch
  container_image: nvcr.io/nvidia/nemo:dev
  nsys_enabled: true

这是Slurm启动器的设置。常规Slurm字段如

job_name

、

nodes

、

ntasks_per_node

、

time

、

account

或

partition

以及

container_image

仍然适用。

当

nsys_enabled: true

时，启动器会用

nsys profile

包裹训练命令，并在Slurm作业的工作目录或输出目录中写入

.nsys-rep

报告文件用于性能分析。性能分析仅用于诊断场景：仅在短时间调查中使用，会带来额外开销并生成大型文件，常规生产训练中需关闭该功能。

Code Anchors

代码锚点

```
components/launcher/slurm/config.py
```
- SlurmConfig dataclass, VolumeMapping
```
components/launcher/slurm/template.py
```
- SBATCH script template generation
```
components/launcher/slurm/utils.py
```
- Slurm submission utilities
```
components/launcher/skypilot/config.py
```
- SkyPilotConfig dataclass
```
_cli/app.py
```
- CLI entry point and launcher routing logic

```
components/launcher/slurm/config.py
```
- SlurmConfig数据类、VolumeMapping
```
components/launcher/slurm/template.py
```
- SBATCH脚本模板生成
```
components/launcher/slurm/utils.py
```
- Slurm提交工具
```
components/launcher/skypilot/config.py
```
- SkyPilotConfig数据类
```
_cli/app.py
```
- CLI入口点和启动器路由逻辑

Pitfalls

注意事项

Port collisions: if the default
```
master_port
```
(13742) is in use by another job on the same node, change it to avoid connection failures.
Container mounts: the
```
source
```
path in
```
extra_mounts
```
must exist on all nodes in the allocation. Missing paths cause container startup failures.
Slurm fault tolerance: the fault tolerance plugin is Slurm-specific and does not work with SkyPilot or interactive mode.
SkyPilot spot preemption: spot instances (
```
use_spot: true
```
) may be preempted by the cloud provider. Enable checkpointing with short intervals to minimize lost work.
Environment variable syntax: use
```
${VAR}
```
syntax in YAML for shell variable expansion. Bare variable names will not be expanded.
Time limit vs async checkpoint: if the Slurm
```
time
```
limit is too short, an in-progress async checkpoint write may be killed before completion, resulting in a corrupted checkpoint. Leave at least 5-10 minutes of margin.

端口冲突：如果默认
```
master_port
```
（13742）被同一节点上的其他作业占用，请修改端口以避免连接失败。
容器挂载：
```
extra_mounts
```
中的
```
source
```
路径必须在分配的所有节点上存在。路径缺失会导致容器启动失败。
Slurm容错：容错插件是Slurm专用的，不适用于SkyPilot或交互式模式。
SkyPilot抢占式实例收回：抢占式实例（
```
use_spot: true
```
）可能会被云提供商收回。启用短间隔检查点以减少工作损失。
环境变量语法：在YAML中使用
```
${VAR}
```
语法进行shell变量展开。裸变量名不会被展开。
时间限制与异步检查点：如果Slurm的
```
time
```
限制过短，正在进行的异步检查点写入可能会在完成前被终止，导致检查点损坏。请预留至少5-10分钟的余量。