launch

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Launch: pre-flight checklist for long ML training jobs

启动：长时间ML训练作业的预检查清单

Long training jobs are expensive to fail. A 12-hour run that crashes on epoch 3 from a missing dataset path or a default

workers=8

against an NFS mount is a full day lost. This skill walks five quick checks before you commit the GPUs.

The agentic Stop hook in this plugin will route here from

reason

when an assistant tries to launch a run without going through the checklist.

长时间训练作业一旦失败，代价高昂。比如一个12小时的运行在第3个epoch因数据集路径缺失或针对NFS挂载使用默认

workers=8

而崩溃，就会浪费一整天的时间。本技能会在你占用GPU前完成五项快速检查。

当助手尝试跳过清单直接启动任务时，本插件中的智能停止钩子会根据

reason

将请求引导至此处。

When to run

适用场景

The user just asked to:

launch / kick off / start / fire up a training run
restart a run that died
kill a current run (also runs the cleanup half of the checklist)
review a launch command before submitting

Or the user is about to run any of:

python train.py

accelerate launch

torchrun

deepspeed

sbatch train.sh

tmux new-session ... python ... train

wandb sweep

当用户提出以下需求时：

启动 / 开启 / 发起训练任务
重启已崩溃的任务
终止当前运行的任务（此时会执行清单中的清理部分）
在提交前审核启动命令

或者用户即将执行以下任一命令：

python train.py

、

accelerate launch

、

torchrun

、

deepspeed

、

sbatch train.sh

、

tmux new-session ... python ... train

、

wandb sweep

。

The checklist

检查清单

1. Config diff against a reference run

1. 与参考任务对比配置差异

The most expensive failure is launching with the wrong knobs. Before starting:

bash

find configs/ recipes/ experiments/ -maxdepth 3 \( -name '*.yaml' -o -name '*.yml' -o -name '*.json' -o -name '*.toml' \) -mtime -30 2> /dev/null | head

Pick the most-recently-modified config that resembles the intended run (same model family, same task). Diff against the intended config:

bash

diff -u configs/baseline_v1.yaml configs/intended.yaml

Walk every diff line. For each, ask: is this difference intentional and motivated, or is it a stale default I forgot to set? Common silent regressors:

```
num_workers
```
/ dataloader workers (default in many repos is 8: wrong on NFS)
```
batch_size
```
(per-device vs global mismatch under DDP)
```
learning_rate
```
(linearly scaled with batch size; if batch changed, lr should too)
```
optimizer
```
betas / weight decay (paper-default vs framework-default)
```
mixed_precision
```
(
```
fp16
```
vs
```
bf16
```
matters for some models)
```
gradient_accumulation_steps
```
```
seed
```
(still set if you care about reproducibility)

If no reference exists in this project, ask the user to point at one. Do not launch with framework defaults alone.

代价最高的失败就是用错误的参数启动任务。开始前执行：

bash

find configs/ recipes/ experiments/ -maxdepth 3 \( -name '*.yaml' -o -name '*.yml' -o -name '*.json' -o -name '*.toml' \) -mtime -30 2> /dev/null | head

选择最近修改的、与目标任务相似的配置（同一模型系列、同一任务），与目标配置对比差异：

bash

diff -u configs/baseline_v1.yaml configs/intended.yaml

逐一查看每一行差异。针对每一处差异，思考：_这个差异是有意且合理的，还是我忘记修改的过时默认值？_常见的隐性问题包括：

```
num_workers
```
/ 数据加载器工作进程数（很多仓库默认值为8，在NFS上并不适用）
```
batch_size
```
（DDP模式下单设备与全局批次大小不匹配）
```
learning_rate
```
（需随批次大小线性缩放；若批次改变，学习率也应调整）
```
optimizer
```
的betas参数 / 权重衰减（论文默认值与框架默认值不同）
```
mixed_precision
```
（
```
fp16
```
与
```
bf16
```
对部分模型影响重大）
```
gradient_accumulation_steps
```
```
seed
```
（若在意可复现性，需保持设置）

若项目中无参考配置，请用户指定一个。切勿仅依赖框架默认值启动任务。

2. Run name discipline

2. 规范运行命名

The run name will live in wandb / neptune / checkpoint dirs / status reports for the rest of its life. It must describe the experiment in plain English without internal codes:

bad:
```
run-1
```
,
```
wave-2
```
,
```
cs-ad
```
,
```
phase2-internal
```

good:

7src-fastvit-s-featmap-mlp-dinov3

coco-baseline-bs256-lr3e-4

swin-t-imagenet-distill-from-vit-l

The pattern:

<dataset/task>-<model>-<key-config>-<distinctive-recipe-piece>

. If you can't describe the experiment from the name in one sentence, the name is wrong. The Stop hook flags any run reference that uses session-local labels.

运行名称会长期存在于wandb / neptune / checkpoint目录 / 状态报告中。必须用通俗易懂的英文描述实验，避免使用内部编码：

错误示例：
```
run-1
```
、
```
wave-2
```
、
```
cs-ad
```
、
```
phase2-internal
```

正确示例：

7src-fastvit-s-featmap-mlp-dinov3

、

coco-baseline-bs256-lr3e-4

、

swin-t-imagenet-distill-from-vit-l

命名模式：

<数据集/任务>-<模型>-<核心配置>-<独特方案细节>

。如果无法通过名称用一句话描述实验，说明命名不合理。停止钩子会标记所有使用会话本地标签的任务引用。

3. Path verification

3. 路径验证

Before launching, every path the run depends on must be confirmed to exist:

bash

undefined

启动前，必须确认任务依赖的所有路径均存在：

bash

undefined

Dataset path

数据集路径

ls -la /path/to/dataset | head

Pretrained checkpoint (if loading)

预训练权重（若加载）

ls -la /path/to/checkpoint.pt

Output directory parent (must exist; the run dir will be created)

输出目录的父目录（必须存在；任务目录会自动创建）

ls -la /path/to/runs/

Config file

配置文件

cat configs/intended.yaml | head


Never trust a path that was recalled from memory. The `destructive_path_guard.sh` hook will already block obvious cases for `rm`/`mv`, but the launch path needs the same scrutiny, a run started with a nonexistent dataset path crashes 30 minutes in instead of immediately.

cat configs/intended.yaml | head


切勿依赖记忆中的路径。`destructive_path_guard.sh`钩子已会阻止`rm`/`mv`等操作的明显错误，但启动路径同样需要仔细检查——若使用不存在的数据集路径启动任务，它不会立即崩溃，而是会在30分钟后才崩溃。

4. Monitoring setup

4. 监控设置

Auto-detect the experiment tracker:

```
WANDB_API_KEY
```
set or
```
wandb
```
import in the launcher → wandb
```
NEPTUNE_API_TOKEN
```
set → neptune
```
MLFLOW_TRACKING_URI
```
set or
```
mlflow
```
in launcher → mlflow
presence of
```
runs/
```
or
```
lightning_logs/
```
→ tensorboard
none of the above → ask the user; "no monitoring" is rarely the right answer for a multi-hour run

Confirm the run will appear under the right project / entity / experiment-name. Confirm any tags / groups for cohort comparison are set.

自动检测实验跟踪工具：

已设置
```
WANDB_API_KEY
```
或启动器中导入
```
wandb
```
→ 使用wandb
已设置
```
NEPTUNE_API_TOKEN
```
→ 使用neptune
已设置
```
MLFLOW_TRACKING_URI
```
或启动器中包含
```
mlflow
```
→ 使用mlflow
存在
```
runs/
```
或
```
lightning_logs/
```
目录 → 使用tensorboard
以上均不满足 → 询问用户；对于持续数小时的任务，“不设置监控”通常不是合理选择

确认任务会显示在正确的项目 / 实体 / 实验名称下。确认已设置用于群组对比的标签 / 分组。

5. ETA in your local timezone

5. 本地时区的预计完成时间

Estimate wall-clock duration:

epochs × seconds-per-epoch / 3600 = hours

. State the ETA in your local TZ (the system's TZ, which the

timezone_scrub.sh

hook validates against). If the run will straddle a meeting / sleep / OOO window, decide whether to defer or split.

估算实际耗时：

epochs × seconds-per-epoch / 3600 = hours

。以本地时区（系统时区，由

timezone_scrub.sh

钩子验证）说明预计完成时间。若任务运行时间会覆盖会议 / 睡眠 / 休假时段，决定是否推迟或拆分任务。

Restart and kill cleanup

重启与终止清理

If this is a restart of a previously-failed run, or a kill before launching a replacement, purge stale artifacts in this exact order:

Local checkpoint dir on the launching machine:
```
rm -rf /local/runs/<run-name>
```
(verify path first; the
```
destructive_path_guard.sh
```
will warn).
Remote artifact dir on the cluster / NFS / object store:
```
rm -rf /remote/runs/<run-name>
```
(or equivalent).
Experiment tracker run: delete via the tracker's API (
```
wandb api.run(...).delete()
```
, neptune
```
run.stop() + delete via UI
```
, etc.). Stale tracker runs corrupt later comparisons.
Scheduler reservation: cancel the SLURM job (
```
scancel <jobid>
```
), the lambda labs reservation, the cron entry, etc. Runs that "killed but the GPUs are still allocated" are a recurring waste.

Skipping any of these creates ghost state that will confuse the next launch or the next comparison.

若要重启之前失败的任务，或终止任务后启动替代任务，请按以下顺序清除过时产物：

启动机器上的本地checkpoint目录：
```
rm -rf /local/runs/<run-name>
```
（先验证路径；
```
destructive_path_guard.sh
```
会发出警告）。
集群 / NFS / 对象存储上的远程产物目录：
```
rm -rf /remote/runs/<run-name>
```
（或等效命令）。
实验跟踪工具中的任务记录：通过跟踪工具的API删除（如
```
wandb api.run(...).delete()
```
、neptune的
```
run.stop() + 通过UI删除
```
等）。过时的跟踪记录会干扰后续对比。
调度器预留资源：取消SLURM任务（
```
scancel <jobid>
```
）、Lambda Labs预留、定时任务等。“已终止但GPU仍被占用”的情况会持续造成资源浪费。

跳过任何一步都会留下“幽灵状态”，干扰下一次启动或对比。

Output

输出

When the user invokes this skill, walk the five checks (or three checks + cleanup, if killing) and report which passed and which failed. Block the launch on any failure unless the user explicitly waives the check.

For a clean launch, end with the launch command itself in a fenced block, ready to copy.

当用户调用本技能时，执行五项检查（若为终止任务则执行三项检查+清理），报告通过和未通过的项。除非用户明确豁免，否则任何检查未通过都需阻止启动。

若所有检查通过，最后输出可直接复制的启动命令，放在代码块中。