spark-training-gotchas

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Spark Training Gotchas

Spark训练常见陷阱

DGX Spark's GB10 chip (Grace Blackwell, SM121, 128GB unified memory, aarch64) has ten recurring failure modes across launch, memory, thermals, bandwidth, and precision. Each is named G1–G10 so it can be checked by number — the numbering is load-bearing for tooling that runs these checks. Read this before a long run, not after hour six.

DGX Spark的GB10芯片（Grace Blackwell，SM121，128GB统一内存，aarch64架构）在启动、内存、散热、带宽和精度方面存在十种反复出现的故障模式。每种模式被命名为G1–G10，可通过编号进行检查——该编号对运行这些检查的工具至关重要。请在长时间运行前阅读本文，而非在运行六小时后才看。

When to Use This Skill

何时使用本技能

A training run fails to start, with an import error or a segfault that doesn't point at the real cause.
A run OOMs while
```
nvidia-smi
```
still shows headroom.
Throughput degrades partway through a run that started fine.
Before any multi-hour or multi-epoch job on GB10.
Wiring two Sparks together, before picking a parallelism strategy.
Choosing between FP8 and NVFP4 for a Spark-hosted run.

训练任务无法启动，出现导入错误或未指向真正原因的段错误。
运行时出现OOM，但
```
nvidia-smi
```
仍显示有剩余内存空间。
初始运行正常，但中途吞吐量下降。
在GB10上执行任何数小时或多轮次训练任务前。
连接两个Spark实例，选择并行策略前。
为Spark托管的任务选择FP8和NVFP4精度前。

Common Issues Quick Reference

常见问题速查

#	Symptom	Fix
G1	undefined symbol / segfault	cu130 wheel or container
G2	flash-attn wrong backend used	skip pip build; monkeypatch on NGC
G3	OOM despite headroom	drop page cache
G4	throughput drop / reboot	expect ~100W sustained cap
G5	memory-bound step slow	budget 180–192 GB/s
G6	cache evicted mid-run	one GPU server at a time
G7	NVFP4 slower than FP8	stay FP8 unless `sm_121a`
G8	playbook fails outright	check upstream issues
G9	env breaks after install	use a container
G10	2-Spark TP hangs	DDP/FSDP only, never TP

#	症状	修复方案
G1	未定义符号/段错误	使用cu130版本的wheel包或容器
G2	flash-attn使用错误后端	跳过pip编译；在NGC上使用猴子补丁
G3	仍有剩余内存却出现OOM	清除页缓存
G4	吞吐量下降/重启	预期持续功耗上限约为100W
G5	内存受限导致步骤变慢	按180–192 GB/s的带宽预算规划
G6	运行中途缓存被驱逐	一次仅运行一个GPU服务器
G7	NVFP4比FP8慢	除非针对 `sm_121a` 优化，否则保持使用FP8
G8	操作手册完全失效	检查上游问题
G9	安装后环境损坏	使用容器
G10	双Spark TP（张量并行）挂起	仅使用DDP/FSDP，绝不使用TP

The Ten Gotchas

十大陷阱

G1: CUDA 12/13 ABI Mismatch

G1: CUDA 12/13 ABI不匹配

SYMPTOM:
```
ImportError: undefined symbol
```
naming a CUDA function, or a segfault on the first
```
.cuda()
```
call.
CAUSE: most PyPI wheels link
```
libcudart.so.12
```
; Spark ships CUDA 13. pip never checks CUDA ABI, so it surfaces only at import or first kernel launch.
CHECK:
```
references/gotcha-checks.md
```
G1 — the wheel's CUDA build tag.
FIX: reinstall from
```
download.pytorch.org/whl/cu130
```
or use a matched container.

症状：
```
ImportError: undefined symbol
```
提示某个CUDA函数，或首次调用
```
.cuda()
```
时出现段错误。
原因： 大多数PyPI wheel包链接
```
libcudart.so.12
```
；而Spark自带CUDA 13。pip不会检查CUDA ABI，因此仅在导入或首次启动内核时暴露问题。
检查： 参考
```
references/gotcha-checks.md
```
中的G1——查看wheel包的CUDA构建标签。
修复： 从
```
download.pytorch.org/whl/cu130
```
重新安装，或使用匹配的容器。

G2: flash-attn — Skip the pip Build, Watch Unsloth's Auto-Detect

G2: flash-attn — 跳过pip编译，关注Unsloth的自动检测

SYMPTOM:
```
pip install flash-attn
```
still fails/hangs. Unsloth may also silently train flash-attn over an explicitly requested SDPA.
CAUSE: no aarch64/sm_121 wheel for bare pip — but NGC containers ship a working SM121 flash-attn, and Unsloth auto-prefers it, dropping
```
attn_implementation="sdpa"
```
.
CHECK:
```
references/gotcha-checks.md
```
G2 — is flash-attn already present and working.
FIX: bare pip — skip flash-attn, use SDPA (unchanged). On NGC — the only reliable override is the monkeypatch in
```
references/gotcha-checks.md
```
G2.

症状：
```
pip install flash-attn
```
仍失败/挂起。Unsloth可能会在显式指定SDPA的情况下，静默使用flash-attn进行训练。
原因： 裸pip没有针对aarch64/sm_121的wheel包——但NGC容器自带可正常工作的SM121版本flash-attn，且Unsloth会自动优先使用它，忽略
```
attn_implementation="sdpa"
```
配置。
检查： 参考
```
references/gotcha-checks.md
```
中的G2——确认flash-attn是否已存在且可正常工作。
修复： 裸pip环境——跳过flash-attn，使用SDPA（无需修改）。NGC环境——唯一可靠的覆盖方式是
```
references/gotcha-checks.md
```
G2中的猴子补丁。

G3: UMA OOM Below 128GB

G3: UMA（统一内存架构）在128GB以下出现OOM

SYMPTOM: OOM during model load/training while
```
nvidia-smi
```
still reports free memory under the 128GB cap — or, on some setups,
```
[N/A]
```
outright instead of a number.
CAUSE: mmap and the CUDA allocator double-count pages during safetensors load; QLoRA can OOM earlier than bf16 since dequantization adds transient allocs.

CHECK:

references/gotcha-checks.md

G3 — read

free -g

and

/proc/meminfo

, not

nvidia-smi

FIX: drop the page cache with
```
sync; echo 3 > /proc/sys/vm/drop_caches
```
— needs root, a between-run reset, not a mid-training step.

症状： 模型加载/训练时出现OOM，但
```
nvidia-smi
```
仍显示128GB上限内有空闲内存——或在某些环境中直接显示
```
[N/A]
```
而非具体数值。
原因： 在加载safetensors时，mmap和CUDA分配器会重复计算内存页；QLoRA比bf16更早出现OOM，因为反量化会增加临时内存分配。

检查： 参考

references/gotcha-checks.md

中的G3——查看

free -g

和

/proc/meminfo

，而非

nvidia-smi

。

修复： 使用
```
sync; echo 3 > /proc/sys/vm/drop_caches
```
清除页缓存——需要root权限，且需在任务之间执行，不能在训练中途操作。

G4: Thermal Throttling

G4: 热节流

SYMPTOM: throughput drops partway through a multi-hour run, or the box spontaneously reboots under sustained load.
CAUSE: sustained power draw caps around 100W versus the 240W rated figure; long runs push into that ceiling and throttle or, sometimes, reboot.

CHECK:

references/gotcha-checks.md

G4 — sample

nvidia-smi --query-gpu=temperature.gpu,power.draw

FIX: if power plateaus under 240W while temperature climbs, treat throttling as the cause; improve cooling or cap run length.

症状： 数小时的训练任务中途吞吐量下降，或设备在持续负载下自发重启。
原因： 持续功耗上限约为100W，而非额定的240W；长时间运行会触及该上限，导致节流甚至重启。

检查： 参考

references/gotcha-checks.md

中的G4——采样

nvidia-smi --query-gpu=temperature.gpu,power.draw

的输出。

修复： 如果功耗在240W以下达到平台期，同时温度上升，则将节流视为问题原因；改善散热或限制运行时长。

G5: Bandwidth Ceiling

G5: 带宽上限

SYMPTOM: memory-bound workloads, decode-heavy RL loops especially, plateau well below expected throughput.
CAUSE: 273 GB/s is a spec ceiling, not sustained; measured bandwidth runs 180–192 GB/s.
CHECK:
```
references/gotcha-checks.md
```
G5 — observed step time vs. the measured range, not spec.
FIX: budget throughput from 180–192 GB/s; revise a plan built on the 273 GB/s figure.

症状： 内存受限的工作负载（尤其是解码密集型RL循环）的吞吐量远低于预期值。
原因： 273 GB/s是规格上限，而非持续可达的带宽；实际测得的带宽为180–192 GB/s。
检查： 参考
```
references/gotcha-checks.md
```
中的G5——对比观察到的步骤时间与实测范围，而非规格值。
修复： 按180–192 GB/s的带宽预算规划吞吐量；修改基于273 GB/s规格制定的计划。

G6: Global UMA Resource Contention

G6: 全局UMA资源竞争

SYMPTOM: a process's KV cache/weights get evicted mid-run silently, no OOM in its own logs.
CAUSE: unified memory is one global pool; an uncapped or near-capacity process competes with anything else and can evict it. A small, bounded workload doesn't — a <4GB LoRA coexists fine alongside vLLM capped at
```
gpu-memory-utilization<=0.5
```
.
CHECK:
```
references/gotcha-checks.md
```
G6 — other GPU-resident processes and whether capped.
FIX: the one-heavy-job rule applies to uncapped or near-capacity workloads — cap or stop unrelated servers first. A small, capped workload need not stop.

症状： 进程的KV缓存/权重在运行中途被静默驱逐，自身日志中无OOM记录。
原因： 统一内存是一个全局池；无限制或接近满载的进程会与其他进程竞争资源，并可能驱逐其内存。小型、受限的工作负载则不会——例如<4GB的LoRA与限制
```
gpu-memory-utilization<=0.5
```
的vLLM可共存。
检查： 参考
```
references/gotcha-checks.md
```
中的G6——查看其他驻留GPU的进程及其是否受限。
修复： 对无限制或接近满载的工作负载，遵循“一次仅运行一个重负载任务”的规则——先限制或停止无关服务器。小型、受限的工作负载无需停止。

G7: NVFP4 Slower Than FP8 on SM121

G7: SM121上NVFP4比FP8慢

SYMPTOM: switching an inference workload from FP8 to NVFP4 on Spark makes it slower, not faster.
CAUSE: SM121 lacks
```
cvt.e2m1x2
```
unless kernels target
```
sm_121a
```
; NVFP4 runs ~32% slower without it.
CHECK:
```
references/gotcha-checks.md
```
G7 — capability reports
```
(12, 1)
```
; does the build target
```
sm_121a
```
?
FIX: stay on FP8 unless the build targets
```
sm_121a
```
.

症状： 在Spark上将推理工作负载从FP8切换为NVFP4后，速度变慢而非变快。
原因： SM121缺少
```
cvt.e2m1x2
```
指令，除非内核针对
```
sm_121a
```
优化；无该优化时NVFP4速度慢约32%。
检查： 参考
```
references/gotcha-checks.md
```
中的G7——查看设备能力是否显示
```
(12, 1)
```
；构建是否针对
```
sm_121a
```
？
修复： 除非构建针对
```
sm_121a
```
优化，否则保持使用FP8。

G8: Stale Official Playbooks

G8: 过时的官方操作手册

SYMPTOM: following an official DGX Spark playbook still fails, with no local misconfiguration explaining it.
CAUSE: official playbooks have shipped broken before; the stack moves faster than the docs.
CHECK:
```
references/gotcha-checks.md
```
G8 — the playbook repo's recent issues.
FIX: check
```
github.com/NVIDIA/dgx-spark-playbooks
```
issues before trusting a recipe for an expensive run.

症状： 遵循官方DGX Spark操作手册仍失败，且无本地配置错误可解释原因。
原因： 官方操作手册曾出现过错误；技术栈更新速度快于文档。
检查： 参考
```
references/gotcha-checks.md
```
中的G8——查看操作手册仓库的近期问题。
修复： 在信任昂贵任务的操作指南前，检查
```
github.com/NVIDIA/dgx-spark-playbooks
```
的问题列表。

G9: Container-First, Not Bare Pip

G9: 优先使用容器，而非裸pip

SYMPTOM: a bare-pip environment that worked yesterday breaks after an unrelated
```
pip install
```
, or two "identical" environments behave differently.
CAUSE: bare pip lets Triton, xformers, and transformers drift independently; nothing pins them to GB10's SM121 target.
CHECK:
```
references/gotcha-checks.md
```
G9 — container or bare pip?
FIX: prefer an NGC container (see
```
spark-environment-setup
```
for tag guidance) or Unsloth's container. If bare pip is unavoidable, follow the NVIDIA install order, including
```
--no-deps
```
on Unsloth.

症状： 昨天还正常工作的裸pip环境，在执行无关的
```
pip install
```
后损坏；或两个“相同”的环境表现不同。
原因： 裸pip允许Triton、xformers和transformers独立版本漂移；没有将它们固定到GB10的SM121目标。
检查： 参考
```
references/gotcha-checks.md
```
中的G9——使用的是容器还是裸pip？
修复： 优先使用NGC容器（查看
```
spark-environment-setup
```
获取标签指导）或Unsloth的容器。如果必须使用裸pip，请遵循NVIDIA的安装顺序，包括对Unsloth使用
```
--no-deps
```
参数。

G10: Dual-Spark Is DDP/FSDP Only

G10: 双Spark仅支持DDP/FSDP

SYMPTOM: a tensor-parallel launch across two Sparks hangs, runs far slower than single-Spark, or errors out.
CAUSE: ConnectX-7 is fast enough for gradient/parameter sync (DDP, FSDP) but too thin for TP's fine-grained traffic.
CHECK:
```
references/gotcha-checks.md
```
G10 — the configured parallelism strategy.
FIX: on a two-Spark setup, choose DDP or FSDP, never tensor parallelism — TP is single-node only here.

症状： 跨两个Spark实例的张量并行（TP）启动后挂起、远慢于单Spark实例，或报错。
原因： ConnectX-7足以支持梯度/参数同步（DDP、FSDP），但带宽不足以承载TP的细粒度流量。
检查： 参考
```
references/gotcha-checks.md
```
中的G10——查看配置的并行策略。
修复： 在双Spark环境中，选择DDP或FSDP，绝不使用张量并行——TP仅适用于单节点场景。

Fast Triage

快速诊断

The cheapest checks to run before anything else:

bash

python3 -c "import torch; print(torch.version.cuda)"  # expect 13.x (G1); NGC builds have no +cu130 tag — that's not a failure

python

import torch; print(torch.cuda.get_device_capability())  # expect (12, 1) (G7)

bash

{ [ -f /.dockerenv -o -f /run/.containerenv ] || grep -qE 'docker|containerd' /proc/1/cgroup; } 2>/dev/null && echo container || echo unknown  # G9

assets/preflight.sh

runs G1, G3, G4, G7, G9 and produces one output line per gotcha in a fixed format: G-number first, then PASS/FAIL/WARN where automatable, SKIP when unavailable, or

INFO:

for a raw reading (G3, G4). Full commands:

references/gotcha-checks.md

. See also

spark-environment-setup

for the environment assumed working.

在执行其他操作前，先运行以下最简便的检查：

bash

python3 -c "import torch; print(torch.version.cuda)"  # 预期为13.x（对应G1）；NGC构建没有+cu130标签——这并非故障

python

import torch; print(torch.cuda.get_device_capability())  # 预期为(12, 1)（对应G7）

bash

{ [ -f /.dockerenv -o -f /run/.containerenv ] || grep -qE 'docker|containerd' /proc/1/cgroup; } 2>/dev/null && echo container || echo unknown  # 对应G9

assets/preflight.sh

会运行G1、G3、G4、G7、G9的检查，并以固定格式输出每个陷阱的结果：开头为G编号，然后是可自动化判断的PASS/FAIL/WARN，不可用则显示SKIP，或针对原始读数显示

INFO:

（G3、G4）。完整命令请参考

references/gotcha-checks.md

。另请查看

spark-environment-setup

了解假定正常工作的环境要求。