profile-model

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Profile a model on MAX

在MAX上分析模型性能

This skill answers three questions about any model you can run on MAX, whether it's a built-in architecture or a custom one loaded with

--custom-architectures

Is my GPU actually being used? (utilization, memory, clocks)
Where does my model spend the most time? (which kernels / phases)
Why is this kernel slow? (occupancy, stalls, roofline)

It works from a

pip

pixi

install of MAX, driving the public

max

CLI or a Python script.

本方法可解答关于任何可在MAX上运行的模型的三个问题，无论是内置架构还是通过

--custom-architectures

加载的自定义架构：

我的GPU是否真正在被使用？（利用率、内存、时钟频率）
我的模型的时间主要消耗在哪里？（哪些内核/阶段）
为什么这个内核运行缓慢？（占用率、停滞、roofline模型）

本方法适用于通过pip或pixi安装的MAX环境，可驱动公开的

max

CLI工具或Python脚本。

How to decide what to run

如何确定要执行的分析步骤

Each question needs a deeper, more expensive capture than the last. Work from the cheapest check toward the most invasive, and let each result decide whether going deeper is even worth it — don't capture more than the question needs.

Start with the utilization check. If the GPU is idle or lightly loaded while the workload runs, the bottleneck is host-side (CPU, transfers, small batch) — stop and report that. A kernel trace won't tell you anything a busy GPU wouldn't.
If the GPU is busy, capture a kernel breakdown. This is the common case and usually the final answer: it shows which kernels dominate GPU time.
Only if one kernel dominates and you need to know why, do a single-kernel deep dive on that one kernel. Skip this unless a breakdown has already pointed at a specific kernel — it replays the kernel many times and is slow.

What you're answering	Tool	Cost
Is the GPU busy?	`max.profiler.gpu` (pure Python, NVIDIA + AMD)	seconds, no extra installs
Where does time go?	`max ... --profile` , or `nsys` (NVIDIA) / `rocprofv3` (AMD)	a couple of profiled runs
Why is one kernel slow?	Nsight Compute ( `ncu` ), NVIDIA only	one slow capture per kernel

Read the reference for the step you're on rather than loading all of them:

Utilization check →

references/utilization-api.md

(and the bundled

scripts/gpu_snapshot.py

)

Kernel breakdown, NVIDIA →
```
references/nsys.md
```
Kernel breakdown, AMD →
```
references/rocprofv3.md
```
Single-kernel deep dive (NVIDIA) →
```
references/ncu.md
```

每个问题所需的捕获操作比上一个更深入、成本更高。从成本最低的检查逐步转向侵入性最强的分析，并根据每个结果判断是否有必要进一步深入——不要进行超出问题需求的捕获操作。

从利用率检查开始。如果在工作负载运行时GPU处于空闲或轻负载状态，那么瓶颈在主机端（CPU、数据传输、小批量）——此时应停止分析并报告该结论。内核跟踪对于未充分利用的GPU无法提供有效信息。
如果GPU处于繁忙状态，则捕获内核细分信息。这是常见情况，通常也是最终结论：它会显示哪些内核占据了GPU的主要运行时间。
仅当单个内核占据主导地位且你需要了解其原因时，才对该单个内核进行深度分析。除非内核细分分析已指向特定内核，否则跳过此步骤——该步骤会多次重放内核，运行速度较慢。

要解答的问题	工具	成本
GPU是否繁忙？	`max.profiler.gpu` （纯Python，支持NVIDIA + AMD）	耗时数秒，无需额外安装
时间消耗在哪里？	`max ... --profile` ，或 `nsys` （NVIDIA）/ `rocprofv3` （AMD）	需执行几次分析运行
为什么单个内核运行缓慢？	Nsight Compute（ `ncu` ），仅支持NVIDIA	每个内核需执行一次耗时的捕获

请阅读当前步骤对应的参考文档，而非全部加载：

利用率检查 →

references/utilization-api.md

（以及附带的

scripts/gpu_snapshot.py

脚本）

内核细分分析（NVIDIA） →
```
references/nsys.md
```
内核细分分析（AMD） →
```
references/rocprofv3.md
```
单个内核深度分析（NVIDIA） →
```
references/ncu.md
```

Models loaded with

--custom-architectures

通过

--custom-architectures

加载的模型

Profiling itself works the same for a custom architecture as for a built-in one. What differs is loading: a custom architecture has extra prerequisites and failure modes — import compatibility with your installed MAX version,

PYTHONPATH

traps, architecture-name collisions, base models with no chat template, and a

max serve

limitation for custom embeddings models. Those stop the model from loading before any profiling can run. If you're profiling a model passed via

--custom-architectures

, read

references/custom-architectures.md

first.

自定义架构的分析流程与内置架构相同，区别在于加载环节：自定义架构有额外的前提条件和失败模式——与已安装的MAX版本的导入兼容性、

PYTHONPATH

陷阱、架构名称冲突、无聊天模板的基础模型，以及

max serve

对自定义嵌入模型的限制。这些问题会导致模型在分析开始前无法加载。如果你要分析通过

--custom-architectures

传入的模型，请先阅读

references/custom-architectures.md

。

Reading the result (what the kernel mix tells you)

解读分析结果（内核组合能告诉你什么）

Profiling answers two things: is the GPU saturated (the utilization check) and where does time go (the kernel breakdown). Map what you see back to a diagnosis:

Decode dominated by
gemv_split_k_*
GEMVs, with
*_mha_*
attention and
rms_norm
below — this is the normal, healthy shape for a dense LLM token generation (decode is memory-bound GEMV). Nothing to chase.
Sparse-MoE decode dominated by grouped / block-scaled matmul plus routing kernels — the per-expert grouped matmul (
```
*grouped*
```
/
```
block_scaled_*
```
matmul) is the dominant kernel instead of a dense GEMM, alongside top-k routing (
```
topk_*
```
,
```
moe_create_indices
```
) and expert gather/scatter; on multi-GPU, EP
```
dispatch
```
/
```
combine
```
collectives appear too. This is the healthy shape for a Mixture-of-Experts model — routing and gather overhead is expected, and only worth chasing if it rivals the matmul itself.
Low-bit weights (FP8 / FP4 / NVFP4) add quant-prep kernels — alongside the matmul you'll see dynamic activation-quantization and scale-layout kernels (
```
*quantize*
```
,
```
*block_scales*
```
) and block-scaled matmul variants rather than plain
```
gemv
```
/
```
gemm
```
. Time spent in quant/dequant prep is normal for a low-bit model; flag it only if it dwarfs the matmul it feeds. Kernel prefixes are GPU-arch-specific (
```
sm100_*
```
on Blackwell, different on Hopper / MI) — match on the kernel family, not the exact name.
A custom architecture not hitting the expected fused kernel (for example attention showing as generic
```
elementwise
```
/
```
matmul
```
instead of a
```
*_mha_*
```
kernel, or norms/RoPE unfused) — a wiring signal: the graph may not be built the way you think, even if logits pass. Worth a look during bring-up.
Pathologically low decode tok/s + an outsized prefill-shaped kernel mix repeating every step — classic O(n²) re-prefill (no/broken KV cache). This is a correctness-adjacent bug, not just slowness.
Low
gpu_usage_percent
while a workload runs — host-bound, launch/sync overhead, or batch too small; the GPU isn't the bottleneck.
Throttle reasons present — the GPU is clock-limited; numbers aren't representative until you address thermal/power.

分析可解答两个问题：GPU是否饱和（利用率检查）和时间消耗在哪里（内核细分分析）。将观察结果映射到诊断结论：

解码阶段以
gemv_split_k_*
类GEMV内核为主，
*_mha_*
注意力内核和
rms_norm
内核占比次之——这是密集型LLM令牌生成（解码阶段受内存限制的GEMV）的正常、健康状态。无需进一步排查。
稀疏MoE解码阶段以分组/块缩放矩阵乘法内核和路由内核为主——每个专家的分组矩阵乘法（
```
*grouped*
```
/
```
block_scaled_*
```
矩阵乘法）是主导内核，而非密集型GEMM，同时伴随top-k路由（
```
topk_*
```
、
```
moe_create_indices
```
）和专家聚合/分散操作；在多GPU环境下，还会出现EP
```
dispatch
```
/
```
combine
```
集合操作。这是混合专家（Mixture-of-Experts）模型的健康状态——路由和聚合开销是预期内的，只有当开销与矩阵乘法相当的时候才值得排查。
低比特权重（FP8/FP4/NVFP4）会增加量化准备内核——除矩阵乘法外，你会看到动态激活量化和缩放布局内核（
```
*quantize*
```
、
```
*block_scales*
```
）以及块缩放矩阵乘法变体，而非普通的
```
gemv
```
/
```
gemm
```
。量化/反量化准备阶段的时间消耗对于低比特模型是正常的；只有当该阶段的耗时远超其供给的矩阵乘法时才需要标记。内核前缀是GPU架构特定的（Blackwell架构为
```
sm100_*
```
，Hopper/MI架构则不同）——请匹配内核类别，而非确切名称。
自定义架构未命中预期的融合内核（例如注意力显示为通用
```
elementwise
```
/
```
matmul
```
而非
```
*_mha_*
```
内核，或归一化/RoPE未融合）——这是一个信号：计算图的构建方式可能与你的预期不符，即使logits输出正确。在上线阶段值得排查。
解码tok/s极低 + 重复出现的超大预填充形状内核组合——典型的O(n²)重复预填充（无KV缓存或KV缓存损坏）。这是与正确性相关的bug，而非单纯的性能问题。
工作负载运行时
gpu_usage_percent
值低——瓶颈在主机端，存在启动/同步开销，或批量大小过小；GPU并非瓶颈。
存在节流原因——GPU受时钟频率限制；在解决散热/电源问题前，数据不具备代表性。

The durable algorithm

持久化算法

Tooling and MAX packaging change often; this shape is the part worth preserving. Whichever capture you run, follow it:

Detect the environment before committing to a tool. Run
```
nvidia-smi
```
or
```
rocm-smi
```
to learn the vendor — the kernel-breakdown and deep-dive tools differ by vendor. Confirm the profiler is installed (
```
which nsys
```
/
```
which rocprofv3
```
/
```
which ncu
```
); if it's missing, tell the user the exact install line (see the reference) rather than failing midway.
Confirm before any long-lived run. Profiling
```
max serve
```
warms a compile cache (cold compile can take minutes), launches a server, and runs a benchmark. Before doing that, show the user the model, the flags, and the planned commands, and wait for confirmation. A one-shot
```
max generate --profile
```
on a tiny model is cheap enough to skip this.
Warm caches first, profile second. The first run of a model pays one-time compile and weight-load costs that drown out the real kernel time. Do an unprofiled warm-up run, then the profiled run, so the capture reflects steady-state inference — not compilation.
Keep the capture window small. A 10-second window at concurrency 1 is enough to see the kernel mix. Long captures produce huge trace files that are slow to open and no more informative.
Fail fast. After launching a server, check within a few seconds that the process is alive and the log has no early error (model not found, OOM, bad flag) before you start polling for
```
/health
```
. Don't wait out a full timeout on a server that already died.
Verify the artifact, then report. Confirm the output file exists and is non-trivial in size before claiming success. Report the artifact path, how to open it, and the headline numbers (top kernels, or peak GPU utilization).
Clean up only what you started. Kill the server / benchmark process group you launched. Avoid broad
```
pkill -f max
```
on a shared box — you may stop someone else's run.

工具和MAX打包方式经常变化；以下流程是值得保留的核心部分。无论执行哪种捕获操作，请遵循以下步骤：

在确定使用的工具前先检测环境。运行
```
nvidia-smi
```
或
```
rocm-smi
```
了解GPU厂商——内核细分分析和深度分析工具因厂商而异。确认分析工具已安装（
```
which nsys
```
/
```
which rocprofv3
```
/
```
which ncu
```
）；如果缺失，请告知用户确切的安装命令（参考对应文档），而非中途失败。
在执行任何长时间运行的操作前确认。分析
```
max serve
```
会预热编译缓存（冷编译可能需要数分钟）、启动服务器并运行基准测试。在此之前，向用户展示模型、参数和计划执行的命令，并等待确认。对于小型模型的一次性
```
max generate --profile
```
操作，成本足够低可跳过此步骤。
先预热缓存，再执行分析。模型的首次运行会产生一次性的编译和权重加载成本，这会掩盖真实的内核运行时间。先执行一次未分析的预热运行，然后再执行分析运行，这样捕获的数据才能反映稳态推理——而非编译过程。
保持捕获窗口较小。并发数为1时，10秒的捕获窗口足以观察内核组合。长时间捕获会生成巨大的跟踪文件，打开速度慢且不会提供更多信息。
快速失败。启动服务器后，在几秒内检查进程是否存活，日志中是否存在早期错误（模型未找到、内存不足、参数错误），然后再开始轮询
```
/health
```
接口。不要在已崩溃的服务器上等待超时结束。
验证产物后再报告。确认输出文件存在且大小合理后再宣告成功。报告产物路径、打开方式和关键数据（顶级内核或峰值GPU利用率）。
仅清理你启动的进程。终止你启动的服务器/基准测试进程组。在共享机器上避免使用宽泛的
```
pkill -f max
```
命令——你可能会终止他人的运行进程。

Nightly vs stable

夜间版 vs 稳定版

Profiling features land in nightly before stable, so this skill targets the nightly

max-serve

build. Install it with pixi:

bash

pixi init quickstart -c https://conda.modular.com/max-nightly/ -c conda-forge
cd quickstart && pixi add max-serve
pixi run max --version          # expect a *.dev build

The utilization API (

GPUDiagContext

BackgroundRecorder

) ships in the conda

max

package, so install via pixi or conda rather than a plain

pip install max

wheel, which may not expose

max.profiler.gpu

. If a command below 404s or an import fails, confirm you're on a recent nightly build.

分析功能会先在夜间版发布，再推送到稳定版，因此本方法针对夜间版

max-serve

构建。通过pixi安装：

bash

pixi init quickstart -c https://conda.modular.com/max-nightly/ -c conda-forge
cd quickstart && pixi add max-serve
pixi run max --version          # 预期为*.dev版本

利用率API（

GPUDiagContext

、

BackgroundRecorder

）随conda版

max

包发布，因此请通过pixi或conda安装，而非单纯的

pip install max

wheel包，后者可能未暴露

max.profiler.gpu

。如果以下命令返回404或导入失败，请确认你使用的是最新的夜间版构建。

Install notes

安装说明

MAX itself: a project with the nightly
```
max-serve
```
package installed via pixi (see Nightly vs stable above). All
```
max
```
CLI commands below assume you can run
```
pixi run max ...
```
in that project. The
```
max-serve
```
package includes extra dependencies required to serve a model with MAX, which aren't included with the main
```
max
```
package.
```
nsys
```
and
```
ncu
```
ship with the CUDA Toolkit. If
```
which nsys
```
fails but CUDA is present,
```
export PATH=/usr/local/cuda/bin:$PATH
```
. Otherwise install Nsight Systems / Nsight Compute from NVIDIA, or
```
sudo apt-get install nsight-systems
```
.
```
rocprofv3
```
ships with ROCm (
```
/opt/rocm/bin
```
). Add it to PATH if needed.
The utilization API needs no extra tooling beyond
```
max
```
— it talks to NVML / ROCm SMI directly through
```
max.profiler.gpu
```
.

MAX本身：通过pixi安装了夜间版
```
max-serve
```
包的项目（见上方夜间版 vs 稳定版）。以下所有
```
max
```
CLI命令均假设你可在该项目中运行
```
pixi run max ...
```
。
```
max-serve
```
包包含使用MAX部署模型所需的额外依赖，这些依赖未包含在主
```
max
```
包中。
```
nsys
```
和
```
ncu
```
随CUDA Toolkit发布。如果
```
which nsys
```
命令失败但CUDA已安装，请执行
```
export PATH=/usr/local/cuda/bin:$PATH
```
。否则从NVIDIA安装Nsight Systems/Nsight Compute，或执行
```
sudo apt-get install nsight-systems
```
。
```
rocprofv3
```
随ROCm发布（路径为
```
/opt/rocm/bin
```
）。如有需要请将其添加到PATH中。
利用率API无需额外工具，仅需
```
max
```
——它通过
```
max.profiler.gpu
```
直接与NVML/ROCm SMI通信。

Fast paths

快速路径

The lowest-friction commands, each expanded in its reference file:

"Is my GPU being used?" → run
```
scripts/gpu_snapshot.py
```
alongside a short benchmark. Pure Python, works on NVIDIA and AMD.
"Give me a kernel breakdown, fast." →
```
pixi run max generate --model <m> --prompt "hello" --num-warmups 1 --profile
```
. Prints a ranked top-N GPU-kernel table and writes a
```
.nsys-rep
```
(NVIDIA). Falls back to a CPU summary without a GPU.
"Profile my serving benchmark." → the serving-endpoint flow with
```
MODULAR_ENABLE_PROFILING=detailed
```
+
```
nsys launch
```
/
```
rocprofv3
```
.

最低摩擦的命令，每个命令在对应的参考文档中有详细说明：

“我的GPU是否在被使用？” → 在短时间基准测试运行时同步执行
```
scripts/gpu_snapshot.py
```
脚本。纯Python实现，支持NVIDIA和AMD。
“快速给我内核细分信息。” →
```
pixi run max generate --model <m> --prompt "hello" --num-warmups 1 --profile
```
。输出排名前N的GPU内核表格，并生成
```
.nsys-rep
```
文件（NVIDIA）。无GPU时会回退到CPU摘要。
“分析我的服务基准测试。” → 采用服务端流程，设置
```
MODULAR_ENABLE_PROFILING=detailed
```
+
```
nsys launch
```
/
```
rocprofv3
```
。

profile-model

Original

Translation

Profile a model on MAX

在MAX上分析模型性能

How to decide what to run

如何确定要执行的分析步骤

Models loaded with
`--custom-architectures`

通过
`--custom-architectures`
加载的模型

Reading the result (what the kernel mix tells you)

解读分析结果（内核组合能告诉你什么）

The durable algorithm

持久化算法

Nightly vs stable

夜间版 vs 稳定版

Install notes

安装说明

Fast paths

快速路径

profile-model

Original

Translation

Profile a model on MAX

在MAX上分析模型性能

How to decide what to run

如何确定要执行的分析步骤

Models loaded with --custom-architectures

通过--custom-architectures加载的模型

Reading the result (what the kernel mix tells you)

解读分析结果（内核组合能告诉你什么）

The durable algorithm

持久化算法

Nightly vs stable

夜间版 vs 稳定版

Install notes

安装说明

Fast paths

快速路径

Models loaded with
`--custom-architectures`

通过
`--custom-architectures`
加载的模型