Profile a model on MAX

This skill answers three questions about any model you can run on MAX, whether it's a built-in architecture or a custom one loaded with

--custom-architectures

Is my GPU actually being used? (utilization, memory, clocks)
Where does my model spend the most time? (which kernels / phases)
Why is this kernel slow? (occupancy, stalls, roofline)

It works from a

pip

pixi

install of MAX, driving the public

max

CLI or a Python script.

How to decide what to run

Each question needs a deeper, more expensive capture than the last. Work from the cheapest check toward the most invasive, and let each result decide whether going deeper is even worth it — don't capture more than the question needs.

Start with the utilization check. If the GPU is idle or lightly loaded while the workload runs, the bottleneck is host-side (CPU, transfers, small batch) — stop and report that. A kernel trace won't tell you anything a busy GPU wouldn't.
If the GPU is busy, capture a kernel breakdown. This is the common case and usually the final answer: it shows which kernels dominate GPU time.
Only if one kernel dominates and you need to know why, do a single-kernel deep dive on that one kernel. Skip this unless a breakdown has already pointed at a specific kernel — it replays the kernel many times and is slow.

What you're answering	Tool	Cost
Is the GPU busy?	`max.profiler.gpu` (pure Python, NVIDIA + AMD)	seconds, no extra installs
Where does time go?	`max ... --profile` , or `nsys` (NVIDIA) / `rocprofv3` (AMD)	a couple of profiled runs
Why is one kernel slow?	Nsight Compute ( `ncu` ), NVIDIA only	one slow capture per kernel

Read the reference for the step you're on rather than loading all of them:

Utilization check →

references/utilization-api.md

(and the bundled

scripts/gpu_snapshot.py

)

Kernel breakdown, NVIDIA →
```
references/nsys.md
```
Kernel breakdown, AMD →
```
references/rocprofv3.md
```
Single-kernel deep dive (NVIDIA) →
```
references/ncu.md
```

Models loaded with

--custom-architectures

Profiling itself works the same for a custom architecture as for a built-in one. What differs is loading: a custom architecture has extra prerequisites and failure modes — import compatibility with your installed MAX version,

PYTHONPATH

traps, architecture-name collisions, base models with no chat template, and a

max serve

limitation for custom embeddings models. Those stop the model from loading before any profiling can run. If you're profiling a model passed via

--custom-architectures

, read

references/custom-architectures.md

first.

Reading the result (what the kernel mix tells you)

Profiling answers two things: is the GPU saturated (the utilization check) and where does time go (the kernel breakdown). Map what you see back to a diagnosis:

Decode dominated by
gemv_split_k_*
GEMVs, with
*_mha_*
attention and
rms_norm
below — this is the normal, healthy shape for a dense LLM token generation (decode is memory-bound GEMV). Nothing to chase.
Sparse-MoE decode dominated by grouped / block-scaled matmul plus routing kernels — the per-expert grouped matmul (
```
*grouped*
```
/
```
block_scaled_*
```
matmul) is the dominant kernel instead of a dense GEMM, alongside top-k routing (
```
topk_*
```
,
```
moe_create_indices
```
) and expert gather/scatter; on multi-GPU, EP
```
dispatch
```
/
```
combine
```
collectives appear too. This is the healthy shape for a Mixture-of-Experts model — routing and gather overhead is expected, and only worth chasing if it rivals the matmul itself.
Low-bit weights (FP8 / FP4 / NVFP4) add quant-prep kernels — alongside the matmul you'll see dynamic activation-quantization and scale-layout kernels (
```
*quantize*
```
,
```
*block_scales*
```
) and block-scaled matmul variants rather than plain
```
gemv
```
/
```
gemm
```
. Time spent in quant/dequant prep is normal for a low-bit model; flag it only if it dwarfs the matmul it feeds. Kernel prefixes are GPU-arch-specific (
```
sm100_*
```
on Blackwell, different on Hopper / MI) — match on the kernel family, not the exact name.
A custom architecture not hitting the expected fused kernel (for example attention showing as generic
```
elementwise
```
/
```
matmul
```
instead of a
```
*_mha_*
```
kernel, or norms/RoPE unfused) — a wiring signal: the graph may not be built the way you think, even if logits pass. Worth a look during bring-up.
Pathologically low decode tok/s + an outsized prefill-shaped kernel mix repeating every step — classic O(n²) re-prefill (no/broken KV cache). This is a correctness-adjacent bug, not just slowness.
Low
gpu_usage_percent
while a workload runs — host-bound, launch/sync overhead, or batch too small; the GPU isn't the bottleneck.
Throttle reasons present — the GPU is clock-limited; numbers aren't representative until you address thermal/power.

The durable algorithm

Tooling and MAX packaging change often; this shape is the part worth preserving. Whichever capture you run, follow it:

Detect the environment before committing to a tool. Run
```
nvidia-smi
```
or
```
rocm-smi
```
to learn the vendor — the kernel-breakdown and deep-dive tools differ by vendor. Confirm the profiler is installed (
```
which nsys
```
/
```
which rocprofv3
```
/
```
which ncu
```
); if it's missing, tell the user the exact install line (see the reference) rather than failing midway.
Confirm before any long-lived run. Profiling
```
max serve
```
warms a compile cache (cold compile can take minutes), launches a server, and runs a benchmark. Before doing that, show the user the model, the flags, and the planned commands, and wait for confirmation. A one-shot
```
max generate --profile
```
on a tiny model is cheap enough to skip this.
Warm caches first, profile second. The first run of a model pays one-time compile and weight-load costs that drown out the real kernel time. Do an unprofiled warm-up run, then the profiled run, so the capture reflects steady-state inference — not compilation.
Keep the capture window small. A 10-second window at concurrency 1 is enough to see the kernel mix. Long captures produce huge trace files that are slow to open and no more informative.
Fail fast. After launching a server, check within a few seconds that the process is alive and the log has no early error (model not found, OOM, bad flag) before you start polling for
```
/health
```
. Don't wait out a full timeout on a server that already died.
Verify the artifact, then report. Confirm the output file exists and is non-trivial in size before claiming success. Report the artifact path, how to open it, and the headline numbers (top kernels, or peak GPU utilization).
Clean up only what you started. Kill the server / benchmark process group you launched. Avoid broad
```
pkill -f max
```
on a shared box — you may stop someone else's run.

Nightly vs stable

Profiling features land in nightly before stable, so this skill targets the nightly

max-serve

build. Install it with pixi:

bash

pixi init quickstart -c https://conda.modular.com/max-nightly/ -c conda-forge
cd quickstart && pixi add max-serve
pixi run max --version          # expect a *.dev build

The utilization API (

GPUDiagContext

BackgroundRecorder

) ships in the conda

max

package, so install via pixi or conda rather than a plain

pip install max

wheel, which may not expose

max.profiler.gpu

. If a command below 404s or an import fails, confirm you're on a recent nightly build.

Install notes

MAX itself: a project with the nightly
```
max-serve
```
package installed via pixi (see Nightly vs stable above). All
```
max
```
CLI commands below assume you can run
```
pixi run max ...
```
in that project. The
```
max-serve
```
package includes extra dependencies required to serve a model with MAX, which aren't included with the main
```
max
```
package.
```
nsys
```
and
```
ncu
```
ship with the CUDA Toolkit. If
```
which nsys
```
fails but CUDA is present,
```
export PATH=/usr/local/cuda/bin:$PATH
```
. Otherwise install Nsight Systems / Nsight Compute from NVIDIA, or
```
sudo apt-get install nsight-systems
```
.
```
rocprofv3
```
ships with ROCm (
```
/opt/rocm/bin
```
). Add it to PATH if needed.
The utilization API needs no extra tooling beyond
```
max
```
— it talks to NVML / ROCm SMI directly through
```
max.profiler.gpu
```
.

Fast paths

The lowest-friction commands, each expanded in its reference file:

"Is my GPU being used?" → run
```
scripts/gpu_snapshot.py
```
alongside a short benchmark. Pure Python, works on NVIDIA and AMD.
"Give me a kernel breakdown, fast." →
```
pixi run max generate --model <m> --prompt "hello" --num-warmups 1 --profile
```
. Prints a ranked top-N GPU-kernel table and writes a
```
.nsys-rep
```
(NVIDIA). Falls back to a CPU summary without a GPU.
"Profile my serving benchmark." → the serving-endpoint flow with
```
MODULAR_ENABLE_PROFILING=detailed
```
+
```
nsys launch
```
/
```
rocprofv3
```
.

profile-model

NPX Install

Tags

SKILL.md Content