rocprof-compute-torch-trace
Compare original and translation side by side
🇺🇸
Original
English🇨🇳
Translation
ChineseMap GPU kernels to PyTorch operators
将 GPU 内核映射到 PyTorch 算子
Profiling a PyTorch model with counters alone gives kernel names, which say
little about the model. Torch tracing annotates the run so each kernel can be
traced back to the operator that launched it.
Torch and Triton tracing are experimental. Every command needs
, and option names can change between releases, so run
first.
--experimentalrocprof-compute profile --experimental --help仅使用计数器分析 PyTorch 模型时,得到的只是内核名称,这些信息几乎无法说明模型本身的情况。Torch tracing 会为运行过程添加注释,使每个内核都能追溯到启动它的算子。
Torch 和 Triton tracing 仍处于实验阶段。每个命令都需要
,且选项名称可能随版本发布而变化,因此请先运行
。
--experimentalrocprof-compute profile --experimental --helpBefore you start
准备开始
If finds the tool and
passes, skip this section.
On ROCm 10.2, use instead of .
command -v rocprof-computerocprof-compute analyze --debug-verify-deps--verify-deps--debug-verify-depsOtherwise, check that ROCm and a GPU are present:
bash
rocminfo
amd-smi listIf either command fails, ROCm is not installed or no GPU is visible. Stop and
point the user to "Install the ROCm Core SDK" in the
install guide. Installing rocprof-compute
alone does not help.
Check the GPU against
compatible accelerators.
The table lists product families, so match the that
prints, and use the name to confirm. If the GPU is not supported, stop and
tell the user.
Marketing NamerocminfogfxThen follow "Install the analyze mode dependencies" in the
install guide, which installs
rocprof-compute in a virtual environment of its own. Use the nightly index so the
version matches these skills. Activate that environment with
before you run rocprof-compute, and confirm with
. If the application is a Python
script, note the full path of its Python before you activate, and run the
script with that path.
source <venv>/bin/activaterocprof-compute analyze --debug-verify-deps如果 能找到该工具,并且
检查通过,则可跳过本节。
在 ROCm 10.2 上,使用 代替 。
command -v rocprof-computerocprof-compute analyze --debug-verify-deps--verify-deps--debug-verify-deps否则,检查 ROCm 和 GPU 是否存在:
bash
rocminfo
amd-smi list如果任一命令失败,说明未安装 ROCm 或看不到 GPU。请停止操作,并指引用户查看
安装指南中的“Install the ROCm Core SDK”。仅安装
rocprof-compute 无济于事。
然后按照安装指南中的“Install the analyze mode dependencies”操作,该步骤会将
rocprof-compute 安装在其独立的虚拟环境中。使用 nightly index,以确保版本与这些 skill 匹配。运行 rocprof-compute 前,使用 激活该环境,并通过 确认。如果应用是 Python 脚本,请先记下 Python 的完整路径再激活环境,并使用该路径运行脚本。
source <venv>/bin/activaterocprof-compute analyze --debug-verify-deps1. Check the requirements
1. 检查要求
Torch tracing needs PyTorch installed against a matching ROCm. PyTorch 2.13
and 2.14 use the prebuilt native collector, which records operators on every
thread, including autograd workers. Other versions, or a missing or unusable
collector, warn and use with reduced coverage on the Python
thread where it is active. Confirm the version and check the loader message
before interpreting missing backward operators as an absence of GPU work.
TorchDispatchModeThe workload must be a Python command or Python script; the tracer cannot
annotate a compiled binary. The collector is a generic prebuilt
and needs no PyTorch or Python development packages
to build. The workload still needs Python-compatible bindings. The
full requirement list is in the Torch trace section of
profile mode.
torch_trace_collector.soroctxTorch tracing 要求安装与 ROCm 匹配的 PyTorch。PyTorch 2.13 和 2.14 使用预构建的原生 collector,它会记录每个线程上的算子,包括 autograd worker。对于其他版本,或者 collector 缺失或不可用的情况,工具会发出警告并改用 ;其覆盖范围会缩小,仅限其所在的 Python 线程。在将缺失的 backward 算子解读为不存在 GPU 工作之前,请先确认版本并检查 loader 消息。
TorchDispatchMode工作负载必须是 Python 命令或 Python 脚本;跟踪器无法为已编译的二进制文件添加注释。collector 是通用的预构建 ,构建时不需要 PyTorch 或 Python 开发包。不过,工作负载仍需具备兼容 Python 的 bindings。完整要求列表见 profile mode文档中的 Torch trace 部分。
torch_trace_collector.soroctx2. Collect
2. 采集
bash
rocprof-compute profile \
--experimental \
--trace-ml-api torch \
--speed-of-light \
--name <workload_name> \
-- python <script.py> <args>On ROCm 10.2 and earlier, use instead of .
-b sol--speed-of-lightFor Triton kernels launched through Triton's Python runtime ():
@triton.jitbash
rocprof-compute profile --experimental --trace-ml-api triton --name <name> -- python <script.py>torch.compile|triton--trace-ml-api torchThe two combine as . User-defined ROCTx regions
are included with every run. A bare traces
all supported ML frameworks as well.
--trace-ml-api torch triton--trace-ml-api--trace-ml-apiOn ROCm 10.2 and earlier, use , and
instead of , and
a bare , and use ,
, and instead of
, and .
--torch-trace--triton-trace--ml-api-trace--trace-ml-api torch--trace-ml-api triton--trace-ml-api--list-torch-operators--list-triton-operators--torch-operator--triton-operator--list-operators torch--list-operators triton--operatorTracing adds overhead on top of the usual counter replay. Profile a few
iterations of a steady-state model rather than a full training run, and skip
warmup. reduces instrumentation
overhead by dropping the device and layout wraps; see
profile mode for what that costs you.
ROCPROFCOMPUTE_ROCTX_DEEP_TENSOR_WRAPS=0Tracing cannot be combined with attaching to a running process, and cannot be
combined with a PC-sampling-only profile.
Profile preserves each pass's raw marker and counter CSVs with the
prefix.
ml_api_tracebash
rocprof-compute profile \
--experimental \
--trace-ml-api torch \
--speed-of-light \
--name <workload_name> \
-- python <script.py> <args>在 ROCm 10.2 及更早版本上,使用 代替 。
-b sol--speed-of-light对于通过 Triton Python runtime()启动的 Triton 内核:
@triton.jitbash
rocprof-compute profile --experimental --trace-ml-api triton --name <name> -- python <script.py>torch.compile|triton--trace-ml-api torch两者可组合使用 。用户定义的 ROCTx 区域会包含在每次 运行中。裸 会跟踪所有受支持的 ML 框架。
--trace-ml-api torch triton--trace-ml-api--trace-ml-api在 ROCm 10.2 及更早版本上,使用 、 和 代替 、 和裸 ;使用 、、 和 代替 、 和 。
--torch-trace--triton-trace--ml-api-trace--trace-ml-api torch--trace-ml-api triton--trace-ml-api--list-torch-operators--list-triton-operators--torch-operator--triton-operator--list-operators torch--list-operators triton--operator跟踪会在常规 counter 重放之外增加开销。请对稳态模型的少量迭代进行性能分析,而不是运行完整的训练,并跳过预热阶段。 可通过丢弃 device 和 layout wraps 来减少 instrumentation 开销;有关相应代价,请参阅 profile mode。
ROCPROFCOMPUTE_ROCTX_DEEP_TENSOR_WRAPS=0跟踪不能与附加到正在运行的进程结合使用,也不能与仅进行 PC 采样的 profile 结合使用。
Profile 会保留每个 pass 的原始 marker 和 counter CSV,并使用 前缀。
ml_api_traceCollect only what the question needs
仅采集问题所需的内容
Never run a full profile. It replays the application once per counter pass and
also runs the roofline benchmarks.
- List what this GPU can collect with , and a block's metrics with
rocprof-compute --list-metrics. On ROCm 10.2 and earlier, userocprof-compute --list-metrics <alias>and--list-blocks <gfx_arch>. Pass--list-metrics <gfx_arch>only the block aliases, block ids, or metric ids (for example-b) the question needs. Aliases differ by architecture.2.1.0 - collects a predefined topic in one pass.
--set <set_name>lists the sets.rocprof-compute profile --list-sets - Add when the user names the kernels that matter.
-k <kernel_name> - Unless the question is about roofline, select blocks with and leave out block 4 (
-b), so the roofline benchmark is skipped.roof - spreads counters over repeated dispatches of each kernel with the same launch parameters, so it only helps when each profiled kernel runs many times with the same launch parameters. Pair it with
--iteration-multiplexing. See Iteration multiplexing in profile mode.-k - Tell the user which analysis panels the narrower profile leaves empty.
切勿运行完整的 profile。它会针对每个 counter pass 重放一次应用程序,并且还会运行 roofline benchmark。
- 使用 列出此 GPU 可采集的指标,使用
rocprof-compute --list-metrics列出某个 block 的指标。在 ROCm 10.2 及更早版本上,使用rocprof-compute --list-metrics <alias>和--list-blocks <gfx_arch>。仅通过--list-metrics <gfx_arch>传入问题所需的 block aliases、block ids 或 metric ids(例如-b)。Aliases 因架构而异。2.1.0 - 可在一次 pass 中采集预定义的主题。
--set <set_name>会列出可用的 sets。rocprof-compute profile --list-sets - 当用户指定需要关注的内核时,添加 。
-k <kernel_name> - 除非问题涉及 roofline,否则应使用 选择 blocks,并排除 block 4(
-b),从而跳过 roofline benchmark。roof - 会将 counters 分散到每个内核使用相同启动参数的重复 dispatch 中,因此仅当每个受分析的内核都以相同启动参数运行多次时才有帮助。请与
--iteration-multiplexing搭配使用。参见 profile mode中的 Iteration multiplexing。-k - 告知用户精简后的 profile 会使哪些分析面板留空。
3. List the captured operators
3. 列出捕获的算子
List when you need the operator names from this run; filtering does not
require a prior list. Do not pass and
on the same command; listing ignores the filter. These flags require
(the default):
--list-operators--operator--output-format stdoutbash
rocprof-compute analyze \
--path ./workloads/<name>/<gpu_model> \
--experimental --list-operators torchThis prints the operators as a call tree sorted by GPU duration, with
on each node that recorded a source location, plus kernel
launch statistics, so it also shows which operators are worth opening.
file:lineThe Triton equivalent is .
--list-operators triton--list-operators user在需要获取本次运行捕获的算子名称时执行此操作;筛选并不要求事先列出。不要在同一条命令中同时传入 和 ;执行列出时过滤器会被忽略。这些标志要求使用 (默认值):
--list-operators--operator--output-format stdoutbash
rocprof-compute analyze \
--path ./workloads/<name>/<gpu_model> \
--experimental --list-operators torch该命令会将算子作为调用树输出,并按 GPU 耗时排序。每个记录了源码位置的节点都会显示 ,同时还会输出内核启动统计信息,因此也能看出哪些算子值得进一步打开分析。
file:lineTriton 的等效命令为 。
--list-operators triton--list-operators user4. Filter to an operator
4. 筛选到算子
bash
rocprof-compute analyze \
--path ./workloads/<name>/<gpu_model> \
--experimental --operator 'torch:*relu*'Patterns are shell-style globs, not regexes, matched against a -separated
operator hierarchy. contains relu, is an addmm leaf
on the reconstructed path. Several patterns can be given. A pattern without
a prefix uses the prefix before it. Bare or matches
everything.
Quote patterns so the shell does not expand them.
/*relu**/aten::addmm--operator**--operator 'triton:*rmsnorm_kernel*'--operator 'user:*UserDefined*'Worked examples are in the PyTorch operator analysis section of
the analyze CLI guide.
bash
rocprof-compute analyze \
--path ./workloads/<name>/<gpu_model> \
--experimental --operator 'torch:*relu*'模式采用 shell 风格的 glob,而不是 regex,并与以 分隔的算子层级进行匹配。 包含 relu; 表示重建路径上的 addmm 叶节点。可以提供多个模式。没有前缀的模式将使用其前面的前缀。裸 或 可匹配全部内容。请为模式添加引号,避免 shell 对其进行展开。
/*relu**/aten::addmm--operator**--operator 'triton:*rmsnorm_kernel*'--operator 'user:*UserDefined*'实际示例见 analyze CLI 指南中的 PyTorch operator analysis 部分。
5. Read the result
5. 解读结果
Attribution tells you which operator owns the kernel time. It does not tell you
why that kernel is slow. Once an operator is identified, analyze its kernels
like any other: Speed-of-Light first, then the memory or roofline path.
An operator that is slow because it launches thousands of tiny kernels is a
different problem from one slow kernel, and the launch statistics in the
operator list are what distinguish the two.
归因结果可告诉你哪个算子占用了内核时间,但不会说明该内核为何较慢。确定算子后,应像分析其他内核一样分析其内核:先查看 Speed-of-Light,再查看 memory 或 roofline 路径。
因启动数千个微小内核而变慢的算子,与仅有一个内核较慢属于不同问题;算子列表中的启动统计信息可用于区分两者。
6. ROCTx scope
6. ROCTx 作用域
rocprof-compute uses ROCTx internally: torch tracing injects ROCTx regions to
annotate operators, which is what makes attribution possible.
rocprof-compute does not start, stop, or resume counter collection at a user-authored
ROCTx range. User-defined ROCTx regions are included with every
run. List or filter them at analyze time with
and . That keeps the region
and nested GPU work.
To keep only some kernels or dispatches from an already collected profile,
use or .
--trace-ml-api--list-operators user--operator user:PATTERN-k-drocprof-compute 内部使用 ROCTx:Torch tracing 会注入 ROCTx 区域来标注算子,这正是实现归因的关键。
rocprof-compute 不会在用户编写的 ROCTx 范围内启动、停止或恢复 counter 采集。每次 运行都会包含用户定义的 ROCTx 区域。分析时,可以使用 和 列出或筛选这些区域,从而保留该区域及其嵌套的 GPU 工作。
--trace-ml-api--list-operators user--operator user:PATTERN若要从已采集的 profile 中仅保留部分内核或 dispatch,请使用 或 。
-k-d