rocprof-compute-torch-trace

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Map GPU kernels to PyTorch operators

将 GPU 内核映射到 PyTorch 算子

Profiling a PyTorch model with counters alone gives kernel names, which say little about the model. Torch tracing annotates the run so each kernel can be traced back to the operator that launched it.
Torch and Triton tracing are experimental. Every command needs
--experimental
, and option names can change between releases, so run
rocprof-compute profile --experimental --help
first.
仅使用计数器分析 PyTorch 模型时,得到的只是内核名称,这些信息几乎无法说明模型本身的情况。Torch tracing 会为运行过程添加注释,使每个内核都能追溯到启动它的算子。
Torch 和 Triton tracing 仍处于实验阶段。每个命令都需要
--experimental
,且选项名称可能随版本发布而变化,因此请先运行
rocprof-compute profile --experimental --help
。

Before you start

准备开始

If
command -v rocprof-compute
finds the tool and
rocprof-compute analyze --debug-verify-deps
passes, skip this section. On ROCm 10.2, use
--verify-deps
instead of
--debug-verify-deps
.
Otherwise, check that ROCm and a GPU are present:
bash
rocminfo
amd-smi list
If either command fails, ROCm is not installed or no GPU is visible. Stop and point the user to "Install the ROCm Core SDK" in the install guide. Installing rocprof-compute alone does not help.
Check the GPU against compatible accelerators. The table lists product families, so match the
Marketing Name
that
rocminfo
prints, and use the
gfx
name to confirm. If the GPU is not supported, stop and tell the user.
Then follow "Install the analyze mode dependencies" in the install guide, which installs rocprof-compute in a virtual environment of its own. Use the nightly index so the version matches these skills. Activate that environment with
source <venv>/bin/activate
before you run rocprof-compute, and confirm with
rocprof-compute analyze --debug-verify-deps
. If the application is a Python script, note the full path of its Python before you activate, and run the script with that path.
如果
command -v rocprof-compute
能找到该工具,并且
rocprof-compute analyze --debug-verify-deps
检查通过,则可跳过本节。 在 ROCm 10.2 上,使用
--verify-deps
代替
--debug-verify-deps
。
否则,检查 ROCm 和 GPU 是否存在:
bash
rocminfo
amd-smi list
如果任一命令失败,说明未安装 ROCm 或看不到 GPU。请停止操作,并指引用户查看 安装指南中的“Install the ROCm Core SDK”。仅安装 rocprof-compute 无济于事。
根据 兼容的加速器检查 GPU。该表列出的是产品系列,因此应匹配
rocminfo
输出的
Marketing Name
,并使用
gfx
名称进行确认。如果 GPU 不受支持,请停止操作并告知用户。
然后按照安装指南中的“Install the analyze mode dependencies”操作,该步骤会将 rocprof-compute 安装在其独立的虚拟环境中。使用 nightly index,以确保版本与这些 skill 匹配。运行 rocprof-compute 前,使用
source <venv>/bin/activate
激活该环境,并通过
rocprof-compute analyze --debug-verify-deps
确认。如果应用是 Python 脚本,请先记下 Python 的完整路径再激活环境,并使用该路径运行脚本。

1. Check the requirements

1. 检查要求

Torch tracing needs PyTorch installed against a matching ROCm. PyTorch 2.13 and 2.14 use the prebuilt native collector, which records operators on every thread, including autograd workers. Other versions, or a missing or unusable collector, warn and use
TorchDispatchMode
with reduced coverage on the Python thread where it is active. Confirm the version and check the loader message before interpreting missing backward operators as an absence of GPU work.
The workload must be a Python command or Python script; the tracer cannot annotate a compiled binary. The collector is a generic prebuilt
torch_trace_collector.so
and needs no PyTorch or Python development packages to build. The workload still needs Python-compatible
roctx
bindings. The full requirement list is in the Torch trace section of profile mode.
Torch tracing 要求安装与 ROCm 匹配的 PyTorch。PyTorch 2.13 和 2.14 使用预构建的原生 collector,它会记录每个线程上的算子,包括 autograd worker。对于其他版本,或者 collector 缺失或不可用的情况,工具会发出警告并改用
TorchDispatchMode
;其覆盖范围会缩小,仅限其所在的 Python 线程。在将缺失的 backward 算子解读为不存在 GPU 工作之前,请先确认版本并检查 loader 消息。
工作负载必须是 Python 命令或 Python 脚本;跟踪器无法为已编译的二进制文件添加注释。collector 是通用的预构建
torch_trace_collector.so
,构建时不需要 PyTorch 或 Python 开发包。不过,工作负载仍需具备兼容 Python 的
roctx
bindings。完整要求列表见 profile mode文档中的 Torch trace 部分。

2. Collect

2. 采集

bash
rocprof-compute profile \
    --experimental \
    --trace-ml-api torch \
    --speed-of-light \
    --name <workload_name> \
    -- python <script.py> <args>
On ROCm 10.2 and earlier, use
-b sol
instead of
--speed-of-light
.
For Triton kernels launched through Triton's Python runtime (
@triton.jit
):
bash
rocprof-compute profile --experimental --trace-ml-api triton --name <name> -- python <script.py>
torch.compile
/ Inductor fused kernels use PyTorch's static launcher, so they do not get
|triton
ranges. Attribute them with
--trace-ml-api torch
.
The two combine as
--trace-ml-api torch triton
. User-defined ROCTx regions are included with every
--trace-ml-api
run. A bare
--trace-ml-api
traces all supported ML frameworks as well.
On ROCm 10.2 and earlier, use
--torch-trace
,
--triton-trace
and
--ml-api-trace
instead of
--trace-ml-api torch
,
--trace-ml-api triton
and a bare
--trace-ml-api
, and use
--list-torch-operators
,
--list-triton-operators
,
--torch-operator
and
--triton-operator
instead of
--list-operators torch
,
--list-operators triton
and
--operator
.
Tracing adds overhead on top of the usual counter replay. Profile a few iterations of a steady-state model rather than a full training run, and skip warmup.
ROCPROFCOMPUTE_ROCTX_DEEP_TENSOR_WRAPS=0
reduces instrumentation overhead by dropping the device and layout wraps; see profile mode for what that costs you.
Tracing cannot be combined with attaching to a running process, and cannot be combined with a PC-sampling-only profile.
Profile preserves each pass's raw marker and counter CSVs with the
ml_api_trace
prefix.
bash
rocprof-compute profile \
    --experimental \
    --trace-ml-api torch \
    --speed-of-light \
    --name <workload_name> \
    -- python <script.py> <args>
在 ROCm 10.2 及更早版本上,使用
-b sol
代替
--speed-of-light
。
对于通过 Triton Python runtime(
@triton.jit
)启动的 Triton 内核:
bash
rocprof-compute profile --experimental --trace-ml-api triton --name <name> -- python <script.py>
torch.compile
/ Inductor 融合内核使用 PyTorch 的 static launcher,因此不会生成
|triton
范围。请使用
--trace-ml-api torch
对其进行归因。
两者可组合使用
--trace-ml-api torch triton
。用户定义的 ROCTx 区域会包含在每次
--trace-ml-api
运行中。裸
--trace-ml-api
会跟踪所有受支持的 ML 框架。
在 ROCm 10.2 及更早版本上,使用
--torch-trace
、
--triton-trace
和
--ml-api-trace
代替
--trace-ml-api torch
、
--trace-ml-api triton
和裸
--trace-ml-api
;使用
--list-torch-operators
、
--list-triton-operators
、
--torch-operator
和
--triton-operator
代替
--list-operators torch
、
--list-operators triton
和
--operator
。
跟踪会在常规 counter 重放之外增加开销。请对稳态模型的少量迭代进行性能分析,而不是运行完整的训练,并跳过预热阶段。
ROCPROFCOMPUTE_ROCTX_DEEP_TENSOR_WRAPS=0
可通过丢弃 device 和 layout wraps 来减少 instrumentation 开销;有关相应代价,请参阅 profile mode。
跟踪不能与附加到正在运行的进程结合使用,也不能与仅进行 PC 采样的 profile 结合使用。
Profile 会保留每个 pass 的原始 marker 和 counter CSV,并使用
ml_api_trace
前缀。

Collect only what the question needs

仅采集问题所需的内容

Never run a full profile. It replays the application once per counter pass and also runs the roofline benchmarks.
  • List what this GPU can collect with
    rocprof-compute --list-metrics
    , and a block's metrics with
    rocprof-compute --list-metrics <alias>
    . On ROCm 10.2 and earlier, use
    --list-blocks <gfx_arch>
    and
    --list-metrics <gfx_arch>
    . Pass
    -b
    only the block aliases, block ids, or metric ids (for example
    2.1.0
    ) the question needs. Aliases differ by architecture.
  • --set <set_name>
    collects a predefined topic in one pass.
    rocprof-compute profile --list-sets
    lists the sets.
  • Add
    -k <kernel_name>
    when the user names the kernels that matter.
  • Unless the question is about roofline, select blocks with
    -b
    and leave out block 4 (
    roof
    ), so the roofline benchmark is skipped.
  • --iteration-multiplexing
    spreads counters over repeated dispatches of each kernel with the same launch parameters, so it only helps when each profiled kernel runs many times with the same launch parameters. Pair it with
    -k
    . See Iteration multiplexing in profile mode.
  • Tell the user which analysis panels the narrower profile leaves empty.
切勿运行完整的 profile。它会针对每个 counter pass 重放一次应用程序,并且还会运行 roofline benchmark。
  • 使用
    rocprof-compute --list-metrics
    列出此 GPU 可采集的指标,使用
    rocprof-compute --list-metrics <alias>
    列出某个 block 的指标。在 ROCm 10.2 及更早版本上,使用
    --list-blocks <gfx_arch>
    和
    --list-metrics <gfx_arch>
    。仅通过
    -b
    传入问题所需的 block aliases、block ids 或 metric ids(例如
    2.1.0
    )。Aliases 因架构而异。
  • --set <set_name>
    可在一次 pass 中采集预定义的主题。
    rocprof-compute profile --list-sets
    会列出可用的 sets。
  • 当用户指定需要关注的内核时,添加
    -k <kernel_name>
    。
  • 除非问题涉及 roofline,否则应使用
    -b
    选择 blocks,并排除 block 4(
    roof
    ),从而跳过 roofline benchmark。
  • --iteration-multiplexing
    会将 counters 分散到每个内核使用相同启动参数的重复 dispatch 中,因此仅当每个受分析的内核都以相同启动参数运行多次时才有帮助。请与
    -k
    搭配使用。参见 profile mode中的 Iteration multiplexing。
  • 告知用户精简后的 profile 会使哪些分析面板留空。

3. List the captured operators

3. 列出捕获的算子

List when you need the operator names from this run; filtering does not require a prior list. Do not pass
--list-operators
and
--operator
on the same command; listing ignores the filter. These flags require
--output-format stdout
(the default):
bash
rocprof-compute analyze \
    --path ./workloads/<name>/<gpu_model> \
    --experimental --list-operators torch
This prints the operators as a call tree sorted by GPU duration, with
file:line
on each node that recorded a source location, plus kernel launch statistics, so it also shows which operators are worth opening.
The Triton equivalent is
--list-operators triton
.
--list-operators user
lists each user ROCTx region with everything that ran inside it.
在需要获取本次运行捕获的算子名称时执行此操作;筛选并不要求事先列出。不要在同一条命令中同时传入
--list-operators
和
--operator
;执行列出时过滤器会被忽略。这些标志要求使用
--output-format stdout
(默认值):
bash
rocprof-compute analyze \
    --path ./workloads/<name>/<gpu_model> \
    --experimental --list-operators torch
该命令会将算子作为调用树输出,并按 GPU 耗时排序。每个记录了源码位置的节点都会显示
file:line
,同时还会输出内核启动统计信息,因此也能看出哪些算子值得进一步打开分析。
Triton 的等效命令为
--list-operators triton
。
--list-operators user
会列出每个用户 ROCTx 区域及其内部运行的所有内容。

4. Filter to an operator

4. 筛选到算子

bash
rocprof-compute analyze \
    --path ./workloads/<name>/<gpu_model> \
    --experimental --operator 'torch:*relu*'
Patterns are shell-style globs, not regexes, matched against a
/
-separated operator hierarchy.
*relu*
contains relu,
*/aten::addmm
is an addmm leaf on the reconstructed path. Several patterns can be given. A pattern without a prefix uses the prefix before it. Bare
--operator
or
**
matches everything. Quote patterns so the shell does not expand them.
--operator 'triton:*rmsnorm_kernel*'
works the same way.
--operator 'user:*UserDefined*'
selects matching user ROCTx regions with everything inside them. Other patterns in the same command add operators. They never remove anything from a selected region.
Worked examples are in the PyTorch operator analysis section of the analyze CLI guide.
bash
rocprof-compute analyze \
    --path ./workloads/<name>/<gpu_model> \
    --experimental --operator 'torch:*relu*'
模式采用 shell 风格的 glob,而不是 regex,并与以
/
分隔的算子层级进行匹配。
*relu*
包含 relu;
*/aten::addmm
表示重建路径上的 addmm 叶节点。可以提供多个模式。没有前缀的模式将使用其前面的前缀。裸
--operator
或
**
可匹配全部内容。请为模式添加引号,避免 shell 对其进行展开。
--operator 'triton:*rmsnorm_kernel*'
的用法相同。
--operator 'user:*UserDefined*'
会选择匹配的用户 ROCTx 区域及其内部的所有内容。同一命令中的其他模式会添加算子,但绝不会从选定区域中移除任何内容。
实际示例见 analyze CLI 指南中的 PyTorch operator analysis 部分。

5. Read the result

5. 解读结果

Attribution tells you which operator owns the kernel time. It does not tell you why that kernel is slow. Once an operator is identified, analyze its kernels like any other: Speed-of-Light first, then the memory or roofline path.
An operator that is slow because it launches thousands of tiny kernels is a different problem from one slow kernel, and the launch statistics in the operator list are what distinguish the two.
归因结果可告诉你哪个算子占用了内核时间,但不会说明该内核为何较慢。确定算子后,应像分析其他内核一样分析其内核:先查看 Speed-of-Light,再查看 memory 或 roofline 路径。
因启动数千个微小内核而变慢的算子,与仅有一个内核较慢属于不同问题;算子列表中的启动统计信息可用于区分两者。

6. ROCTx scope

6. ROCTx 作用域

rocprof-compute uses ROCTx internally: torch tracing injects ROCTx regions to annotate operators, which is what makes attribution possible.
rocprof-compute does not start, stop, or resume counter collection at a user-authored ROCTx range. User-defined ROCTx regions are included with every
--trace-ml-api
run. List or filter them at analyze time with
--list-operators user
and
--operator user:PATTERN
. That keeps the region and nested GPU work. To keep only some kernels or dispatches from an already collected profile, use
-k
or
-d
.
rocprof-compute 内部使用 ROCTx:Torch tracing 会注入 ROCTx 区域来标注算子,这正是实现归因的关键。
rocprof-compute 不会在用户编写的 ROCTx 范围内启动、停止或恢复 counter 采集。每次
--trace-ml-api
运行都会包含用户定义的 ROCTx 区域。分析时,可以使用
--list-operators user
和
--operator user:PATTERN
列出或筛选这些区域,从而保留该区域及其嵌套的 GPU 工作。
若要从已采集的 profile 中仅保留部分内核或 dispatch,请使用
-k
或
-d
。