Map GPU kernels to PyTorch operators
Profiling a PyTorch model with counters alone gives kernel names, which say
little about the model. Torch tracing annotates the run so each kernel can be
traced back to the operator that launched it.
Torch and Triton tracing are experimental. Every command needs
, and option names can change between releases, so run
rocprof-compute profile --experimental --help
first.
Before you start
If
command -v rocprof-compute
finds the tool and
rocprof-compute analyze --debug-verify-deps
passes, skip this section.
On ROCm 10.2, use
instead of
.
Otherwise, check that ROCm and a GPU are present:
If either command fails, ROCm is not installed or no GPU is visible. Stop and
point the user to "Install the ROCm Core SDK" in the
install guide. Installing rocprof-compute
alone does not help.
Check the GPU against
compatible accelerators.
The table lists product families, so match the
that
prints, and use the
name to confirm. If the GPU is not supported, stop and
tell the user.
Then follow "Install the analyze mode dependencies" in the
install guide, which installs
rocprof-compute in a virtual environment of its own. Use the nightly index so the
version matches these skills. Activate that environment with
source <venv>/bin/activate
before you run rocprof-compute, and confirm with
rocprof-compute analyze --debug-verify-deps
. If the application is a Python
script, note the full path of its Python before you activate, and run the
script with that path.
1. Check the requirements
Torch tracing needs PyTorch installed against a matching ROCm. PyTorch 2.13
and 2.14 use the prebuilt native collector, which records operators on every
thread, including autograd workers. Other versions, or a missing or unusable
collector, warn and use
with reduced coverage on the Python
thread where it is active. Confirm the version and check the loader message
before interpreting missing backward operators as an absence of GPU work.
The workload must be a Python command or Python script; the tracer cannot
annotate a compiled binary. The collector is a generic prebuilt
and needs no PyTorch or Python development packages
to build. The workload still needs Python-compatible
bindings. The
full requirement list is in the Torch trace section of
profile mode.
2. Collect
bash
rocprof-compute profile \
--experimental \
--trace-ml-api torch \
--speed-of-light \
--name <workload_name> \
-- python <script.py> <args>
On ROCm 10.2 and earlier, use
instead of
.
For Triton kernels launched through Triton's Python runtime (
):
bash
rocprof-compute profile --experimental --trace-ml-api triton --name <name> -- python <script.py>
/ Inductor fused kernels use PyTorch's static launcher, so they
do not get
ranges. Attribute them with
.
The two combine as
--trace-ml-api torch triton
. User-defined ROCTx regions
are included with every
run. A bare
traces
all supported ML frameworks as well.
On ROCm 10.2 and earlier, use
,
and
instead of
,
and
a bare
, and use
,
,
and
instead of
,
and
.
Tracing adds overhead on top of the usual counter replay. Profile a few
iterations of a steady-state model rather than a full training run, and skip
warmup.
ROCPROFCOMPUTE_ROCTX_DEEP_TENSOR_WRAPS=0
reduces instrumentation
overhead by dropping the device and layout wraps; see
profile mode for what that costs you.
Tracing cannot be combined with attaching to a running process, and cannot be
combined with a PC-sampling-only profile.
Profile preserves each pass's raw marker and counter CSVs with the
prefix.
Collect only what the question needs
Never run a full profile. It replays the application once per counter pass and
also runs the roofline benchmarks.
- List what this GPU can collect with
rocprof-compute --list-metrics
, and a
block's metrics with rocprof-compute --list-metrics <alias>
. On ROCm 10.2
and earlier, use and --list-metrics <gfx_arch>
.
Pass only the block aliases, block ids, or metric ids (for example
) the question needs. Aliases differ by architecture.
- collects a predefined topic in one pass.
rocprof-compute profile --list-sets
lists the sets.
- Add when the user names the kernels that matter.
- Unless the question is about roofline, select blocks with and leave out
block 4 (), so the roofline benchmark is skipped.
- spreads counters over repeated dispatches of each
kernel with the same launch parameters, so it only helps when each profiled
kernel runs many times with the same launch parameters. Pair it with .
See Iteration multiplexing in
profile mode.
- Tell the user which analysis panels the narrower profile leaves empty.
3. List the captured operators
List when you need the operator names from this run; filtering does not
require a prior list. Do not pass
and
on the same command; listing ignores the filter. These flags require
(the default):
bash
rocprof-compute analyze \
--path ./workloads/<name>/<gpu_model> \
--experimental --list-operators torch
This prints the operators as a call tree sorted by GPU duration, with
on each node that recorded a source location, plus kernel
launch statistics, so it also shows which operators are worth opening.
The Triton equivalent is
.
lists each user ROCTx region with everything that
ran inside it.
4. Filter to an operator
bash
rocprof-compute analyze \
--path ./workloads/<name>/<gpu_model> \
--experimental --operator 'torch:*relu*'
Patterns are shell-style globs, not regexes, matched against a
-separated
operator hierarchy.
contains relu,
is an addmm leaf
on the reconstructed path. Several patterns can be given. A pattern without
a prefix uses the prefix before it. Bare
or
matches
everything.
Quote patterns so the shell does not expand them.
--operator 'triton:*rmsnorm_kernel*'
works the same way.
--operator 'user:*UserDefined*'
selects matching user ROCTx regions with
everything inside them. Other patterns in the same command add operators.
They never remove anything from a selected region.
Worked examples are in the PyTorch operator analysis section of
the analyze CLI guide.
5. Read the result
Attribution tells you which operator owns the kernel time. It does not tell you
why that kernel is slow. Once an operator is identified, analyze its kernels
like any other: Speed-of-Light first, then the memory or roofline path.
An operator that is slow because it launches thousands of tiny kernels is a
different problem from one slow kernel, and the launch statistics in the
operator list are what distinguish the two.
6. ROCTx scope
rocprof-compute uses ROCTx internally: torch tracing injects ROCTx regions to
annotate operators, which is what makes attribution possible.
rocprof-compute does not start, stop, or resume counter collection at a user-authored
ROCTx range. User-defined ROCTx regions are included with every
run. List or filter them at analyze time with
and
. That keeps the region
and nested GPU work.
To keep only some kernels or dispatches from an already collected profile,
use
or
.