Profile an AMD GPU application and find the kernel bottleneck
Start here for any "profile this" or "why is this kernel slow" request. This
skill collects counters, narrows a workload down to one kernel, and identifies
what limits it. Hand off to a focused skill once the limit is clear.
Run
rocprof-compute profile --help
and
rocprof-compute analyze --help
before choosing flags. Options change between releases; do not guess one.
Two sources of performance data
rocprof-compute collects from two independent sources, and they answer
different questions. Choose by what the user is asking, not by difficulty.
| Source | Answers | Skill |
|---|
| Perfmon counters | How the architecture behaves: bandwidth, cache, occupancy, pipeline utilization | this skill, then rocprof-compute-speed-of-light
, , |
| PC sampling | How the source code behaves: which instruction or line is hot and why it stalls | rocprof-compute-pc-sampling
|
Counters are the default starting point because they cover the whole kernel
cheaply. Go straight to
rocprof-compute-pc-sampling
when the user asks about instructions,
source lines, or stall reasons, and use both when an architectural limit needs
to be traced back to the code that causes it.
Before you start
If
command -v rocprof-compute
finds the tool and
rocprof-compute analyze --debug-verify-deps
passes, skip this section.
On ROCm 10.2, use
instead of
.
Otherwise, check that ROCm and a GPU are present:
If either command fails, ROCm is not installed or no GPU is visible. Stop and
point the user to "Install the ROCm Core SDK" in the
install guide. Installing rocprof-compute
alone does not help.
Check the GPU against
compatible accelerators.
The table lists product families, so match the
that
prints, and use the
name to confirm. If the GPU is not supported, stop and
tell the user.
Then follow "Install the analyze mode dependencies" in the
install guide, which installs
rocprof-compute in a virtual environment of its own. Use the nightly index so the
version matches these skills. Activate that environment with
source <venv>/bin/activate
before you run rocprof-compute, and confirm with
rocprof-compute analyze --debug-verify-deps
. If the application is a Python
script, note the full path of its Python before you activate, and run the
script with that path.
1. Check the environment
It reports what the profiler detected, including
,
,
and
. Not every feature works on every GPU. The docs for each feature
say which GPUs support it. When rocprof-compute reports that a feature is not
supported, tell the user, and do not work around it.
2. Collect what you need before profiling
Ask once:
- Launch command and arguments
- A short workload name, used as
- Which kernels matter, if not all
Profiling replays the application once per counter pass, so the command must be
safe to run repeatedly and short enough to run several times. Ask for a reduced
iteration count or problem size rather than profiling a long run. Do not invent
a timeout; profile mode has no such option.
3. Profile
Start with Speed-of-Light for the kernels that matter:
bash
rocprof-compute profile --name <workload_name> --speed-of-light -k <kernel_name> -- <application> <args>
On ROCm 10.2 and earlier, use
instead of
.
Drop
when the user has not named kernels. Once Speed-of-Light points at a
limit, profile again with only the blocks section 5 opens, such as
,
, or
.
Output lands in
./workloads/<workload_name>/<gpu_model>/
, or
under
MPI. Change it with
, which replaces
for building
the path. Re-profiling into a non-empty directory needs
; prefer a
fresh directory.
To restrict which GPU the application uses, set
. Under
Slurm, derive it from
, not from
; the
indices differ.
See
profile mode for output layout,
filtering, and the full option list. To profile a process that is already
running, see
live attach and detach.
Collect only what the question needs
Never run a full profile. It replays the application once per counter pass and
also runs the roofline benchmarks.
- List what this GPU can collect with
rocprof-compute --list-metrics
, and a
block's metrics with rocprof-compute --list-metrics <alias>
. On ROCm 10.2
and earlier, use and --list-metrics <gfx_arch>
.
Pass only the block aliases, block ids, or metric ids (for example
) the question needs. Aliases differ by architecture.
- collects a predefined topic in one pass.
rocprof-compute profile --list-sets
lists the sets.
- Add when the user names the kernels that matter.
- Unless the question is about roofline, select blocks with and leave out
block 4 (), so the roofline benchmark is skipped.
- spreads counters over repeated dispatches of each
kernel with the same launch parameters, so it only helps when each profiled
kernel runs many times with the same launch parameters. Pair it with .
See Iteration multiplexing in
profile mode.
- Tell the user which analysis panels the narrower profile leaves empty.
4. Find the hot kernel
bash
rocprof-compute analyze --path ./workloads/<name>/<gpu_model> --list-stats
must point at the directory holding
. Analyze
does not search for it in nearby directories. If the path you have does not
contain that file, look one level down for the GPU model or rank directory.
prints
Detected Kernels (sorted descending by duration)
and a
. The kernel index is what
takes. Dispatch ids are 1-based
and go to
.
Work on one kernel at a time:
bash
rocprof-compute analyze --path ./workloads/<name>/<gpu_model> -k 0
at profile time is different: it filters by kernel name and is used as a
regex, so
matches every kernel whose name contains
.
5. Narrow to a reason
Get the overview, then open exactly one detailed block:
bash
rocprof-compute analyze --path ./workloads/<name>/<gpu_model> -k 0 --speed-of-light
Read the
column. The report prints the peak for each metric,
so compare against that rather than against a remembered threshold, and never
compute a peak yourself.
Resolve block names with aliases, which are stable across releases, instead of
numeric ids, which are architecture-specific:
bash
rocprof-compute analyze --path ./workloads/<name>/<gpu_model> --list-metrics
rocprof-compute analyze --path ./workloads/<name>/<gpu_model> --list-metrics sol
Where to go next:
| Symptom in Speed-of-Light | Next step |
|---|
| Memory throughput near peak, low compute | the skill |
| Compute near peak, or unclear compute vs memory | the skill |
| Everything far below peak | for occupancy, then for launch limits, both below |
| Question is about instructions, source lines, or stall reasons | the rocprof-compute-pc-sampling
skill |
| Kernels come from PyTorch | the rocprof-compute-torch-trace
skill |
Occupancy and wavefront limits
bash
rocprof-compute analyze --path ./workloads/<name>/<gpu_model> -k 0 -b wavefront
Compare active waves per CU against the theoretical occupancy in the same
table. When theoretical occupancy is itself low, the limit is VGPR, SGPR, or
LDS allocation. Scratch traffic points at register spilling. Low occupancy is
not a defect when the kernel already saturates its limiting resource. Worked
examples are in
occupancy-limiters-example.rst.
Scheduler and pipeline limits
bash
# Workgroup launch rate and resource allocation limits
rocprof-compute analyze --path ./workloads/<name>/<gpu_model> -k 0 -b spi
# Pipeline utilization and instruction mix
rocprof-compute analyze --path ./workloads/<name>/<gpu_model> -k 0 -b cu_pipe
There is no single stall panel. Stall counters sit in the block that owns the
hardware: LDS stalls in
, cache stalls in
and
, fabric stalls
in
. Open the block the Speed-of-Light result points at, and
read what each counter means in the
performance model. Ground every
recommendation in a number you read from the report.
6. Normalization
/
applies to every metric in the report at once:
(default),
,
,
. Keep the default
for a mixed report. Change it only when the user asks about one metric family
and the unit still makes sense for it. Bandwidth per cycle does not.
7. Compare two runs
Pass
twice to put two workloads side by side. Use this for:
- Measuring a software optimization against its baseline on the same GPU.
- Checking that the same code performs consistently across nodes with the
same GPU architecture.
- Comparing the same code across different GPU architectures.
bash
rocprof-compute analyze \
--path ./workloads/baseline/<gpu_model> \
--path ./workloads/optimized/<gpu_model>
Cases 1 and 2 expect the numbers to match except where the change intended
otherwise, so an unexplained difference is the finding. Case 3 compares
different hardware, so compare percent of peak rather than absolute values.
Never present a comparison across different GPUs, partition modes, or clock
settings without stating that those differ.
8. Save a report
Terminal output is the default. Only choose a format when the user asks to save
or export.
takes
,
, or
, each of which
disables terminal output, and
sets the file name. See
analysis output format.
9. When something looks wrong
| Symptom | Likely cause |
|---|
| No workload data | is not the directory with |
| Empty or missing panels | those counters were not collected; re-profile |
| All metrics zero | kernel too short, or no dispatches were captured |
| No kernel ids | the application launched no GPU kernels |
| Roofline missing | not collected, or unsupported on this architecture |
Stop and report permission, firmware, and ROCm version failures instead of
working around them. More in the
FAQ.