Roofline analysis
A roofline places a kernel's arithmetic intensity against two measured
ceilings, peak compute and peak bandwidth, and answers one question: which of
the two is the limit.
Run
rocprof-compute profile --help
and
rocprof-compute analyze --help
before choosing flags. Do not open the generated HTML chart. The analyze table
carries the same numbers in readable form.
Before you start
If
command -v rocprof-compute
finds the tool and
rocprof-compute analyze --debug-verify-deps
passes, skip this section.
On ROCm 10.2, use
instead of
.
Otherwise, check that ROCm and a GPU are present:
If either command fails, ROCm is not installed or no GPU is visible. Stop and
point the user to "Install the ROCm Core SDK" in the
install guide. Installing rocprof-compute
alone does not help.
Check the GPU against
compatible accelerators.
The table lists product families, so match the
that
prints, and use the
name to confirm. If the GPU is not supported, stop and
tell the user.
Then follow "Install the analyze mode dependencies" in the
install guide, which installs
rocprof-compute in a virtual environment of its own. Use the nightly index so the
version matches these skills. Activate that environment with
source <venv>/bin/activate
before you run rocprof-compute, and confirm with
rocprof-compute analyze --debug-verify-deps
. If the application is a Python
script, note the full path of its Python before you activate, and run the
script with that path.
1. Collect
Collect only the roofline block:
bash
rocprof-compute profile --name <workload_name> --roofline -- <application> <args>
On ROCm 10.2 and earlier, use
instead of
.
Collecting it runs microbenchmarks in addition to profiling the application,
which takes extra time. They are skipped when
or
selects blocks
that leave out roofline (block 4), or with the deprecated
, so an
existing profile collected that way has no roofline data.
MI100 (gfx908) does not support the roofline microbenchmarks. Every other
supported architecture does. On MI100 a full profile skips the benchmark and
still succeeds, and selecting roofline fails, so a missing roofline there is not
a mistake in how the profile was run. Never tell an MI100 user to re-profile to get roofline; use
Speed-of-Light and the
skill instead. Confirm support
for a new architecture against
compatible-accelerators.rst.
selects the GPU for the roofline microbenchmarks. It does not
choose which GPU the application runs on; use
for that.
See
standalone roofline for two more options:
- selects the roofline block; used alone, it collects only roofline data.
- runs only the microbenchmarks, without an application.
On ROCm 10.2 and earlier, these options are named
,
and
.
Collect only what the question needs
Never run a full profile. It replays the application once per counter pass and
also runs the roofline benchmarks.
- List what this GPU can collect with
rocprof-compute --list-metrics
, and a
block's metrics with rocprof-compute --list-metrics <alias>
. On ROCm 10.2
and earlier, use and --list-metrics <gfx_arch>
.
Pass only the block aliases, block ids, or metric ids (for example
) the question needs. Aliases differ by architecture.
- collects a predefined topic in one pass.
rocprof-compute profile --list-sets
lists the sets.
- Add when the user names the kernels that matter.
- Unless the question is about roofline, select blocks with and leave out
block 4 (), so the roofline benchmark is skipped.
- spreads counters over repeated dispatches of each
kernel with the same launch parameters, so it only helps when each profiled
kernel runs many times with the same launch parameters. Pair it with .
See Iteration multiplexing in
profile mode.
- Tell the user which analysis panels the narrower profile leaves empty.
2. Read the table
bash
rocprof-compute analyze --path ./workloads/<name>/<gpu_model> --list-stats
rocprof-compute analyze --path ./workloads/<name>/<gpu_model> -k <kernel_id> --roofline
Always pass one kernel id. The table gives the kernel's arithmetic intensity in
FLOPs per byte alongside the empirical peak FLOPs and peak bandwidth measured
on this machine. Compare against those printed numbers; never compute a peak
yourself.
Report two things first, before any tuning advice:
- Which ceiling limits the kernel, memory or compute.
- How far the kernel sits below that ceiling, as a percentage of it.
Read the verdict from the table as:
- Low arithmetic intensity, close to the bandwidth ceiling: memory-bound. Work
on traffic, reuse, tiling, and locality. Continue with the
skill.
- High arithmetic intensity, close to the compute ceiling: compute-bound. Work
on instruction mix, matrix instruction use, and divergence.
- Well below both ceilings: neither is the limit. The kernel is held back by
occupancy, launch configuration, or stalls. Go back to the
rocprof-compute-kernel-bottleneck
skill.
A kernel sitting below both ceilings is the common case and the most
misread one. Never report it as compute-bound merely because its arithmetic
intensity is high.
The distance to the ceiling is what makes the verdict actionable. "Memory-bound
at 45% of peak bandwidth" tells the user there is headroom; "memory-bound at
94%" tells them to reduce traffic instead of chasing it.
3. Pick the right precision and memory level
The table lists a peak for every precision and an arithmetic intensity for
every memory level. Compare against the peak row for the precision the kernel
actually issues, and the intensity row for the memory level in question.
A mixed-precision or matrix kernel compared against the FP32 peak will look far
worse than it is. Read the precision from the instruction mix with
rather than assuming it. Ask the user which precision the kernel is meant to
use only when the mix is genuinely mixed and no single type dominates; they
know the intent, the counters know what ran.
HBM answers "am I limited by main memory", while L2 and vL1D show whether
caches are absorbing the traffic.
To compare many kernels, write the
analysis database with
and query it with SQL.
4. When roofline data is missing
| Cause | What to do |
|---|
| Profiled with , or / without block 4 | re-profile without ; add to , or replace with blocks plus (no set includes block 4) |
| Architecture has no microbenchmark support | use Speed-of-Light and the skill instead |
| absent from the workload directory | the benchmark did not complete; check the profile log |
The concepts behind the ceilings are in the
performance model.