rocprof-compute-kernel-bottleneck

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Profile an AMD GPU application and find the kernel bottleneck

分析 AMD GPU 应用并找出 kernel 瓶颈

Start here for any "profile this" or "why is this kernel slow" request. This skill collects counters, narrows a workload down to one kernel, and identifies what limits it. Hand off to a focused skill once the limit is clear.
Run
rocprof-compute profile --help
and
rocprof-compute analyze --help
before choosing flags. Options change between releases; do not guess one.
对于任何“分析这个”或“这个 kernel 为什么慢”的请求,都从这里开始。该 skill 会收集 counters,将 workload 缩小到单个 kernel,并确定 限制它的因素。确定限制后,再移交给专门的 skill。
在选择 flags 之前,运行
rocprof-compute profile --help
和
rocprof-compute analyze --help
。不同 release 之间 options 会发生变化; 不要凭空猜测。

Two sources of performance data

两种性能数据来源

rocprof-compute collects from two independent sources, and they answer different questions. Choose by what the user is asking, not by difficulty.
SourceAnswersSkill
Perfmon countersHow the architecture behaves: bandwidth, cache, occupancy, pipeline utilizationthis skill, then
rocprof-compute-speed-of-light
,
rocprof-compute-memory
,
rocprof-compute-roofline
PC samplingHow the source code behaves: which instruction or line is hot and why it stalls
rocprof-compute-pc-sampling
Counters are the default starting point because they cover the whole kernel cheaply. Go straight to
rocprof-compute-pc-sampling
when the user asks about instructions, source lines, or stall reasons, and use both when an architectural limit needs to be traced back to the code that causes it.
rocprof-compute 从两个独立来源收集数据,而它们回答的是不同的问题。 应根据用户的问题选择来源,而不是根据操作难度选择。
SourceAnswersSkill
Perfmon counters架构如何运行:bandwidth、cache、occupancy、pipeline utilization此 skill,然后使用
rocprof-compute-speed-of-light
、
rocprof-compute-memory
、
rocprof-compute-roofline
PC sampling源代码如何运行:哪条 instruction 或哪一行是热点,以及它为何 stall
rocprof-compute-pc-sampling
默认从 counters 开始,因为它们可以低成本地覆盖整个 kernel。 当用户询问 instructions、source lines 或 stall reasons 时,直接使用
rocprof-compute-pc-sampling
;当需要将某个架构限制追溯到导致该限制的 代码时,则结合使用两种方式。

Before you start

开始之前

If
command -v rocprof-compute
finds the tool and
rocprof-compute analyze --debug-verify-deps
passes, skip this section. On ROCm 10.2, use
--verify-deps
instead of
--debug-verify-deps
.
Otherwise, check that ROCm and a GPU are present:
bash
rocminfo
amd-smi list
If either command fails, ROCm is not installed or no GPU is visible. Stop and point the user to "Install the ROCm Core SDK" in the install guide. Installing rocprof-compute alone does not help.
Check the GPU against compatible accelerators. The table lists product families, so match the
Marketing Name
that
rocminfo
prints, and use the
gfx
name to confirm. If the GPU is not supported, stop and tell the user.
Then follow "Install the analyze mode dependencies" in the install guide, which installs rocprof-compute in a virtual environment of its own. Use the nightly index so the version matches these skills. Activate that environment with
source <venv>/bin/activate
before you run rocprof-compute, and confirm with
rocprof-compute analyze --debug-verify-deps
. If the application is a Python script, note the full path of its Python before you activate, and run the script with that path.
如果
command -v rocprof-compute
找到了该工具,并且
rocprof-compute analyze --debug-verify-deps
检查通过,则跳过本节。 在 ROCm 10.2 上,使用
--verify-deps
代替
--debug-verify-deps
。
否则,检查 ROCm 和 GPU 是否存在:
bash
rocminfo
amd-smi list
如果任一命令失败,说明未安装 ROCm 或没有检测到 GPU。停止操作,并将用户引导至 安装指南 中的“Install the ROCm Core SDK”。仅安装 rocprof-compute 没有帮助。
根据 兼容加速器列表 检查 GPU。该表列出产品系列,因此应匹配
rocminfo
输出的
Marketing Name
, 并使用
gfx
名称进行确认。如果 GPU 不受支持,请停止操作并告知用户。
然后按照 安装指南 中的“Install the analyze mode dependencies”操作,它会将 rocprof-compute 安装到独立的 virtual environment 中。使用 nightly index,以确保版本与这些 skill 匹配。运行 rocprof-compute 前,使用
source <venv>/bin/activate
激活该环境,并通过
rocprof-compute analyze --debug-verify-deps
进行确认。 如果 application 是 Python script,请在激活环境前记录其 Python 的完整路径, 并使用该路径运行 script。

1. Check the environment

1. 检查环境

bash
rocprof-compute --specs
It reports what the profiler detected, including
ROCm Version
,
GPU Model
, and
GPU Arch
. Not every feature works on every GPU. The docs for each feature say which GPUs support it. When rocprof-compute reports that a feature is not supported, tell the user, and do not work around it.
bash
rocprof-compute --specs
它会报告 profiler 检测到的信息,包括
ROCm Version
、
GPU Model
和
GPU Arch
。并非每项 feature 都支持所有 GPU。每个 feature 的文档都会说明 支持它的 GPU。当 rocprof-compute 报告某项 feature 不受支持时,应告知用户, 不要尝试绕过该限制。

2. Collect what you need before profiling

2. 在 profile 前收集必要信息

Ask once:
  1. Launch command and arguments
  2. A short workload name, used as
    --name
  3. Which kernels matter, if not all
Profiling replays the application once per counter pass, so the command must be safe to run repeatedly and short enough to run several times. Ask for a reduced iteration count or problem size rather than profiling a long run. Do not invent a timeout; profile mode has no such option.
一次性询问以下信息:
  1. 启动命令及其参数
  2. 一个简短的 workload 名称,用作
    --name
  3. 哪些 kernels 很重要(如果并非全部 kernels)
Profiling 会为每个 counter pass 重放 application 一次,因此该命令必须能够 安全地重复运行,并且执行时间应足够短,以便运行多次。请用户减少 iteration count 或 problem size,而不是 profile 长时间运行的任务。不要自行设置 timeout; profile mode 没有此类选项。

3. Profile

3. Profile

Start with Speed-of-Light for the kernels that matter:
bash
rocprof-compute profile --name <workload_name> --speed-of-light -k <kernel_name> -- <application> <args>
On ROCm 10.2 and earlier, use
-b sol
instead of
--speed-of-light
.
Drop
-k
when the user has not named kernels. Once Speed-of-Light points at a limit, profile again with only the blocks section 5 opens, such as
wavefront
,
spi
, or
cu_pipe
.
Output lands in
./workloads/<workload_name>/<gpu_model>/
, or
<rank>/
under MPI. Change it with
--output-directory
, which replaces
--name
for building the path. Re-profiling into a non-empty directory needs
--overwrite
; prefer a fresh directory.
To restrict which GPU the application uses, set
HIP_VISIBLE_DEVICES
. Under Slurm, derive it from
ROCR_VISIBLE_DEVICES
, not from
SLURM_STEP_GPUS
; the indices differ.
See profile mode for output layout, filtering, and the full option list. To profile a process that is already running, see live attach and detach.
先对重要 kernels 运行 Speed-of-Light:
bash
rocprof-compute profile --name <workload_name> --speed-of-light -k <kernel_name> -- <application> <args>
在 ROCm 10.2 及更早版本上,使用
-b sol
代替
--speed-of-light
。
如果用户没有指定 kernels,请移除
-k
。当 Speed-of-Light 指向某个限制后, 再次进行 profile,但仅收集第 5 节所打开的 blocks,例如
wavefront
、
spi
或
cu_pipe
。
输出位于
./workloads/<workload_name>/<gpu_model>/
,或者在 MPI 下位于
<rank>/
。可以使用
--output-directory
更改输出位置;构建路径时,该参数 取代
--name
。重新 profile 到非空目录需要使用
--overwrite
,但最好使用 一个新目录。
要限制 application 使用的 GPU,请设置
HIP_VISIBLE_DEVICES
。在 Slurm 下, 应从
ROCR_VISIBLE_DEVICES
推导该值,而不是使用
SLURM_STEP_GPUS
,因为 两者的索引不同。
有关输出布局、过滤方式和完整选项列表,请参阅 profile mode。 要 profile 已在运行的 process,请参阅 live attach and detach。

Collect only what the question needs

仅收集问题所需的数据

Never run a full profile. It replays the application once per counter pass and also runs the roofline benchmarks.
  • List what this GPU can collect with
    rocprof-compute --list-metrics
    , and a block's metrics with
    rocprof-compute --list-metrics <alias>
    . On ROCm 10.2 and earlier, use
    --list-blocks <gfx_arch>
    and
    --list-metrics <gfx_arch>
    . Pass
    -b
    only the block aliases, block ids, or metric ids (for example
    2.1.0
    ) the question needs. Aliases differ by architecture.
  • --set <set_name>
    collects a predefined topic in one pass.
    rocprof-compute profile --list-sets
    lists the sets.
  • Add
    -k <kernel_name>
    when the user names the kernels that matter.
  • Unless the question is about roofline, select blocks with
    -b
    and leave out block 4 (
    roof
    ), so the roofline benchmark is skipped.
  • --iteration-multiplexing
    spreads counters over repeated dispatches of each kernel with the same launch parameters, so it only helps when each profiled kernel runs many times with the same launch parameters. Pair it with
    -k
    . See Iteration multiplexing in profile mode.
  • Tell the user which analysis panels the narrower profile leaves empty.
不要运行完整 profile。它会为每个 counter pass 重放 application,并且还会运行 roofline benchmarks。
  • 使用
    rocprof-compute --list-metrics
    列出该 GPU 可以收集的数据,并使用
    rocprof-compute --list-metrics <alias>
    列出某个 block 的 metrics。在 ROCm 10.2 及更早版本上,使用
    --list-blocks <gfx_arch>
    和
    --list-metrics <gfx_arch>
    。仅通过
    -b
    传入问题所需的 block aliases、 block ids 或 metric ids,例如
    2.1.0
    。不同架构的 aliases 有所不同。
  • --set <set_name>
    一次 pass 即可收集一个预定义主题。
    rocprof-compute profile --list-sets
    可列出这些 sets。
  • 当用户指定重要 kernels 时,添加
    -k <kernel_name>
    。
  • 除非问题涉及 roofline,否则使用
    -b
    选择 blocks,并排除 block 4 (
    roof
    ),从而跳过 roofline benchmark。
  • --iteration-multiplexing
    会将 counters 分散到每个 kernel 的重复 dispatches 中,并要求这些 dispatches 具有相同的启动参数。因此,只有当每个被 profile 的 kernel 都使用相同启动参数运行很多次时,该选项才有用。应将其与
    -k
    配合使用。 有关 Iteration multiplexing,请参阅 profile mode。
  • 告知用户缩小 profile 范围后,哪些分析面板会保持为空。

4. Find the hot kernel

4. 找出热点 kernel

bash
rocprof-compute analyze --path ./workloads/<name>/<gpu_model> --list-stats
--path
must point at the directory holding
profiling_config.yaml
. Analyze does not search for it in nearby directories. If the path you have does not contain that file, look one level down for the GPU model or rank directory.
--list-stats
prints
Detected Kernels (sorted descending by duration)
and a
Dispatch list
. The kernel index is what
-k
takes. Dispatch ids are 1-based and go to
-d
.
--list-stats
cannot be used with
--output-format csv
or
db
.
Work on one kernel at a time:
bash
rocprof-compute analyze --path ./workloads/<name>/<gpu_model> -k 0
-k
at profile time is different: it filters by kernel name and is used as a regex, so
-k gemm
matches every kernel whose name contains
gemm
.
bash
rocprof-compute analyze --path ./workloads/<name>/<gpu_model> --list-stats
--path
必须指向包含
profiling_config.yaml
的目录。Analyze 不会在附近目录中 搜索该文件。如果当前路径不包含此文件,请向下查找一层,定位 GPU model 或 rank 目录。
--list-stats
会输出
Detected Kernels (sorted descending by duration)
和一个
Dispatch list
。Kernel index 即
-k
参数使用的值。Dispatch ids 从 1 开始, 用于
-d
。
--list-stats
不能与
--output-format csv
或
db
一起使用。
每次处理一个 kernel:
bash
rocprof-compute analyze --path ./workloads/<name>/<gpu_model> -k 0
Profile 阶段的
-k
用法不同:它按 kernel name 过滤,并作为 regex 使用。 因此,
-k gemm
会匹配名称中包含
gemm
的每个 kernel。

5. Narrow to a reason

5. 缩小范围并确定原因

Get the overview, then open exactly one detailed block:
bash
rocprof-compute analyze --path ./workloads/<name>/<gpu_model> -k 0 --speed-of-light
Read the
Percent of Peak
column. The report prints the peak for each metric, so compare against that rather than against a remembered threshold, and never compute a peak yourself.
Resolve block names with aliases, which are stable across releases, instead of numeric ids, which are architecture-specific:
bash
rocprof-compute analyze --path ./workloads/<name>/<gpu_model> --list-metrics
rocprof-compute analyze --path ./workloads/<name>/<gpu_model> --list-metrics sol
Where to go next:
Symptom in Speed-of-LightNext step
Memory throughput near peak, low computethe
rocprof-compute-memory
skill
Compute near peak, or unclear compute vs memorythe
rocprof-compute-roofline
skill
Everything far below peak
-b wavefront
for occupancy, then
-b spi
for launch limits, both below
Question is about instructions, source lines, or stall reasonsthe
rocprof-compute-pc-sampling
skill
Kernels come from PyTorchthe
rocprof-compute-torch-trace
skill
先获取 overview,然后只打开一个详细 block:
bash
rocprof-compute analyze --path ./workloads/<name>/<gpu_model> -k 0 --speed-of-light
读取
Percent of Peak
列。报告会给每个 metric 打印其峰值,因此应与该峰值 进行比较,而不是使用记忆中的阈值,并且绝不要自行计算峰值。
使用 aliases 解析 block names,因为 aliases 在不同 release 之间保持稳定; 不要使用 architecture-specific 的 numeric ids:
bash
rocprof-compute analyze --path ./workloads/<name>/<gpu_model> --list-metrics
rocprof-compute analyze --path ./workloads/<name>/<gpu_model> --list-metrics sol
后续操作:
Speed-of-Light 中的症状下一步
Memory throughput 接近峰值,但 compute 较低
rocprof-compute-memory
skill
Compute 接近峰值,或 compute 与 memory 的对比不明确
rocprof-compute-roofline
skill
所有指标都远低于峰值使用
-b wavefront
检查 occupancy,然后使用
-b spi
检查 launch limits,两者结果均偏低
问题涉及 instructions、source lines 或 stall reasons
rocprof-compute-pc-sampling
skill
Kernels 来自 PyTorch
rocprof-compute-torch-trace
skill

Occupancy and wavefront limits

Occupancy 与 wavefront limits

bash
rocprof-compute analyze --path ./workloads/<name>/<gpu_model> -k 0 -b wavefront
Compare active waves per CU against the theoretical occupancy in the same table. When theoretical occupancy is itself low, the limit is VGPR, SGPR, or LDS allocation. Scratch traffic points at register spilling. Low occupancy is not a defect when the kernel already saturates its limiting resource. Worked examples are in occupancy-limiters-example.rst.
bash
rocprof-compute analyze --path ./workloads/<name>/<gpu_model> -k 0 -b wavefront
将每个 CU 的 active waves 与同一表格中的 theoretical occupancy 进行比较。 如果 theoretical occupancy 本身较低,则限制因素是 VGPR、SGPR 或 LDS allocation。 Scratch traffic 表明存在 register spilling。当 kernel 已经充分利用其限制资源时, 低 occupancy 并不是缺陷。相关实例请参阅 occupancy-limiters-example.rst。

Scheduler and pipeline limits

Scheduler 与 pipeline limits

bash
undefined
bash
undefined

Workgroup launch rate and resource allocation limits

Workgroup launch rate and resource allocation limits

rocprof-compute analyze --path ./workloads/<name>/<gpu_model> -k 0 -b spi
rocprof-compute analyze --path ./workloads/<name>/<gpu_model> -k 0 -b spi

Pipeline utilization and instruction mix

Pipeline utilization and instruction mix

rocprof-compute analyze --path ./workloads/<name>/<gpu_model> -k 0 -b cu_pipe

There is no single stall panel. Stall counters sit in the block that owns the
hardware: LDS stalls in `lds`, cache stalls in `vl1d` and `l2`, fabric stalls
in `l2_per_channel`. Open the block the Speed-of-Light result points at, and
read what each counter means in the
[performance model](https://github.com/ROCm/rocm-systems/blob/d46f9b4416abca78eede7cbe8ed8c567010333c6/projects/rocprofiler-compute/docs/conceptual/performance-model.rst). Ground every
recommendation in a number you read from the report.
rocprof-compute analyze --path ./workloads/<name>/<gpu_model> -k 0 -b cu_pipe

没有单一的 stall panel。Stall counters 位于负责相应硬件的 block 中:LDS stalls
位于 `lds`,cache stalls 位于 `vl1d` 和 `l2`,fabric stalls 位于
`l2_per_channel`。打开 Speed-of-Light 结果所指向的 block,并在
[performance model](https://github.com/ROCm/rocm-systems/blob/d46f9b4416abca78eede7cbe8ed8c567010333c6/projects/rocprofiler-compute/docs/conceptual/performance-model.rst)
中了解每个 counter 的含义。每项建议都应以报告中读取到的数值为依据。

6. Normalization

6. 归一化

-n
/
--normal-unit
applies to every metric in the report at once:
per_kernel
(default),
per_wave
,
per_cycle
,
per_second
. Keep the default for a mixed report. Change it only when the user asks about one metric family and the unit still makes sense for it. Bandwidth per cycle does not.
-n
/
--normal-unit
会同时应用于报告中的每个 metric:
per_kernel
(默认)、
per_wave
、
per_cycle
、
per_second
。对于混合报告, 请保留默认值。只有当用户询问某一个 metric family,并且该单位仍然适用时, 才应更改它。Bandwidth per cycle 不适用。

7. Compare two runs

7. 比较两次运行

Pass
--path
twice to put two workloads side by side. Use this for:
  1. Measuring a software optimization against its baseline on the same GPU.
  2. Checking that the same code performs consistently across nodes with the same GPU architecture.
  3. Comparing the same code across different GPU architectures.
bash
rocprof-compute analyze \
    --path ./workloads/baseline/<gpu_model> \
    --path ./workloads/optimized/<gpu_model>
Cases 1 and 2 expect the numbers to match except where the change intended otherwise, so an unexplained difference is the finding. Case 3 compares different hardware, so compare percent of peak rather than absolute values.
Never present a comparison across different GPUs, partition modes, or clock settings without stating that those differ.
传入两次
--path
,即可并排比较两个 workloads。可用于:
  1. 在同一 GPU 上衡量 software optimization 相对于 baseline 的效果。
  2. 检查相同代码在具有相同 GPU architecture 的不同 nodes 上是否表现一致。
  3. 比较相同代码在不同 GPU architectures 上的表现。
bash
rocprof-compute analyze \
    --path ./workloads/baseline/<gpu_model> \
    --path ./workloads/optimized/<gpu_model>
对于情况 1 和 2,除非代码变更本身有意造成差异,否则数值应当一致,因此任何 无法解释的差异就是需要报告的发现。对于情况 3,由于硬件不同,应比较 percent of peak,而不是 absolute values。
比较不同 GPUs、partition modes 或 clock settings 时,务必明确说明这些条件存在差异。

8. Save a report

8. 保存报告

Terminal output is the default. Only choose a format when the user asks to save or export.
--output-format
takes
txt
,
csv
, or
db
, each of which disables terminal output, and
--output-name
sets the file name. See analysis output format.
默认使用 terminal output。只有当用户要求保存或导出时,才选择输出格式。
--output-format
支持
txt
、
csv
或
db
,选择其中任意一种都会禁用 terminal output;
--output-name
用于设置文件名。参阅 analysis output format。

9. When something looks wrong

9. 出现异常时

SymptomLikely cause
No workload data
--path
is not the directory with
profiling_config.yaml
Empty or missing panelsthose counters were not collected; re-profile
All metrics zerokernel too short, or no dispatches were captured
No kernel idsthe application launched no GPU kernels
Roofline missingnot collected, or unsupported on this architecture
Stop and report permission, firmware, and ROCm version failures instead of working around them. More in the FAQ.
症状可能原因
没有 workload 数据
--path
不是包含
profiling_config.yaml
的目录
Panel 为空或缺失未收集相应 counters;需要重新 profile
所有 metrics 均为零kernel 运行时间过短,或未捕获到 dispatches
没有 kernel idsapplication 未启动任何 GPU kernels
Roofline 缺失未收集,或此 architecture 不支持
遇到 permissions、firmware 或 ROCm version failures 时,应停止操作并报告问题, 不要尝试绕过。更多内容请参阅 FAQ。