rocprof-compute-kernel-bottleneck
Compare original and translation side by side
🇺🇸
Original
English🇨🇳
Translation
ChineseProfile an AMD GPU application and find the kernel bottleneck
分析 AMD GPU 应用并找出 kernel 瓶颈
Start here for any "profile this" or "why is this kernel slow" request. This
skill collects counters, narrows a workload down to one kernel, and identifies
what limits it. Hand off to a focused skill once the limit is clear.
Run and
before choosing flags. Options change between releases; do not guess one.
rocprof-compute profile --helprocprof-compute analyze --help对于任何“分析这个”或“这个 kernel 为什么慢”的请求,都从这里开始。该
skill 会收集 counters,将 workload 缩小到单个 kernel,并确定
限制它的因素。确定限制后,再移交给专门的 skill。
在选择 flags 之前,运行 和
。不同 release 之间 options 会发生变化;
不要凭空猜测。
rocprof-compute profile --helprocprof-compute analyze --helpTwo sources of performance data
两种性能数据来源
rocprof-compute collects from two independent sources, and they answer
different questions. Choose by what the user is asking, not by difficulty.
| Source | Answers | Skill |
|---|---|---|
| Perfmon counters | How the architecture behaves: bandwidth, cache, occupancy, pipeline utilization | this skill, then |
| PC sampling | How the source code behaves: which instruction or line is hot and why it stalls | |
Counters are the default starting point because they cover the whole kernel
cheaply. Go straight to when the user asks about instructions,
source lines, or stall reasons, and use both when an architectural limit needs
to be traced back to the code that causes it.
rocprof-compute-pc-samplingrocprof-compute 从两个独立来源收集数据,而它们回答的是不同的问题。
应根据用户的问题选择来源,而不是根据操作难度选择。
| Source | Answers | Skill |
|---|---|---|
| Perfmon counters | 架构如何运行:bandwidth、cache、occupancy、pipeline utilization | 此 skill,然后使用 |
| PC sampling | 源代码如何运行:哪条 instruction 或哪一行是热点,以及它为何 stall | |
默认从 counters 开始,因为它们可以低成本地覆盖整个 kernel。
当用户询问 instructions、source lines 或 stall reasons 时,直接使用
;当需要将某个架构限制追溯到导致该限制的
代码时,则结合使用两种方式。
rocprof-compute-pc-samplingBefore you start
开始之前
If finds the tool and
passes, skip this section.
On ROCm 10.2, use instead of .
command -v rocprof-computerocprof-compute analyze --debug-verify-deps--verify-deps--debug-verify-depsOtherwise, check that ROCm and a GPU are present:
bash
rocminfo
amd-smi listIf either command fails, ROCm is not installed or no GPU is visible. Stop and
point the user to "Install the ROCm Core SDK" in the
install guide. Installing rocprof-compute
alone does not help.
Check the GPU against
compatible accelerators.
The table lists product families, so match the that
prints, and use the name to confirm. If the GPU is not supported, stop and
tell the user.
Marketing NamerocminfogfxThen follow "Install the analyze mode dependencies" in the
install guide, which installs
rocprof-compute in a virtual environment of its own. Use the nightly index so the
version matches these skills. Activate that environment with
before you run rocprof-compute, and confirm with
. If the application is a Python
script, note the full path of its Python before you activate, and run the
script with that path.
source <venv>/bin/activaterocprof-compute analyze --debug-verify-deps如果 找到了该工具,并且
检查通过,则跳过本节。
在 ROCm 10.2 上,使用 代替 。
command -v rocprof-computerocprof-compute analyze --debug-verify-deps--verify-deps--debug-verify-deps否则,检查 ROCm 和 GPU 是否存在:
bash
rocminfo
amd-smi list如果任一命令失败,说明未安装 ROCm 或没有检测到 GPU。停止操作,并将用户引导至
安装指南
中的“Install the ROCm Core SDK”。仅安装 rocprof-compute 没有帮助。
根据
兼容加速器列表
检查 GPU。该表列出产品系列,因此应匹配 输出的 ,
并使用 名称进行确认。如果 GPU 不受支持,请停止操作并告知用户。
rocminfoMarketing Namegfx然后按照
安装指南
中的“Install the analyze mode dependencies”操作,它会将 rocprof-compute
安装到独立的 virtual environment 中。使用 nightly index,以确保版本与这些
skill 匹配。运行 rocprof-compute 前,使用
激活该环境,并通过 进行确认。
如果 application 是 Python script,请在激活环境前记录其 Python 的完整路径,
并使用该路径运行 script。
source <venv>/bin/activaterocprof-compute analyze --debug-verify-deps1. Check the environment
1. 检查环境
bash
rocprof-compute --specsIt reports what the profiler detected, including , ,
and . Not every feature works on every GPU. The docs for each feature
say which GPUs support it. When rocprof-compute reports that a feature is not
supported, tell the user, and do not work around it.
ROCm VersionGPU ModelGPU Archbash
rocprof-compute --specs它会报告 profiler 检测到的信息,包括 、 和
。并非每项 feature 都支持所有 GPU。每个 feature 的文档都会说明
支持它的 GPU。当 rocprof-compute 报告某项 feature 不受支持时,应告知用户,
不要尝试绕过该限制。
ROCm VersionGPU ModelGPU Arch2. Collect what you need before profiling
2. 在 profile 前收集必要信息
Ask once:
- Launch command and arguments
- A short workload name, used as
--name - Which kernels matter, if not all
Profiling replays the application once per counter pass, so the command must be
safe to run repeatedly and short enough to run several times. Ask for a reduced
iteration count or problem size rather than profiling a long run. Do not invent
a timeout; profile mode has no such option.
一次性询问以下信息:
- 启动命令及其参数
- 一个简短的 workload 名称,用作
--name - 哪些 kernels 很重要(如果并非全部 kernels)
Profiling 会为每个 counter pass 重放 application 一次,因此该命令必须能够
安全地重复运行,并且执行时间应足够短,以便运行多次。请用户减少 iteration
count 或 problem size,而不是 profile 长时间运行的任务。不要自行设置 timeout;
profile mode 没有此类选项。
3. Profile
3. Profile
Start with Speed-of-Light for the kernels that matter:
bash
rocprof-compute profile --name <workload_name> --speed-of-light -k <kernel_name> -- <application> <args>On ROCm 10.2 and earlier, use instead of .
-b sol--speed-of-lightDrop when the user has not named kernels. Once Speed-of-Light points at a
limit, profile again with only the blocks section 5 opens, such as ,
, or .
-kwavefrontspicu_pipeOutput lands in , or under
MPI. Change it with , which replaces for building
the path. Re-profiling into a non-empty directory needs ; prefer a
fresh directory.
./workloads/<workload_name>/<gpu_model>/<rank>/--output-directory--name--overwriteTo restrict which GPU the application uses, set . Under
Slurm, derive it from , not from ; the
indices differ.
HIP_VISIBLE_DEVICESROCR_VISIBLE_DEVICESSLURM_STEP_GPUSSee profile mode for output layout,
filtering, and the full option list. To profile a process that is already
running, see
live attach and detach.
先对重要 kernels 运行 Speed-of-Light:
bash
rocprof-compute profile --name <workload_name> --speed-of-light -k <kernel_name> -- <application> <args>在 ROCm 10.2 及更早版本上,使用 代替 。
-b sol--speed-of-light如果用户没有指定 kernels,请移除 。当 Speed-of-Light 指向某个限制后,
再次进行 profile,但仅收集第 5 节所打开的 blocks,例如 、
或 。
-kwavefrontspicu_pipe输出位于 ,或者在 MPI 下位于
。可以使用 更改输出位置;构建路径时,该参数
取代 。重新 profile 到非空目录需要使用 ,但最好使用
一个新目录。
./workloads/<workload_name>/<gpu_model>/<rank>/--output-directory--name--overwrite要限制 application 使用的 GPU,请设置 。在 Slurm 下,
应从 推导该值,而不是使用 ,因为
两者的索引不同。
HIP_VISIBLE_DEVICESROCR_VISIBLE_DEVICESSLURM_STEP_GPUS有关输出布局、过滤方式和完整选项列表,请参阅
profile mode。
要 profile 已在运行的 process,请参阅
live attach and detach。
Collect only what the question needs
仅收集问题所需的数据
Never run a full profile. It replays the application once per counter pass and
also runs the roofline benchmarks.
- List what this GPU can collect with , and a block's metrics with
rocprof-compute --list-metrics. On ROCm 10.2 and earlier, userocprof-compute --list-metrics <alias>and--list-blocks <gfx_arch>. Pass--list-metrics <gfx_arch>only the block aliases, block ids, or metric ids (for example-b) the question needs. Aliases differ by architecture.2.1.0 - collects a predefined topic in one pass.
--set <set_name>lists the sets.rocprof-compute profile --list-sets - Add when the user names the kernels that matter.
-k <kernel_name> - Unless the question is about roofline, select blocks with and leave out block 4 (
-b), so the roofline benchmark is skipped.roof - spreads counters over repeated dispatches of each kernel with the same launch parameters, so it only helps when each profiled kernel runs many times with the same launch parameters. Pair it with
--iteration-multiplexing. See Iteration multiplexing in profile mode.-k - Tell the user which analysis panels the narrower profile leaves empty.
不要运行完整 profile。它会为每个 counter pass 重放 application,并且还会运行
roofline benchmarks。
- 使用 列出该 GPU 可以收集的数据,并使用
rocprof-compute --list-metrics列出某个 block 的 metrics。在 ROCm 10.2 及更早版本上,使用rocprof-compute --list-metrics <alias>和--list-blocks <gfx_arch>。仅通过--list-metrics <gfx_arch>传入问题所需的 block aliases、 block ids 或 metric ids,例如-b。不同架构的 aliases 有所不同。2.1.0 - 一次 pass 即可收集一个预定义主题。
--set <set_name>可列出这些 sets。rocprof-compute profile --list-sets - 当用户指定重要 kernels 时,添加 。
-k <kernel_name> - 除非问题涉及 roofline,否则使用 选择 blocks,并排除 block 4 (
-b),从而跳过 roofline benchmark。roof - 会将 counters 分散到每个 kernel 的重复 dispatches 中,并要求这些 dispatches 具有相同的启动参数。因此,只有当每个被 profile 的 kernel 都使用相同启动参数运行很多次时,该选项才有用。应将其与
--iteration-multiplexing配合使用。 有关 Iteration multiplexing,请参阅 profile mode。-k - 告知用户缩小 profile 范围后,哪些分析面板会保持为空。
4. Find the hot kernel
4. 找出热点 kernel
bash
rocprof-compute analyze --path ./workloads/<name>/<gpu_model> --list-stats--pathprofiling_config.yaml--list-statsDetected Kernels (sorted descending by duration)Dispatch list-k-d--list-stats--output-format csvdbWork on one kernel at a time:
bash
rocprof-compute analyze --path ./workloads/<name>/<gpu_model> -k 0-k-k gemmgemmbash
rocprof-compute analyze --path ./workloads/<name>/<gpu_model> --list-stats--pathprofiling_config.yaml--list-statsDetected Kernels (sorted descending by duration)Dispatch list-k-d--list-stats--output-format csvdb每次处理一个 kernel:
bash
rocprof-compute analyze --path ./workloads/<name>/<gpu_model> -k 0Profile 阶段的 用法不同:它按 kernel name 过滤,并作为 regex 使用。
因此, 会匹配名称中包含 的每个 kernel。
-k-k gemmgemm5. Narrow to a reason
5. 缩小范围并确定原因
Get the overview, then open exactly one detailed block:
bash
rocprof-compute analyze --path ./workloads/<name>/<gpu_model> -k 0 --speed-of-lightRead the column. The report prints the peak for each metric,
so compare against that rather than against a remembered threshold, and never
compute a peak yourself.
Percent of PeakResolve block names with aliases, which are stable across releases, instead of
numeric ids, which are architecture-specific:
bash
rocprof-compute analyze --path ./workloads/<name>/<gpu_model> --list-metrics
rocprof-compute analyze --path ./workloads/<name>/<gpu_model> --list-metrics solWhere to go next:
| Symptom in Speed-of-Light | Next step |
|---|---|
| Memory throughput near peak, low compute | the |
| Compute near peak, or unclear compute vs memory | the |
| Everything far below peak | |
| Question is about instructions, source lines, or stall reasons | the |
| Kernels come from PyTorch | the |
先获取 overview,然后只打开一个详细 block:
bash
rocprof-compute analyze --path ./workloads/<name>/<gpu_model> -k 0 --speed-of-light读取 列。报告会给每个 metric 打印其峰值,因此应与该峰值
进行比较,而不是使用记忆中的阈值,并且绝不要自行计算峰值。
Percent of Peak使用 aliases 解析 block names,因为 aliases 在不同 release 之间保持稳定;
不要使用 architecture-specific 的 numeric ids:
bash
rocprof-compute analyze --path ./workloads/<name>/<gpu_model> --list-metrics
rocprof-compute analyze --path ./workloads/<name>/<gpu_model> --list-metrics sol后续操作:
| Speed-of-Light 中的症状 | 下一步 |
|---|---|
| Memory throughput 接近峰值,但 compute 较低 | |
| Compute 接近峰值,或 compute 与 memory 的对比不明确 | |
| 所有指标都远低于峰值 | 使用 |
| 问题涉及 instructions、source lines 或 stall reasons | |
| Kernels 来自 PyTorch | |
Occupancy and wavefront limits
Occupancy 与 wavefront limits
bash
rocprof-compute analyze --path ./workloads/<name>/<gpu_model> -k 0 -b wavefrontCompare active waves per CU against the theoretical occupancy in the same
table. When theoretical occupancy is itself low, the limit is VGPR, SGPR, or
LDS allocation. Scratch traffic points at register spilling. Low occupancy is
not a defect when the kernel already saturates its limiting resource. Worked
examples are in
occupancy-limiters-example.rst.
bash
rocprof-compute analyze --path ./workloads/<name>/<gpu_model> -k 0 -b wavefront将每个 CU 的 active waves 与同一表格中的 theoretical occupancy 进行比较。
如果 theoretical occupancy 本身较低,则限制因素是 VGPR、SGPR 或 LDS allocation。
Scratch traffic 表明存在 register spilling。当 kernel 已经充分利用其限制资源时,
低 occupancy 并不是缺陷。相关实例请参阅
occupancy-limiters-example.rst。
Scheduler and pipeline limits
Scheduler 与 pipeline limits
bash
undefinedbash
undefinedWorkgroup launch rate and resource allocation limits
Workgroup launch rate and resource allocation limits
rocprof-compute analyze --path ./workloads/<name>/<gpu_model> -k 0 -b spi
rocprof-compute analyze --path ./workloads/<name>/<gpu_model> -k 0 -b spi
Pipeline utilization and instruction mix
Pipeline utilization and instruction mix
rocprof-compute analyze --path ./workloads/<name>/<gpu_model> -k 0 -b cu_pipe
There is no single stall panel. Stall counters sit in the block that owns the
hardware: LDS stalls in `lds`, cache stalls in `vl1d` and `l2`, fabric stalls
in `l2_per_channel`. Open the block the Speed-of-Light result points at, and
read what each counter means in the
[performance model](https://github.com/ROCm/rocm-systems/blob/d46f9b4416abca78eede7cbe8ed8c567010333c6/projects/rocprofiler-compute/docs/conceptual/performance-model.rst). Ground every
recommendation in a number you read from the report.rocprof-compute analyze --path ./workloads/<name>/<gpu_model> -k 0 -b cu_pipe
没有单一的 stall panel。Stall counters 位于负责相应硬件的 block 中:LDS stalls
位于 `lds`,cache stalls 位于 `vl1d` 和 `l2`,fabric stalls 位于
`l2_per_channel`。打开 Speed-of-Light 结果所指向的 block,并在
[performance model](https://github.com/ROCm/rocm-systems/blob/d46f9b4416abca78eede7cbe8ed8c567010333c6/projects/rocprofiler-compute/docs/conceptual/performance-model.rst)
中了解每个 counter 的含义。每项建议都应以报告中读取到的数值为依据。6. Normalization
6. 归一化
-n--normal-unitper_kernelper_waveper_cycleper_second-n--normal-unitper_kernelper_waveper_cycleper_second7. Compare two runs
7. 比较两次运行
Pass twice to put two workloads side by side. Use this for:
--path- Measuring a software optimization against its baseline on the same GPU.
- Checking that the same code performs consistently across nodes with the same GPU architecture.
- Comparing the same code across different GPU architectures.
bash
rocprof-compute analyze \
--path ./workloads/baseline/<gpu_model> \
--path ./workloads/optimized/<gpu_model>Cases 1 and 2 expect the numbers to match except where the change intended
otherwise, so an unexplained difference is the finding. Case 3 compares
different hardware, so compare percent of peak rather than absolute values.
Never present a comparison across different GPUs, partition modes, or clock
settings without stating that those differ.
传入两次 ,即可并排比较两个 workloads。可用于:
--path- 在同一 GPU 上衡量 software optimization 相对于 baseline 的效果。
- 检查相同代码在具有相同 GPU architecture 的不同 nodes 上是否表现一致。
- 比较相同代码在不同 GPU architectures 上的表现。
bash
rocprof-compute analyze \
--path ./workloads/baseline/<gpu_model> \
--path ./workloads/optimized/<gpu_model>对于情况 1 和 2,除非代码变更本身有意造成差异,否则数值应当一致,因此任何
无法解释的差异就是需要报告的发现。对于情况 3,由于硬件不同,应比较 percent
of peak,而不是 absolute values。
比较不同 GPUs、partition modes 或 clock settings 时,务必明确说明这些条件存在差异。
8. Save a report
8. 保存报告
Terminal output is the default. Only choose a format when the user asks to save
or export. takes , , or , each of which
disables terminal output, and sets the file name. See
analysis output format.
--output-formattxtcsvdb--output-name默认使用 terminal output。只有当用户要求保存或导出时,才选择输出格式。
支持 、 或 ,选择其中任意一种都会禁用 terminal
output; 用于设置文件名。参阅
analysis output format。
--output-formattxtcsvdb--output-name9. When something looks wrong
9. 出现异常时
| Symptom | Likely cause |
|---|---|
| No workload data | |
| Empty or missing panels | those counters were not collected; re-profile |
| All metrics zero | kernel too short, or no dispatches were captured |
| No kernel ids | the application launched no GPU kernels |
| Roofline missing | not collected, or unsupported on this architecture |
Stop and report permission, firmware, and ROCm version failures instead of
working around them. More in the FAQ.
| 症状 | 可能原因 |
|---|---|
| 没有 workload 数据 | |
| Panel 为空或缺失 | 未收集相应 counters;需要重新 profile |
| 所有 metrics 均为零 | kernel 运行时间过短,或未捕获到 dispatches |
| 没有 kernel ids | application 未启动任何 GPU kernels |
| Roofline 缺失 | 未收集,或此 architecture 不支持 |
遇到 permissions、firmware 或 ROCm version failures 时,应停止操作并报告问题,
不要尝试绕过。更多内容请参阅
FAQ。