serving-llms-on-instinct

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Serving LLMs on AMD Instinct

在AMD Instinct上部署LLM

Get a vLLM endpoint running on AMD Instinct GPU hardware.

在AMD Instinct GPU硬件上运行vLLM端点。

Prerequisites

前提条件

ROCm driver and
```
amd-smi
```
installed on the GPU host
Docker running and accessible (check with
```
docker ps
```
)
```
/dev/kfd
```
and
```
/dev/dri
```
present on the GPU host
HuggingFace token in
```
HF_TOKEN
```
env var (required for gated models; not required for Qwen3 or Gemma). For gated models (Llama 3.2, Gemma, etc.), the HF token must belong to an account that has accepted the model's license at
```
huggingface.co/<model_id>
```
. A valid token without license acceptance will fail with an opaque "Engine core initialization failed" error.
For remote GPU: SSH key access configured (
```
ssh <user>@<host>
```
must work without a password prompt). If only password access is available, set up keys first:
```
ssh-copy-id <user>@<host>
```

GPU主机上已安装ROCm驱动和
```
amd-smi
```
Docker已运行且可访问（通过
```
docker ps
```
检查）
GPU主机上存在
```
/dev/kfd
```
和
```
/dev/dri
```
环境变量
```
HF_TOKEN
```
中配置了HuggingFace令牌（ gated模型需要；Qwen3或Gemma不需要）。对于 gated模型（如Llama 3.2、Gemma等），HF令牌所属账户必须已在
```
huggingface.co/<model_id>
```
接受模型许可。未接受许可的有效令牌会导致模糊的“Engine core initialization failed”错误。
对于远程GPU：已配置SSH密钥访问（
```
ssh <user>@<host>
```
无需密码提示即可正常工作）。如果仅支持密码访问，请先设置密钥：
```
ssh-copy-id <user>@<host>
```

Data files

数据文件

Read these files directly to get model and GPU configuration:

data/recipes_cache.json
-- model configs synced from vllm-project/recipes. Each entry under

models.<HF_ID>.recipe

contains the full recipe with

model.base_args

model.base_env

features.tool_calling.args

features.reasoning.args

hardware_overrides.amd.extra_args

hardware_overrides.amd.extra_env

. The top-level

docker_image

field has the latest resolved vLLM ROCm image.

data/gpu_overrides.json
-- GPU-specific configuration. Contains
```
docker_flags
```
(mandatory for all AMD Instinct),
```
gpu_configs
```
keyed by gfx_version with
```
env_defaults
```
and
```
workarounds
```
, and
```
legacy_models
```
for models not yet in vLLM recipes.
data/blacklist.json
-- models in vLLM recipes that cannot be served as LLM endpoints. Includes diffusion/image/audio generation models, embedding models, rerankers, ASR models needing audio pipelines, and models requiring unreleased vLLM nightly builds. Check this before attempting to serve a model. If the user requests a blacklisted model, explain why it won't work and suggest an alternative.

If the user doesn't specify a model, default to Qwen/Qwen3.5-9B: dense multimodal with MTP, Apache 2.0 license (no HF token needed), fits on a single GPU, strong reasoning and tool-calling.

直接读取以下文件获取模型和GPU配置：

data/recipes_cache.json
—— 从vllm-project/recipes同步的模型配置。

models.<HF_ID>.recipe

下的每个条目包含完整配置，包括

model.base_args

、

model.base_env

、

features.tool_calling.args

、

features.reasoning.args

、

hardware_overrides.amd.extra_args

、

hardware_overrides.amd.extra_env

。顶层

docker_image

字段包含最新解析的vLLM ROCm镜像。

data/gpu_overrides.json
—— GPU专属配置。包含所有AMD Instinct必需的
```
docker_flags
```
、按gfx版本分类的
```
gpu_configs
```
（含
```
env_defaults
```
和
```
workarounds
```
），以及尚未纳入vLLM配置的
```
legacy_models
```
。
data/blacklist.json
—— vLLM配置中无法作为LLM端点部署的模型。包含扩散/图像/音频生成模型、嵌入模型、重排序模型、需要音频管道的ASR模型，以及需要未发布vLLM nightly版本的模型。部署模型前请先检查此文件。如果用户请求部署黑名单中的模型，请解释原因并建议替代方案。

如果用户未指定模型，默认使用Qwen/Qwen3.5-9B：支持MTP的密集型多模态模型，采用Apache 2.0许可（无需HF令牌），可在单GPU上运行，具备出色的推理和工具调用能力。

Step 1: Detect the GPU

步骤1：检测GPU

bash

python3 scripts/detect.py

bash

python3 scripts/detect.py

Remote:

远程部署：

python3 scripts/detect.py --host user@hostname


Returns JSON with `gfx_version`, `vram_gb`, `gpu_count`, `rocm_version`.

| gfx_version | Hardware | VRAM |
|---|---|---|
| gfx950 | MI350X / MI355X | 288 GB HBM3E |
| gfx942 | MI300X (192 GB) / MI325X (256 GB) / MI300A (128 GB) | varies |

If `gfx_version` is `unknown`: `amd-smi` ran but found no GPU. Check
`lsmod | grep amdgpu`.

python3 scripts/detect.py --host user@hostname


返回包含`gfx_version`、`vram_gb`、`gpu_count`、`rocm_version`的JSON数据。

| gfx_version | 硬件 | 显存 |
|---|---|---|
| gfx950 | MI350X / MI355X | 288 GB HBM3E |
| gfx942 | MI300X (192 GB) / MI325X (256 GB) / MI300A (128 GB) | 按需分配 |

如果`gfx_version`为`unknown`：`amd-smi`已运行但未检测到GPU。请检查`lsmod | grep amdgpu`。

Step 2: Validate the environment

步骤2：验证环境

bash

python3 scripts/validate.py --auto-fix

bash

python3 scripts/validate.py --auto-fix

Remote:

远程部署：

python3 scripts/validate.py --auto-fix --host user@hostname


Returns JSON with `ready` (bool), `errors`, `warnings`, `fixes_applied`.
Do not proceed if `ready` is `false`.

python3 scripts/validate.py --auto-fix --host user@hostname


返回包含`ready`（布尔值）、`errors`、`warnings`、`fixes_applied`的JSON数据。如果`ready`为`false`，请勿继续操作。

Step 3: Refresh recipes (if stale)

步骤3：刷新配置（若已过期）

Check

fetched_at

data/recipes_cache.json

. If older than 24 hours or the file is missing, refresh:

bash

python3 scripts/sync_recipes.py

This shallow-clones vllm-project/recipes from GitHub and fetches the latest Docker tag from Docker Hub. Takes ~10 seconds. If it fails, the existing cache still works.

查看

data/recipes_cache.json

中的

fetched_at

字段。如果超过24小时或文件缺失，请刷新：

bash

python3 scripts/sync_recipes.py

此命令会浅克隆GitHub上的vllm-project/recipes，并从Docker Hub获取最新的Docker标签。耗时约10秒。如果刷新失败，现有缓存仍可正常使用。

Step 4: Construct the Docker command

步骤4：构建Docker命令

Read

data/recipes_cache.json

and

data/gpu_overrides.json

directly. Build the Docker command by combining:

Docker flags from
```
gpu_overrides.json > docker_flags
```
(mandatory for all AMD GPUs)
HF cache mount:
```
-v ~/.cache/huggingface:/root/.cache/huggingface
```
(if a shared model cache directory exists on the host, check whether
```
models--*
```
directories are at the cache root or inside a
```
hub/
```
subdirectory -- mount accordingly to
```
/root/.cache/huggingface
```
or
```
/root/.cache/huggingface/hub
```
)
Port:
```
-p <port>:<port>
```
(default 8000)

Environment variables: merge

gpu_configs.<gfx_version>.env_defaults

with the recipe's

model.base_env

and

hardware_overrides.amd.extra_env

. Always add

--env HF_TOKEN=${HF_TOKEN}

Docker image: use
```
docker_image
```
from
```
recipes_cache.json
```
top level (unless the model needs a pinned image, e.g. GLM-4.5 needs
```
v0.15.1
```
). If the user specifies a Docker image version, check it against the recipe's
```
model.min_vllm_version
```
. Warn if the image is older -- the model may crash on startup with an opaque "Engine core initialization failed" error.
Model ID:
```
--model <HF_ID>
```

vLLM args: combine the recipe's

model.base_args

hardware_overrides.amd.extra_args

features.tool_calling.args

features.reasoning.args

. Add

--enable-auto-tool-choice

if not present. For multi-GPU, add

--tensor-parallel-size N

(see VRAM estimation below). For MoE models on multi-GPU, also add

--distributed-executor-backend mp

Port arg:
```
--port <port>
```

If the exact model ID is not in

recipes_cache.json

, check for a base model match by stripping date/version suffixes (e.g.,

Kimi-K2-Instruct

matches

Kimi-K2-Instruct-0905

). Use the base model's recipe if found.

If no recipe match, check

legacy_models

gpu_overrides.json

. If not there either, use a generic config with

--enable-auto-tool-choice --trust-remote-code --tool-call-parser hermes

Precision variant selection: Recipes may offer variants (default, fp8, nvfp4). Check

gpu_configs.<gfx_version>.precision.native

gpu_overrides.json

before selecting a variant. On gfx942 (MI300X), only

bf16

fp16

fp8_fnuz

, and

int8

are hardware-native. MXFP4 and NVFP4 compute is emulated (dequant to BF16 during matmul), but weights stay compressed in VRAM so quantized models still fit in less memory. On gfx950 (MI350X), MXFP4 is hardware-native.

VRAM estimation and fit check: Before constructing the Docker command, estimate whether the model fits the available hardware:

bash

python3 scripts/estimate_vram.py --model-id <HF_ID> --vram-gb <per_gpu_vram> --tp <N>

This queries the HuggingFace Hub API (no model download) and returns JSON with:

```
weight_memory_gb
```
-- total weight size
```
kv_cache_bytes_per_token
```
-- KV cache cost per token at BF16
```
fit.weights_fit
```
-- whether weights fit at the given TP
```
fit.recommended_max_model_len
```
-- max context the GPU can serve
```
fit.context_limited
```
-- true if KV cache limits context below the model's native max
```
fit.min_tp_required
```
-- minimum TP needed (only if weights don't fit)

Understanding the overhead: The script reserves ~4 GB for vLLM's runtime overhead (activation profiling, HIP graph capture, internal buffers). During startup, vLLM runs a profiling forward pass to measure peak activations, then captures HIP graphs for optimized decode. This startup peak is higher than steady-state. The

remaining_for_kv_gb

field reflects what's left after weights and this overhead.

Use

remaining_for_kv_gb

to decide:

remaining_for_kv_gb >= 6
: safe to run. If

context_limited: true

, add

--max-model-len <recommended_max_model_len>

to the vLLM args. Mention the FP8 KV cache option (

--kv-cache-dtype fp8

) if the user needs longer context (

fit.max_seq_len_fp8_kv

shows the gain).

remaining_for_kv_gb
between 2 and 6: tight but worth trying. Launch normally. If vLLM OOMs during HIP graph capture (check container logs for "out of memory" after "capturing CUDA/HIP graphs"), retry with
```
--enforce-eager
```
added to the vLLM args. This skips graph capture and frees 1-2 GB. The only cost is slightly higher decode latency.
remaining_for_kv_gb < 2
: too tight. Will likely OOM during the activation profiling step. Do not attempt.
weights_fit: false
with multiple GPUs: re-run with
```
--tp <min_tp_required>
```
and check again.
weights_fit: false
, not enough GPUs: look for quantized alternatives in this order: a. Recipe variants: the recipe may have
```
fp8
```
or
```
mxfp4
```
variants with a different
```
model_id
```
that points to a quantized checkpoint. b. Same provider: many providers release quantized versions alongside the base model (e.g.
```
Qwen/Qwen3.5-122B-FP8
```
from Qwen). Search HuggingFace for
```
<provider>/<model-name>
```
with FP8/GPTQ/AWQ suffixes. c. AMD quantized: AMD's Quark team publishes quantized models under the
```
amd/
```
org on HuggingFace (e.g.
```
amd/Kimi-K2-Instruct-w-mxfp4-a-fp8
```
). Search for
```
amd/<model-name>
```
variants. Run
```
estimate_vram.py
```
on the quantized model ID to verify it fits, then use that model ID instead.
Still doesn't fit: tell the user the model requires more VRAM than available and suggest either a smaller model or multi-GPU hardware. Do not attempt to launch.

Docker command template:

docker run -d --name vllm-<model-slug> \
  <docker_flags> \
  -v <hf_cache_mount> \
  -p <port>:<port> \
  --env <key>=<value> (for each env var) \
  --env HF_TOKEN=${HF_TOKEN} \
  <docker_image> \
  --model <model_id> \
  <vllm_args> \
  --port <port>

直接读取

data/recipes_cache.json

和

data/gpu_overrides.json

，通过以下步骤构建Docker命令：

Docker标志：来自
```
gpu_overrides.json > docker_flags
```
（所有AMD GPU必需）
HF缓存挂载：
```
-v ~/.cache/huggingface:/root/.cache/huggingface
```
（如果主机上存在共享模型缓存目录，请检查
```
models--*
```
目录是否位于缓存根目录或
```
hub/
```
子目录下，据此挂载到
```
/root/.cache/huggingface
```
或
```
/root/.cache/huggingface/hub
```
）
端口：
```
-p <port>:<port>
```
（默认8000）

环境变量：合并

gpu_configs.<gfx_version>.env_defaults

、配置中的

model.base_env

和

hardware_overrides.amd.extra_env

。务必添加

--env HF_TOKEN=${HF_TOKEN}

。

Docker镜像：使用
```
recipes_cache.json
```
顶层的
```
docker_image
```
（除非模型需要固定版本镜像，例如GLM-4.5需要
```
v0.15.1
```
）。如果用户指定了Docker镜像版本，请与配置中的
```
model.min_vllm_version
```
进行核对。如果镜像版本过旧，请发出警告——模型启动时可能会出现模糊的“Engine core initialization failed”错误。
模型ID：
```
--model <HF_ID>
```
vLLM参数：合并配置中的
```
model.base_args
```
+
```
hardware_overrides.amd.extra_args
```
+
```
features.tool_calling.args
```
+
```
features.reasoning.args
```
。如果未包含
```
--enable-auto-tool-choice
```
，请添加该参数。对于多GPU部署，添加
```
--tensor-parallel-size N
```
（请参考下方显存估算）。对于多GPU上的MoE模型，还需添加
```
--distributed-executor-backend mp
```
。
端口参数：
```
--port <port>
```

如果精确的模型ID不在

recipes_cache.json

中，请通过去除日期/版本后缀（例如

Kimi-K2-Instruct

匹配

Kimi-K2-Instruct-0905

）查找基础模型匹配项。如果找到，使用基础模型的配置。

如果未找到匹配的配置，请检查

gpu_overrides.json

中的

legacy_models

。如果仍未找到，使用通用配置：

--enable-auto-tool-choice --trust-remote-code --tool-call-parser hermes

。

精度变体选择：配置可能提供多种变体（默认、fp8、nvfp4）。选择变体前，请查看

gpu_overrides.json

中的

gpu_configs.<gfx_version>.precision.native

。在gfx942（MI300X）上，仅

bf16

、

fp16

、

fp8_fnuz

和

int8

是硬件原生支持的。MXFP4和NVFP4计算为模拟实现（矩阵乘法期间反量化为BF16），但权重仍以压缩形式存储在显存中，因此量化模型仍可节省内存。在gfx950（MI350X）上，MXFP4是硬件原生支持的。

显存估算与适配检查：构建Docker命令前，请估算模型是否适配可用硬件：

bash

python3 scripts/estimate_vram.py --model-id <HF_ID> --vram-gb <per_gpu_vram> --tp <N>

此命令查询HuggingFace Hub API（无需下载模型），返回包含以下内容的JSON数据：

```
weight_memory_gb
```
—— 总权重大小
```
kv_cache_bytes_per_token
```
—— BF16精度下每个令牌的KV缓存开销
```
fit.weights_fit
```
—— 给定TP配置下权重是否适配
```
fit.recommended_max_model_len
```
—— GPU可支持的最大上下文长度
```
fit.context_limited
```
—— 如果KV缓存限制上下文长度低于模型原生最大值，则为true
```
fit.min_tp_required
```
—— 所需的最小TP值（仅当权重无法适配时返回）

理解开销：该脚本为vLLM运行时开销（激活分析、HIP图捕获、内部缓冲区）预留约4GB显存。启动期间，vLLM会运行一次分析前向传递以测量峰值激活，然后捕获HIP图以优化解码。此启动峰值高于稳态。

remaining_for_kv_gb

字段反映权重和此开销后的剩余显存。

使用

remaining_for_kv_gb

进行决策：

remaining_for_kv_gb >= 6
：可安全运行。如果
```
context_limited: true
```
，请在vLLM参数中添加
```
--max-model-len <recommended_max_model_len>
```
。如果用户需要更长上下文（
```
fit.max_seq_len_fp8_kv
```
显示增益），可提及FP8 KV缓存选项（
```
--kv-cache-dtype fp8
```
）。
remaining_for_kv_gb
介于2和6之间：显存紧张但值得尝试。正常启动。如果vLLM在HIP图捕获期间出现OOM（检查容器日志中“capturing CUDA/HIP graphs”后的“out of memory”），请重试并在vLLM参数中添加
```
--enforce-eager
```
。这会跳过图捕获并释放1-2GB显存，代价是解码 latency略有增加。
remaining_for_kv_gb < 2
：显存过于紧张。激活分析步骤可能会出现OOM。请勿尝试部署。
weights_fit: false
且有多个GPU：使用
```
--tp <min_tp_required>
```
重新运行并再次检查。
weights_fit: false
且GPU数量不足：按以下顺序查找量化替代方案： a. 配置变体：配置可能包含
```
fp8
```
或
```
mxfp4
```
变体，其
```
model_id
```
指向量化检查点。 b. 同一提供商：许多提供商在发布基础模型的同时也会发布量化版本（例如Qwen的
```
Qwen/Qwen3.5-122B-FP8
```
）。在HuggingFace上搜索
```
<provider>/<model-name>
```
并带有FP8/GPTQ/AWQ后缀。 c. AMD量化模型：AMD的Quark团队在HuggingFace的
```
amd/
```
组织下发布量化模型（例如
```
amd/Kimi-K2-Instruct-w-mxfp4-a-fp8
```
）。搜索
```
amd/<model-name>
```
变体。对量化模型ID运行
```
estimate_vram.py
```
以验证是否适配，然后使用该模型ID进行部署。
仍无法适配：告知用户模型所需显存超过可用显存，并建议使用更小的模型或多GPU硬件。请勿尝试部署。

Docker命令模板：

docker run -d --name vllm-<model-slug> \
  <docker_flags> \
  -v <hf_cache_mount> \
  -p <port>:<port> \
  --env <key>=<value> (每个环境变量对应一行) \
  --env HF_TOKEN=${HF_TOKEN} \
  <docker_image> \
  --model <model_id> \
  <vllm_args> \
  --port <port>

Step 5: Confirm with the user

步骤5：与用户确认

Before launching, present a summary and ask the user to confirm:

Model: full HuggingFace ID (e.g.
```
Qwen/Qwen3.5-122B-Instruct
```
)
Precision: variant being used (e.g. BF16, FP8) and why
Weight memory: from estimate_vram.py
GPU: detected hardware and VRAM
TP: tensor parallelism degree (1, 2, 4, 8)
Context: max achievable context length (and whether it's limited)
Port: which port the endpoint will be on

If a quantized alternative was selected (Step 4 fit check), explain that the original model doesn't fit and which alternative is being used.

Wait for the user's confirmation before proceeding.

启动前，请向用户展示摘要并请求确认：

模型：完整的HuggingFace ID（例如
```
Qwen/Qwen3.5-122B-Instruct
```
）
精度：使用的变体（例如BF16、FP8）及原因
权重内存：来自estimate_vram.py的结果
GPU：检测到的硬件和显存
TP：张量并行度（1、2、4、8）
上下文：可实现的最大上下文长度（以及是否受限）
端口：端点将使用的端口

如果在步骤4的适配检查中选择了量化替代方案，请解释原模型无法适配，并说明使用的替代模型。

等待用户确认后再继续操作。

Step 6: Launch and verify

步骤6：启动与验证

Before launching, check for port conflicts:

bash

ss -tlnp 2>/dev/null | grep ':<port> '

If a Docker container is on that port, stop it with

docker rm -f <name>

Run the Docker command. Then poll health using this loop:

bash

while docker inspect --format='{{.State.Running}}' <container_name> 2>/dev/null | grep -q true; do
  curl -sf http://localhost:<port>/health && echo "READY" && exit 0
  sleep 60
done
echo "FAILED -- container exited"

A 503 during loading is normal. Choose the polling strategy based on model size (weight memory from hf-mem):

Small models (< 100 GB weights): run the poll as a blocking command with the Bash tool's
```
timeout
```
set to 600000 (10 minutes). Most cached models are ready within 2-5 minutes.
Large models (>= 100 GB weights): run the poll with the Bash tool's
```
run_in_background
```
set to
```
true
```
. Then use
```
TaskOutput
```
with
```
block: true
```
and
```
timeout: 600000
```
to wait up to 10 minutes per check. If the task is still running after that, call
```
TaskOutput
```
again with the same parameters. This uses only 1 turn per 10-minute wait instead of burning a turn every check. The background loop runs until the container is healthy or dies.

After health returns 200, send a warmup request (triggers HIP kernel compilation, ~40-45 seconds on gfx942):

bash

curl -s http://localhost:<port>/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"<model_id>","messages":[{"role":"user","content":"say hi"}],"max_tokens":5}'

After the warmup succeeds, present a connection table so the user can call the endpoint immediately:

Field	Value
Model	`<model_id>`
Served model name	`<served-model-name or model_id>`
Base URL	`http://<host>:<port>/v1`
API key	none (local)
Port	`<port>`
Tensor parallel	`<tp>`
Max context	`<context>`
GPU	`<detected GPU>`

Then give a ready-to-run example using those exact values:

bash

curl -s http://<host>:<port>/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"<model_id>","messages":[{"role":"user","content":"Hello"}]}'

启动前，请检查端口冲突：

bash

ss -tlnp 2>/dev/null | grep ':<port> '

如果该端口已被Docker容器占用，请使用

docker rm -f <name>

停止容器。

运行Docker命令。然后使用以下循环轮询健康状态：

bash

while docker inspect --format='{{.State.Running}}' <container_name> 2>/dev/null | grep -q true; do
  curl -sf http://localhost:<port>/health && echo "READY" && exit 0
  sleep 60
done
echo "FAILED -- container exited"

加载期间出现503错误是正常现象。根据模型大小（来自hf-mem的权重内存）选择轮询策略：

小型模型（权重<100GB）：将轮询作为阻塞命令运行，Bash工具的
```
timeout
```
设置为600000（10分钟）。大多数缓存模型会在2-5分钟内就绪。
大型模型（权重>=100GB）：将轮询设置为后台运行（Bash工具的
```
run_in_background
```
设为
```
true
```
）。然后使用
```
TaskOutput
```
，设置
```
block: true
```
和
```
timeout: 600000
```
，每次检查最多等待10分钟。如果任务在该时间后仍在运行，请再次调用
```
TaskOutput
```
并使用相同参数。这样每10分钟仅占用一次交互轮次，而非每次检查都消耗轮次。后台循环会持续运行，直到容器健康或停止。

健康检查返回200后，发送预热请求（触发HIP内核编译，在gfx942上约需40-45秒）：

bash

curl -s http://localhost:<port>/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"<model_id>","messages":[{"role":"user","content":"say hi"}],"max_tokens":5}'

预热成功后，向用户展示连接表，以便立即调用端点：

字段	值
模型	`<model_id>`
部署模型名称	`<served-model-name or model_id>`
基础URL	`http://<host>:<port>/v1`
API密钥	无（本地部署）
端口	`<port>`
张量并行	`<tp>`
最大上下文	`<context>`
GPU	`<detected GPU>`

然后提供使用上述值的可直接运行示例：

bash

curl -s http://<host>:<port>/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"<model_id>","messages":[{"role":"user","content":"Hello"}]}'

Remote vs. local

远程部署 vs 本地部署

All scripts accept

--host user@hostname

. When given, they SSH to the target. Set

ROCM_SSH_HOST

and

ROCM_SSH_USER

env vars to avoid passing

--host

every time.

For remote Docker commands, run them over SSH:

bash

ssh user@host 'docker run -d ...'

Use

localhost

for health/warmup curl URLs (curl runs on the remote host).

所有脚本均支持

--host user@hostname

参数。指定该参数后，脚本会通过SSH连接到目标主机。可设置环境变量

ROCM_SSH_HOST

和

ROCM_SSH_USER

，避免每次都传递

--host

参数。

对于远程Docker命令，通过SSH运行：

bash

ssh user@host 'docker run -d ...'

健康检查/预热的curl URL使用

localhost

（curl在远程主机上运行）。

Gotchas

常见问题

CUDA_VISIBLE_DEVICES
set to empty string -- ROCm maps this variable to

HIP_VISIBLE_DEVICES

. Setting it to an empty string hides all GPUs.

CUDA_VISIBLE_DEVICES=0,1

works fine for restricting GPUs (same as

HIP_VISIBLE_DEVICES=0,1

). If the host has it set to empty, unset it:

unset CUDA_VISIBLE_DEVICES

. Do not pass

--env CUDA_VISIBLE_DEVICES=

(empty) into Docker -- that also hides all GPUs inside the container.

FP4BMM crash on gfx942 (MI300X) -- If the container exits immediately with a segfault or illegal instruction:

VLLM_ROCM_USE_AITER_FP4BMM

must be

on gfx942. This is set correctly in

gpu_overrides.json

for gfx942. See vLLM issue #34641.

HIP error: no kernel image
-- The Docker image has no compiled kernel for your GPU's gfx version. Use

vllm/vllm-openai-rocm:latest

; it includes gfx942 and gfx950 kernels.

MLA models need
--block-size 1
-- DeepSeek-R1/V3, Kimi-K2.5. Without it the MLA attention backend silently falls back to a slower path. This is in the recipe args for these models.

MoE models on multi-GPU need
--distributed-executor-backend mp
-- Qwen3-235B, GLM-4.5, MiniMax-M2. The default distributed executor does not work reliably with MoE on ROCm.

OOM during HIP graph capture -- If the container logs show "out of memory" after "capturing CUDA graphs" or "capturing HIP graphs", the model fits in VRAM but there isn't enough headroom for graph capture. Retry with

--enforce-eager

added to the vLLM args. This disables graph capture and frees 1-2 GB. Trade-off: slightly higher decode latency, but the model runs.

"Engine core initialization failed" -- This opaque error means the engine core subprocess died. Check early container logs:

docker logs <name> 2>&1 | head -50

. Common causes: gated model access denied (license not accepted on HF), unsupported architecture on this vLLM version, OOM during weight loading, missing

--trust-remote-code

for custom architectures, or vLLM version too old for the model (check

min_vllm_version

in the recipe).

/dev/kfd
permission denied -- User is not in the

video

render

group. Fix:

sudo usermod -aG video,render $USER

(requires re-login).

SSH key not configured -- The scripts use

BatchMode=yes

SSH. If SSH fails with

Permission denied (publickey)

, configure key-based access first.

Restricting GPUs on shared hosts -- Use

--env HIP_VISIBLE_DEVICES=0,1

--env CUDA_VISIBLE_DEVICES=0,1

to target specific GPUs by index.

HIP_VISIBLE_DEVICES

is the canonical AMD variable;

CUDA_VISIBLE_DEVICES

also works (ROCm maps it). Never set either to an empty string.

CUDA_VISIBLE_DEVICES
设置为空字符串 —— ROCm会将此变量映射为

HIP_VISIBLE_DEVICES

。设置为空字符串会隐藏所有GPU。

CUDA_VISIBLE_DEVICES=0,1

可正常用于限制GPU（与

HIP_VISIBLE_DEVICES=0,1

效果相同）。如果主机上该变量为空，请取消设置：

unset CUDA_VISIBLE_DEVICES

。请勿将

--env CUDA_VISIBLE_DEVICES=

（空值）传入Docker——这会在容器内隐藏所有GPU。

gfx942（MI300X）上的FP4BMM崩溃 —— 如果容器立即退出并出现段错误或非法指令：gfx942上必须将

VLLM_ROCM_USE_AITER_FP4BMM

设置为

。

gpu_overrides.json

中已为gfx942正确配置此参数。请参考vLLM问题#34641。

HIP error: no kernel image
—— Docker镜像中没有针对您GPU gfx版本的编译内核。使用

vllm/vllm-openai-rocm:latest

镜像；它包含gfx942和gfx950内核。

MLA模型需要
--block-size 1
—— DeepSeek-R1/V3、Kimi-K2.5。如果不设置该参数，MLA注意力后端会静默回退到较慢路径。这些模型的配置参数中已包含该设置。

多GPU上的MoE模型需要
--distributed-executor-backend mp
—— Qwen3-235B、GLM-4.5、MiniMax-M2。默认分布式执行器在ROCm上无法可靠运行MoE模型。

HIP图捕获期间出现OOM —— 如果容器日志中“capturing CUDA graphs”或“capturing HIP graphs”后显示“out of memory”，说明模型可适配显存，但图捕获的显存余量不足。请重试并在vLLM参数中添加

--enforce-eager

。这会禁用图捕获并释放1-2GB显存，代价是解码 latency略有增加，但模型可正常运行。

"Engine core initialization failed" —— 此模糊错误表示引擎核心子进程已终止。请查看容器早期日志：

docker logs <name> 2>&1 | head -50

。常见原因：gated模型访问被拒绝（未在HF上接受许可）、此vLLM版本不支持该架构、权重加载期间出现OOM、自定义架构缺少

--trust-remote-code

参数，或vLLM版本过旧无法支持该模型（请检查配置中的

min_vllm_version

）。

/dev/kfd
权限被拒绝 —— 用户不在

video

或

render

用户组中。修复方法：

sudo usermod -aG video,render $USER

（需要重新登录）。

SSH密钥未配置 —— 脚本使用

BatchMode=yes

模式的SSH。如果SSH出现

Permission denied (publickey)

错误，请先配置基于密钥的访问。

共享主机上限制GPU —— 使用

--env HIP_VISIBLE_DEVICES=0,1

或

--env CUDA_VISIBLE_DEVICES=0,1

按索引指定目标GPU。

HIP_VISIBLE_DEVICES

是AMD的标准变量；

CUDA_VISIBLE_DEVICES

也可使用（ROCm会映射该变量）。请勿将任一变量设置为空字符串。

Reference

参考资料

Precision compatibility, VRAM estimation, Docker flags, and known quirks: reference.md

精度兼容性、显存估算、Docker标志及已知问题： reference.md