hf-cloud-serving-image-selection

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Serving Image Selection

服务镜像选择

The serving container is the single thing most likely to break a SageMaker deployment that "looked correct on paper". Wrong container, stale tag, or the wrong AMI — all produce the same opaque

Failed to pass health check

error.

服务容器是最可能导致SageMaker部署失败的因素，即使部署方案“理论上看起来没问题”。错误的容器、过时的标签或不匹配的AMI，都会导致相同的模糊错误：

Failed to pass health check

（健康检查未通过）。

Rule zero: HuggingFace images always win

规则零：HuggingFace镜像必须优先选用

When both a HuggingFace-curated family (

huggingface-vllm

huggingface-vllm-omni

huggingface-sglang

tei

huggingface-pytorch-inference

) and a generic family (

vllm

vllm-omni

sglang

djl-inference

) can serve the model, the HuggingFace one is mandatory, not preferred. The only valid reasons to use a generic image:

Verified incompatibility — the model needs an architecture/modality/feature no available HuggingFace tag supports, confirmed against the catalog (not assumed).
No HuggingFace tag exists in the target region and mirroring is not an option.
The HuggingFace image is in "Known-broken images" below.

A newer version number on the generic repo is not a reason. The AWS

vllm

repo often publishes a higher vLLM version than

huggingface-vllm

; an older-but-compatible

huggingface-vllm

tag still wins. "Latest vLLM" is not a requirement anyone stated — compatibility with the model is. If you fall back, record in the deployment log which of the three reasons applied.

当HuggingFace维护的镜像系列（

huggingface-vllm

、

huggingface-vllm-omni

、

huggingface-sglang

、

tei

、

huggingface-pytorch-inference

）和通用镜像系列（

vllm

、

vllm-omni

、

sglang

、

djl-inference

）都能为模型提供服务时，必须选用HuggingFace镜像，而非优先考虑。只有以下三种情况才允许使用通用镜像：

已验证不兼容——模型所需的架构/模态/功能没有可用的HuggingFace标签支持，且已通过目录确认（而非主观假设）。
目标区域没有HuggingFace标签，且无法进行镜像复制。
该HuggingFace镜像属于下文的“已知故障镜像”。

通用镜像仓库的版本更新不能成为选用理由。AWS的

vllm

仓库发布的vLLM版本通常比

huggingface-vllm

更高，但只要

huggingface-vllm

的旧版本与模型兼容，仍需选用它。“最新vLLM版本”并非硬性要求——与模型兼容才是核心。如果不得不 fallback 到通用镜像，请在部署日志中记录具体是上述哪一种原因。

Where image URIs come from

镜像URI的来源

Primary source: AWS's official Deep Learning Containers catalog.

URL: https://aws.github.io/deep-learning-containers/reference/available_images/

This page is AWS-maintained and lists every image family with example URIs, tags, CUDA versions, Python versions, and platform (SageMaker vs EC2/ECS/EKS). When picking a URI for a deployment, read it from this page directly — copy the example URL, substitute

<region>

with the user's region, and pass it to

deploy.py --image-uri

The example URLs use

763104351884

as the account ID for most regions. A few regions use different accounts (e.g.

eu-south-1

uses

692866216735

). Check the Region Availability page when in doubt.

Exception: none currently. Every image family used by this workflow is now on the AWS catalog page (TEI was added in late 2026). If you encounter a new family that isn't there, mirror it via

mirror_image.py

and pass the resulting URI directly.

主要来源：AWS官方深度学习容器目录

URL: https://aws.github.io/deep-learning-containers/reference/available_images/

该页面由AWS维护，列出了每个镜像系列的示例URI、标签、CUDA版本、Python版本以及适用平台（SageMaker vs EC2/ECS/EKS）。为部署选择URI时，直接从此页面读取——复制示例URL，将

<region>

替换为用户所在区域，然后传递给

deploy.py --image-uri

。

示例URL中，大多数区域使用

763104351884

作为账户ID。少数区域使用不同的账户（例如

eu-south-1

使用

692866216735

）。如有疑问，请查看区域可用性页面。

例外情况：目前无。本工作流使用的所有镜像系列现已收录在AWS目录页面中（TEI于2026年末添加）。如果遇到未收录的新系列，请通过

mirror_image.py

复制镜像，并直接传递生成的URI。

Quick decision

快速决策表

Model	Container family	How to get the URI
HuggingFace text-generation LLM (Llama, Qwen, Mistral, etc.)	HuggingFace vLLM	AWS catalog → "HuggingFace vLLM Inference" (ECR repo `huggingface-vllm` )
Same as above, multimodal	HuggingFace vLLM-Omni	AWS catalog → "HuggingFace vLLM-Omni Inference" (ECR repo `huggingface-vllm-omni` )
HuggingFace embeddings	TEI	AWS catalog → "HuggingFace Text Embeddings Inference"
Encoder / cross-encoder rerankers (BERT-family `*ForSequenceClassification` )	TEI	Same as embeddings
Generative rerankers (causal-LM, e.g. Qwen3-Reranker)	HuggingFace vLLM	Same as text-generation LLMs — not TEI, see "Rerankers: TEI or vLLM?"
Text-to-image / diffusion (Stable Diffusion, FLUX)	DJL Inference	AWS catalog → "DJL Inference" — not HF Inference Toolkit, see "Known-broken images"
HuggingFace classifiers, NER, QA, summarization	HF Inference Toolkit (CPU)	AWS catalog → "HuggingFace PyTorch Inference"; GPU tags currently broken — see "Known-broken images"
User specifically wants SGLang	HuggingFace SGLang	AWS catalog → "HuggingFace SGLang Inference"
No compatible `huggingface-vllm` tag (verified incompatibility or region gap — see "Rule zero")	vLLM (AWS)	AWS catalog → "vLLM" section — fallback only, never for version freshness
User specifically wants DJL-LMI	DJL Inference	AWS catalog → "DJL Inference"
Amazon Nova	SageMaker JumpStart	Use JumpStart, not raw endpoint creation
Custom inference code	BYOC	User provides URI

HuggingFace-curated DLCs are mandatory when one is compatible (see "Rule zero").

huggingface-vllm

is layered directly on the AWS vLLM DLC — identical
SM_VLLM_*
env contract and the same cu130 AMI rule — and adds current

transformers

, current

huggingface_hub

hf_xet

(avoids the XET-CDN 403 download failures older images hit), and HF performance defaults. It is also what SageMaker SDK v3 auto-routes to. The AWS

vllm

image is a compatibility escape hatch only; it usually shows a higher vLLM version than

huggingface-vllm

, and that is not a reason to pick it.

Do not use TGI. Text Generation Inference is archived. Models released after the archive (Qwen3 most famously) fail ping health checks on TGI. Use vLLM instead. (The SageMaker SDK v3 agrees: since PR #5960, June 2026, its

ModelBuilder

auto-routes

text-generation

to the HuggingFace vLLM DLC and multimodal tasks to HuggingFace vLLM-Omni.)

Full reasoning for each family in

references/model-to-image.md

模型类型	容器系列	URI获取方式
HuggingFace文本生成LLM（Llama、Qwen、Mistral等）	HuggingFace vLLM	AWS目录 → "HuggingFace vLLM Inference"（ECR仓库 `huggingface-vllm` ）
上述类型的多模态模型	HuggingFace vLLM-Omni	AWS目录 → "HuggingFace vLLM-Omni Inference"（ECR仓库 `huggingface-vllm-omni` ）
HuggingFace嵌入模型	TEI	AWS目录 → "HuggingFace Text Embeddings Inference"
编码器/交叉编码器重排序器（BERT家族 `*ForSequenceClassification` ）	TEI	与嵌入模型相同
生成式重排序器（因果语言模型，例如Qwen3-Reranker）	HuggingFace vLLM	与文本生成LLM相同——不要用TEI，详见“重排序器：TEI还是vLLM？”
文本转图像/扩散模型（Stable Diffusion、FLUX）	DJL Inference	AWS目录 → "DJL Inference"——不要用HF Inference Toolkit，详见“已知故障镜像”
HuggingFace分类器、命名实体识别（NER）、问答、摘要模型	HF Inference Toolkit（CPU版）	AWS目录 → "HuggingFace PyTorch Inference"；GPU标签目前存在故障——详见“已知故障镜像”
用户明确要求使用SGLang	HuggingFace SGLang	AWS目录 → "HuggingFace SGLang Inference"
无兼容的 `huggingface-vllm` 标签（已验证不兼容或区域缺失——见“规则零”）	vLLM（AWS官方版）	AWS目录 → "vLLM"章节——仅作为 fallback，绝不能因版本更新而选用
用户明确要求使用DJL-LMI	DJL Inference	AWS目录 → "DJL Inference"
Amazon Nova	SageMaker JumpStart	使用JumpStart，而非直接创建端点
自定义推理代码	BYOC（自定义容器）	用户提供URI

当存在兼容的HuggingFace维护DLC时，必须选用该镜像（见“规则零”）。

huggingface-vllm

直接基于AWS vLLM DLC构建——具有相同的
SM_VLLM_*
环境变量约定和cu130 AMI规则，并添加了最新版

transformers

、最新版

huggingface_hub

hf_xet

（避免旧镜像遇到的XET-CDN 403下载失败问题），以及HF性能默认配置。同时，SageMaker SDK v3会自动路由到该镜像。AWS官方

vllm

镜像仅作为兼容性逃生舱；它通常显示比

huggingface-vllm

更高的vLLM版本，但这不能成为选用它的理由。

不要使用TGI。Text Generation Inference已被归档。归档后发布的模型（最典型的是Qwen3）在TGI上会健康检查失败。请改用vLLM。（SageMaker SDK v3也认同这一点：自2026年6月的PR #5960起，其

ModelBuilder

会自动将

text-generation

任务路由到HuggingFace vLLM DLC，将多模态任务路由到HuggingFace vLLM-Omni。）

每个系列的完整选型理由见

references/model-to-image.md

。

Rerankers: TEI or vLLM?

重排序器：TEI还是vLLM？

"Reranker" covers two very different architectures, and picking wrong wastes a full endpoint-creation cycle (~20 min) before TEI rejects the model:

Encoder cross-encoders (BAAI/bge-reranker-*, mixedbread, most
```
sentence-transformers
```
rerankers) — BERT-family models with a classification head.
```
config.json
```
has
```
architectures: [..ForSequenceClassification]
```
on a TEI-supported encoder type. → TEI.
Generative rerankers (Qwen/Qwen3-Reranker-*, and similar causal-LM judges) — decoder LLMs that score relevance via the logprob of a yes/no token.
```
config.json
```
has
```
architectures: [..ForCausalLM]
```
. → HuggingFace vLLM, deployed exactly like a text-generation LLM. TEI will load the architecture then reject the
```
classifier
```
model type (Qwen3 support in TEI is embeddings-only). Invocation pattern (raw completions API,
```
max_tokens=1
```
, logprobs scoring) is in
```
hf-cloud-sagemaker-production-defaults
```
.

Preflight before creating any resources — one HTTP GET settles it:

bash

curl -s https://huggingface.co/<model-id>/raw/main/config.json

“重排序器”涵盖两种截然不同的架构，选型错误会浪费完整的端点创建周期（约20分钟），之后TEI才会拒绝模型：

编码器交叉编码器（BAAI/bge-reranker-*、mixedbread、大多数
```
sentence-transformers
```
重排序器）——带分类头的BERT家族模型。
```
config.json
```
中包含
```
architectures: [..ForSequenceClassification]
```
，且编码器类型受TEI支持。→ 选用TEI。
生成式重排序器（Qwen/Qwen3-Reranker-*及类似的因果语言模型判别器）——通过是/否标记的对数概率对相关性打分的解码器LLM。
```
config.json
```
中包含
```
architectures: [..ForCausalLM]
```
。→ 选用HuggingFace vLLM，部署方式与文本生成LLM完全相同。TEI会加载架构，但随后会拒绝
```
classifier
```
模型类型（TEI对Qwen3的支持仅针对嵌入模型）。调用模式（原始补全API，
```
max_tokens=1
```
，对数概率打分）见
```
hf-cloud-sagemaker-production-defaults
```
。

创建任何资源前请先预检——一次HTTP GET请求即可确定：

bash

curl -s https://huggingface.co/<model-id>/raw/main/config.json

"architectures": ["Qwen3ForCausalLM"] → vLLM

"architectures": ["Qwen3ForCausalLM"] → 使用vLLM

"architectures": ["XLMRobertaForSequenceClassification"] → TEI

"architectures": ["XLMRobertaForSequenceClassification"] → 使用TEI


For TEI also confirm the *(architecture, task)* pair: an architecture appearing in TEI's supported list means embeddings support, not necessarily classification/reranking support.

Heads-up: SageMaker SDK v3 (PR #5960) routes the `text-ranking` task to TEI **unconditionally** — correct for cross-encoders, wrong for generative rerankers. Don't treat the SDK's routing as evidence that TEI can serve a given reranker.


对于TEI，还需确认（架构，任务）配对：TEI支持列表中的架构仅表示支持嵌入，不一定支持分类/重排序。

注意：SageMaker SDK v3（PR #5960）会无条件将`text-ranking`任务路由到TEI——这对交叉编码器是正确的，但对生成式重排序器是错误的。不要将SDK的路由作为TEI可服务某重排序器的依据。

Workflow

工作流

For every family: read the URI from the AWS catalog page.

Open https://aws.github.io/deep-learning-containers/reference/available_images/
Find the section for the right family (e.g. "HuggingFace vLLM Inference" for HuggingFace LLMs, "HuggingFace Text Embeddings Inference" for embeddings)
Pick the newest row marked
```
SageMaker
```
for the platform column — newest within that family. Do not switch to another family's section because it lists a higher engine version (see "Rule zero")
Substitute
```
<region>
```
with the user's region (from
```
hf-cloud-aws-context-discovery
```
)
For vLLM: also check the AMI requirement (see "vLLM AMI requirement" below)

Pass the URI to

deploy.py --image-uri

(real-time) or

deploy_async.py --image-uri

(async)

对于所有镜像系列：直接从AWS目录页面读取URI。

打开https://aws.github.io/deep-learning-containers/reference/available_images/
找到对应系列的章节（例如，HuggingFace LLM对应“HuggingFace vLLM Inference”，嵌入模型对应“HuggingFace Text Embeddings Inference”）
选择平台列标记为
```
SageMaker
```
的最新行——在该系列内选择最新版本。不要因其他系列列出了更高的引擎版本而切换系列（见“规则零”）
将
```
<region>
```
替换为用户所在区域（来自
```
hf-cloud-aws-context-discovery
```
）
对于vLLM：还需检查AMI要求（见下文“vLLM AMI要求”）
将URI传递给
```
deploy.py --image-uri
```
（实时部署）或
```
deploy_async.py --image-uri
```
（异步部署）

TEI: pick the right variant

TEI：选择正确的变体

The TEI catalog row lists two URIs — GPU (

tei

repo) and CPU (

tei-cpu

repo). Pick based on the instance type:

```
ml.g*
```
,
```
ml.p*
```
,
```
ml.inf*
```
→ GPU variant
```
ml.c*
```
,
```
ml.m*
```
,
```
ml.t*
```
→ CPU variant

Mixing them fails: CPU image on a GPU instance wastes hardware, GPU image on a CPU instance fails to start.

Note on the TEI account ID: the catalog page shows

683313688378

as the example account, but TEI is published from a different account namespace than the main AWS DLCs and the per-region account IDs vary. If

683313688378.dkr.ecr.<region>.amazonaws.com/tei:...

returns an ECR pull error for a region other than us-east-1, check the Region Availability page for the correct account ID for that region.

TEI目录行列出两个URI——GPU版（

tei

仓库）和CPU版（

tei-cpu

仓库）。根据实例类型选择：

```
ml.g*
```
、
```
ml.p*
```
、
```
ml.inf*
```
→ GPU变体
```
ml.c*
```
、
```
ml.m*
```
、
```
ml.t*
```
→ CPU变体

混用会失败：CPU镜像在GPU实例上会浪费硬件资源，GPU镜像在CPU实例上无法启动。

关于TEI账户ID的说明：目录页面显示示例账户为

683313688378

，但TEI的发布账户命名空间与AWS主DLC不同，且各区域的账户ID不同。如果

683313688378.dkr.ecr.<region>.amazonaws.com/tei:...

在us-east-1以外的区域返回ECR拉取错误，请查看区域可用性页面获取该区域的正确账户ID。

vLLM AMI requirement

vLLM AMI要求

vLLM DLC images with CUDA 13 or higher (current default:

cu130

) require setting

InferenceAmiVersion=al2-ami-sagemaker-inference-gpu-3-1

on the ProductionVariant. This applies equally to

huggingface-vllm

and

huggingface-vllm-omni

(layered on the same cu130 base) and to the AWS

vllm

repo. Without it the container dies on startup with no CloudWatch logs ever created. The failure looks identical to many other things (account-level issues, quota, networking) and routinely sends people down wrong diagnostic paths.

Lookup table:

Tag contains	InferenceAmiVersion to pass
`cu130` (or higher)	`al2-ami-sagemaker-inference-gpu-3-1`
`cu129` or lower	(omit the flag; default AMI works)

Rule of thumb: if the vLLM tag you picked contains

cu130

or later, pass

--inference-ami-version al2-ami-sagemaker-inference-gpu-3-1

deploy.py

. If a future CUDA version (cu140+) needs a different AMI, add a row to the table when AWS publishes the new image.

This is a vLLM-specific concern. TEI and HF Inference Toolkit images don't need an AMI override.

CUDA 13或更高版本的vLLM DLC镜像（当前默认：

cu130

）需要在ProductionVariant上设置

InferenceAmiVersion=al2-ami-sagemaker-inference-gpu-3-1

。此要求同样适用于

huggingface-vllm

和

huggingface-vllm-omni

（基于相同的cu130基础镜像构建）以及AWS官方

vllm

仓库。如果不设置该参数，容器会在启动时崩溃，且不会生成任何CloudWatch日志。这种失败与许多其他问题（账户级问题、配额、网络）的表现相同，通常会导致排查方向错误。

对照表：

标签包含内容	需传递的InferenceAmiVersion
`cu130` （或更高）	`al2-ami-sagemaker-inference-gpu-3-1`
`cu129` 或更低	（省略该参数；默认AMI即可）

经验法则：如果您选择的vLLM标签包含

cu130

或更高版本，请向

deploy.py

传递

--inference-ami-version al2-ami-sagemaker-inference-gpu-3-1

。如果未来的CUDA版本（如cu140+）需要不同的AMI，请在AWS发布新镜像后更新此表格。

这是vLLM特有的要求。TEI和HF Inference Toolkit镜像不需要覆盖AMI设置。

Configuring the vLLM DLCs (HuggingFace vLLM and AWS vLLM)

配置vLLM DLC（HuggingFace vLLM和AWS官方vLLM）

Both images share the same contract: configuration as environment variables on the SageMaker model definition,

SM_VLLM_*

mapped to vLLM CLI flags. The

huggingface-vllm

entrypoint additionally auto-detects the model when

SM_VLLM_MODEL

is unset — from

/opt/ml/model

if artifacts are mounted, else from

HF_MODEL_ID

— but setting

SM_VLLM_MODEL

explicitly works on both and is what our examples use.

两种镜像共享相同的配置约定：在SageMaker模型定义中通过环境变量配置，

SM_VLLM_*

对应vLLM CLI参数。

huggingface-vllm

入口点还会在

SM_VLLM_MODEL

未设置时自动检测模型——如果挂载了工件则从

/opt/ml/model

读取，否则从

HF_MODEL_ID

读取——但显式设置

SM_VLLM_MODEL

对两种镜像都有效，也是我们示例中使用的方式。

Required for every HuggingFace LLM deployment

每个HuggingFace LLM部署的必填配置

Env var	Purpose	Notes
`SM_VLLM_MODEL`	HF model ID (e.g. `Qwen/Qwen3-0.6B` ) or `/opt/ml/model` if loading from S3	—
`SM_VLLM_HOST`	Must be `0.0.0.0`	Otherwise vLLM binds localhost only, ping fails, container dies before logs. Top cause of mystery failures with this image.
`SM_VLLM_TRUST_REMOTE_CODE`	`true` for Qwen and several recent architectures	Set unconditionally — downside negligible, upside is the model loads.
`HUGGING_FACE_HUB_TOKEN`	HF token	Required for gated models.

环境变量	用途	说明
`SM_VLLM_MODEL`	HF模型ID（例如 `Qwen/Qwen3-0.6B` ），如果从S3加载则为 `/opt/ml/model`	—
`SM_VLLM_HOST`	必须设置为 `0.0.0.0`	否则vLLM仅绑定localhost，健康检查ping失败，容器在生成日志前就会崩溃。这是该镜像神秘故障的首要原因。
`SM_VLLM_TRUST_REMOTE_CODE`	对于Qwen和其他近期架构需设置为 `true`	无条件设置——负面影响可忽略，确保模型能加载。
`HUGGING_FACE_HUB_TOKEN`	HF令牌	gated模型必填。

Tuning (optional)

调优配置（可选）

Env var	Purpose
`SM_VLLM_MAX_MODEL_LEN`	Max sequence length — set this; defaults can be wrong for fine-tunes
`SM_VLLM_GPU_MEMORY_UTILIZATION`	Float 0.0–1.0, ~0.9 reasonable
`SM_VLLM_TENSOR_PARALLEL_SIZE`	GPU count for multi-GPU instances
`SM_VLLM_DTYPE`	`auto` , `bfloat16` , `float16`

Any vLLM CLI flag works — uppercase, replace dashes with underscores, prepend

SM_VLLM_

环境变量	用途
`SM_VLLM_MAX_MODEL_LEN`	最大序列长度——请设置此值；默认值可能不适用于微调模型
`SM_VLLM_GPU_MEMORY_UTILIZATION`	0.0–1.0之间的浮点数，设置为0.9较为合理
`SM_VLLM_TENSOR_PARALLEL_SIZE`	多GPU实例的GPU数量
`SM_VLLM_DTYPE`	`auto` 、 `bfloat16` 、 `float16`

任何vLLM CLI参数都可使用——转换为大写，将连字符替换为下划线，前缀添加

SM_VLLM_

。

Configuring TEI

配置TEI

Simpler env contract than vLLM:

Env var	Purpose	Required
`HF_MODEL_ID`	HF model ID (e.g. `BAAI/bge-large-en-v1.5` ) or `/opt/ml/model`	Yes
`HF_TOKEN`	HF auth token	Only for gated models
`MAX_BATCH_TOKENS`	Max tokens per batch (default 16384)	No
`MAX_CLIENT_BATCH_SIZE`	Max requests per client batch (default 32)	No

No host-binding to configure, no trust-remote-code flag. The architectures TEI supports (BERT, CamemBERT, RoBERTa, XLM-RoBERTa, NomicBert, JinaBert, JinaCodeBert, Mistral, Qwen2/3, Gemma2/3, ModernBert) are baked into the image.

TEI的环境变量约定比vLLM更简单：

环境变量	用途	是否必填
`HF_MODEL_ID`	HF模型ID（例如 `BAAI/bge-large-en-v1.5` ），如果从本地加载则为 `/opt/ml/model`	是
`HF_TOKEN`	HF认证令牌	仅gated模型需要
`MAX_BATCH_TOKENS`	每批最大令牌数（默认16384）	否
`MAX_CLIENT_BATCH_SIZE`	每个客户端批次的最大请求数（默认32）	否

无需配置主机绑定，也无需trust-remote-code参数。TEI支持的架构（BERT、CamemBERT、RoBERTa、XLM-RoBERTa、NomicBert、JinaBert、JinaCodeBert、Mistral、Qwen2/3、Gemma2/3、ModernBert）已内置到镜像中。

CUDA / instance compatibility

CUDA / 实例兼容性

Critical and easy to get wrong:

CUDA in image tag	Default AMI	With `al2-ami-sagemaker-inference-gpu-3-1`
cu124 / cu128	g5, g6, p5 all work	(not needed)
cu129	g6, p5; g5 fails (driver mismatch → CannotStartContainerError)	expected to fix g5 (unverified)
cu130+	fails everywhere — AMI flag is mandatory	g5, g6, p5 all work (cu130-on-g5 verified June 2026)

The driver comes from the host AMI, not the instance family — so passing the gpu-3-1 AMI (which vLLM cu130 images require anyway) also makes

ml.g5.*

viable for cu129+ images.

这一点至关重要且容易出错：

镜像标签中的CUDA版本	默认AMI	使用 `al2-ami-sagemaker-inference-gpu-3-1`
cu124 / cu128	g5、g6、p5均可用	（无需设置）
cu129	g6、p5可用；g5失败（驱动不匹配→CannotStartContainerError）	预计可修复g5问题（未验证）
cu130+	所有实例均失败——必须设置AMI参数	g5、g6、p5均可用（cu130在g5上的兼容性已在2026年6月验证）

驱动来自宿主机AMI，而非实例系列——因此传递gpu-3-1 AMI（vLLM cu130镜像本身就需要此AMI）也能让

ml.g5.*

实例适用于cu129+镜像。

VPC / NAT gateway problem

VPC / NAT网关问题

SageMaker endpoints inside a VPC without a NAT gateway can't pull from

public.ecr.aws

. The deployment fails with an image-pull error that doesn't mention "VPC" or "egress".

For images on AWS's regional ECR (everything in the catalog): SageMaker reaches them through built-in routing, no NAT needed. Use the regional URI pattern (

<account>.dkr.ecr.<region>.amazonaws.com/...

), not the

public.ecr.aws/...

pattern.

For images requiring

public.ecr.aws

access (less common): mirror to a private ECR repo in your account with

scripts/mirror_image.py

(cross-platform; needs Docker + the

aws

CLI). Run it from the shell where the AWS CLI works.

bash

undefined

位于VPC内但没有NAT网关的SageMaker端点无法从

public.ecr.aws

拉取镜像。部署会失败，且错误信息不会提及“VPC”或“出口”。

对于AWS区域ECR中的镜像（目录中的所有镜像）：SageMaker通过内置路由访问它们，无需NAT。请使用区域URI格式（

<account>.dkr.ecr.<region>.amazonaws.com/...

），而非

public.ecr.aws/...

格式。

对于需要访问

public.ecr.aws

的镜像（较少见）：使用

scripts/mirror_image.py

（跨平台；需要Docker和

aws

CLI）将镜像复制到您账户中的私有ECR仓库。在AWS CLI可用的Shell中运行该脚本。

bash

undefined

macOS / Linux

PRIVATE_URI=$(python3 scripts/mirror_image.py
public.ecr.aws/deep-learning-containers/vllm:<tag>
vllm-mirror)


```powershell

PRIVATE_URI=$(python3 scripts/mirror_image.py
public.ecr.aws/deep-learning-containers/vllm:<tag>
vllm-mirror)


```powershell

Windows (PowerShell) — capture stdout into a variable

Windows (PowerShell) — 将标准输出捕获到变量中

$PRIVATE_URI = python scripts\mirror_image.py ` public.ecr.aws/deep-learning-containers/vllm:<tag> vllm-mirror

undefined

$PRIVATE_URI = python scripts\mirror_image.py ` public.ecr.aws/deep-learning-containers/vllm:<tag> vllm-mirror

undefined

When the catalog page won't render, is stale, or is wrong

当目录页面无法渲染、过时或错误时

The page won't render / fetch returns junk: the catalog page is JavaScript-heavy and some fetch tools get an empty shell. Fallbacks, in order:

The catalog's source data on GitHub — the page is generated from one YAML file per version, listing exact tags, CUDA, and Python versions. List a family's files, then fetch the newest:

bash

curl -s https://api.github.com/repos/aws/deep-learning-containers/contents/docs/src/data/huggingface-vllm
curl -s https://raw.githubusercontent.com/aws/deep-learning-containers/main/docs/src/data/huggingface-vllm/0.21.0-gpu-sagemaker.yml

Directory names match ECR repos (

huggingface-vllm

huggingface-vllm-omni

huggingface-tei

vllm

djl-inference

, ...).

Query ECR directly for current tags in the target region (works with credentials that can read the DLC registry; if it returns AccessDenied, use the YAML files):

bash

aws ecr describe-images --registry-id 763104351884 --repository-name huggingface-vllm \
    --region <region> --query 'sort_by(imageDetails,&imagePushedAt)[-5:].imageTags' --output json

Release notes on the DLC GitHub repo.

A tag was just released and isn't on the page yet: rare; AWS updates the page on each release. Check the release notes above.

An architecture you need isn't supported by the listed image yet: for TEI specifically, you can mirror the upstream image from GHCR (

ghcr.io/huggingface/text-embeddings-inference:<version>

) into private ECR and pass the resulting URI directly to

deploy.py --image-uri

. Same

mirror_image.py

script.

页面无法渲染 / 请求返回无效内容：目录页面依赖JavaScript渲染，部分请求工具可能获取到空页面。按以下顺序使用 fallback 方案：

GitHub上的目录源数据——页面由每个版本对应的YAML文件生成，包含精确的标签、CUDA和Python版本。列出某系列的文件，然后获取最新版本：

bash

curl -s https://api.github.com/repos/aws/deep-learning-containers/contents/docs/src/data/huggingface-vllm
curl -s https://raw.githubusercontent.com/aws/deep-learning-containers/main/docs/src/data/huggingface-vllm/0.21.0-gpu-sagemaker.yml

目录名称与ECR仓库名称匹配（

huggingface-vllm

、

huggingface-vllm-omni

、

huggingface-tei

、

vllm

、

djl-inference

等）。

直接查询ECR获取目标区域的当前标签（需要能读取DLC注册表的凭证；如果返回AccessDenied，请使用YAML文件）：

bash

aws ecr describe-images --registry-id 763104351884 --repository-name huggingface-vllm \
    --region <region> --query 'sort_by(imageDetails,&imagePushedAt)[-5:].imageTags' --output json

DLC GitHub仓库的发布说明。

某标签刚发布但未出现在页面上：这种情况很少见；AWS会在每次发布时更新页面。请查看上述发布说明。

您需要的架构未被列出的镜像支持：特别是TEI，您可以将上游镜像从GHCR（

ghcr.io/huggingface/text-embeddings-inference:<version>

）复制到私有ECR，然后直接将生成的URI传递给

deploy.py --image-uri

。使用相同的

mirror_image.py

脚本即可。

Known-broken images (last checked July 2026)

已知故障镜像（最后检查时间：2026年7月）

Image	Defect	Use instead
`huggingface-pytorch-inference` GPU tags — all recent ones tested (PT 2.3–2.6, cu121/cu124, transformers 4.48–5.5.3)	`ImportError: libtorch_cuda.so: undefined symbol: ncclCommResume` at `import torch` . The NCCL bundled in the image is older than what torch links against — a packaging defect inside the container, on g5 and g6, regardless of AMI, model, or inference code. The MMS Java front-end keeps answering `/ping` , so the endpoint can reach InService while the Python worker crash-loops and serves nothing.	DJL Inference (bundles its own complete CUDA/NCCL stack) or BYOC. CPU tags are unaffected.

Re-check when AWS publishes new

huggingface-pytorch-inference

GPU tags — remove the row once a fixed image is confirmed.

General fallback rule: when an HF DLC fails with CUDA/NCCL linker errors, switch to DJL Inference rather than iterating over sibling tags — the defect class is per-repo, not per-tag (three different tags were tried for the case above; all broken).

Related HF Hub gotcha: older DLCs can fail model download with

403 Forbidden

from HF's XET CDN (their bundled

huggingface_hub

predates XET auth). Set

HF_HUB_ENABLE_HF_TRANSFER=0

to force the standard download path, or pre-stage weights in S3.

镜像	缺陷	替代方案
`huggingface-pytorch-inference` GPU标签——所有测试过的近期版本（PT 2.3–2.6，cu121/cu124，transformers 4.48–5.5.3）	执行 `import torch` 时出现 `ImportError: libtorch_cuda.so: undefined symbol: ncclCommResume` 错误。镜像中捆绑的NCCL版本低于torch依赖的版本——这是容器内部的打包缺陷，在g5和g6实例上均会出现，与AMI、模型或推理代码无关。MMS Java前端会持续响应 `/ping` ，因此端点会进入InService状态，但Python工作进程会崩溃循环，无法提供任何服务。	使用DJL Inference（捆绑了完整的CUDA/NCCL栈）或BYOC。CPU标签不受影响。

当AWS发布新的

huggingface-pytorch-inference

GPU标签时，请重新检查——确认修复后移除该行。

通用 fallback 规则：当HF DLC因CUDA/NCCL链接错误失败时，请切换到DJL Inference，而非尝试同系列的其他标签——此类缺陷是针对仓库而非标签的（上述案例中测试了三个不同标签，均存在故障）。

相关HF Hub陷阱：旧版DLC可能在模型下载时遇到HF XET CDN的

403 Forbidden

错误（它们捆绑的

huggingface_hub

版本早于XET认证功能）。设置

HF_HUB_ENABLE_HF_TRANSFER=0

强制使用标准下载路径，或提前将权重存储到S3中。

Hub download time at first boot

首次启动时的Hub下载时间

Loading the model from HF Hub happens inside the container after the endpoint starts — expect 5–15+ minutes before InService even for small models, longer for multi-GB ones. A slow first boot is not a failure; don't tear down or re-diagnose before the deploy script's 30-minute wait expires.

For production or repeated deployments, pre-stage the weights in S3 and pass

--model-s3-uri

deploy.py

(the model then loads from

/opt/ml/model

) — faster, immune to Hub rate limits/outages, and no

HUGGING_FACE_HUB_TOKEN

needed at runtime.

从HF Hub加载模型是在端点启动后在容器内部进行的——即使是小型模型，也需要5–15+分钟才能进入InService状态，多GB的模型耗时更长。首次启动缓慢并非故障；在部署脚本的30分钟等待时间到期前，请勿拆除或重新排查。

对于生产环境或重复部署，请提前将权重存储到S3中，并向

deploy.py

传递

--model-s3-uri

（模型将从

/opt/ml/model

加载）——速度更快，不受Hub速率限制/故障影响，且运行时无需

HUGGING_FACE_HUB_TOKEN

。