Back to Details

hf-cloud-sagemaker-production-defaults

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

SageMaker Production Defaults

SageMaker 生产环境默认配置

The difference between a demo endpoint and one you can leave running is: it scales with traffic, it tells you when it breaks, and you can debug it later. This skill makes those three the default rather than optional extras.
By the time this skill runs, the planner has chosen a real-time endpoint, IAM has a usable role, and image-selection has resolved a container URI + AMI version. This skill turns those into an actual deployment.
演示用端点与可长期运行的端点的区别在于：前者能随流量自动扩缩容、故障时主动告警，且后续可调试。本技能将这三项设为默认配置，而非可选附加功能。
当执行本技能时，规划器已选定实时端点类型、IAM已配置可用角色，且镜像选择环节已确定容器URI + AMI版本。本技能会将这些资源转化为实际部署。

What gets created

创建的资源内容

For every endpoint, the skill creates these as a unit:
  1. SageMaker Model — image + env vars + execution role + S3 artifacts
  2. Endpoint config — instance type, initial count, optional data capture
  3. Endpoint — the real-time endpoint serving inference
  4. Autoscaling target + policy — target tracking on invocations per instance
  5. CloudWatch alarms — latency, errors, platform overhead
Data capture (logging requests/responses to S3) is off by default — useful for debugging but creates ongoing S3 costs the user didn't necessarily ask for. Enable with 
--enable-data-capture
.
All resources get a consistent tag set including 
CreatedBy=agentic-deploy-skills
for later cleanup.
Defaults and reasoning in 
references/deployment-template.md
.
针对每个端点，本技能会统一创建以下资源：
  1. SageMaker Model — 镜像 + 环境变量 + 执行角色 + S3制品
  2. 端点配置 — 实例类型、初始实例数、可选数据捕获
  3. 端点 — 提供推理服务的实时端点
  4. 自动扩缩容目标 + 策略 — 基于单实例调用次数的目标追踪
  5. CloudWatch告警 — 延迟、错误、平台开销相关告警
数据捕获（将请求/响应日志存储到S3）默认关闭——虽有助于调试，但会产生用户未明确要求的持续S3成本。可通过
--enable-data-capture
开启。
所有资源都会添加统一标签集，包括用于后续清理的
CreatedBy=agentic-deploy-skills
标签。
默认配置及设计思路详见
references/deployment-template.md
。

Running the deployment

执行部署

—

文本生成大模型（vLLM）部署示例：

For a text-generation LLM (vLLM):
bash
python scripts/deploy.py \
    --model-name qwen3-medical \
    --image-uri "$IMAGE_URI" \
    --inference-ami-version "$AMI" \
    --role-arn "$ROLE_ARN" \
    --instance-type ml.g5.xlarge \
    --region "$REGION" \
    --env SM_VLLM_MODEL=Qwen/Qwen3-0.6B \
    --env SM_VLLM_HOST=0.0.0.0 \
    --env SM_VLLM_TRUST_REMOTE_CODE=true \
    --env SM_VLLM_MAX_MODEL_LEN=4096
For an embedding model (TEI, often on CPU):
bash
python scripts/deploy.py \
    --model-name bge-large-embeddings \
    --image-uri "$IMAGE_URI" \
    --role-arn "$ROLE_ARN" \
    --instance-type ml.c6i.2xlarge \
    --region "$REGION" \
    --env HF_MODEL_ID=BAAI/bge-large-en-v1.5
Note: TEI deployments do not need 
--inference-ami-version
. That flag is vLLM-specific. TEI env vars are also simpler (
HF_MODEL_ID
instead of 
SM_VLLM_*
, no host or trust-remote-code to configure).
Where each value comes from:
ParameterSource
--image-uri
hf-cloud-serving-image-selection
— agent reads from the AWS DLC catalog page
--inference-ami-version
hf-cloud-serving-image-selection
— required for vLLM tags containing cu130+
--role-arn
hf-cloud-sagemaker-iam-preflight
(
check_role.py
)
--region
hf-cloud-aws-context-discovery
--instance-type
User input or planner recommendation
--env
Model-specific; see 
hf-cloud-serving-image-selection
for required 
SM_VLLM_*
vars
--model-s3-uri
Optional — S3 path to model artifacts; omit if loading from HF Hub
The script creates resources in order with error handling, waits for 
InService
(up to 30 min), surfaces failure reasons, registers autoscaling and alarms, and prints a summary including the teardown command. Outputs a JSON blob on stdout with endpoint/config/model names for downstream scripting.
The scripts ship with this skill. If the installed copy is missing the 
scripts/
directory (some harnesses copy only SKILL.md on install), fetch them from the source repo rather than re-implementing them from this description.
Cold-start expectation: when the model loads from HF Hub, the download happens inside the container after the endpoint starts — 5–15+ minutes to InService is normal, not a failure. 
deploy.py
waits 30 minutes; if you write custom wait code, don't time out at 15. Pre-staging weights in S3 (
--model-s3-uri
) cuts this and removes the Hub dependency.
bash
python scripts/deploy.py \
    --model-name qwen3-medical \
    --image-uri "$IMAGE_URI" \
    --inference-ami-version "$AMI" \
    --role-arn "$ROLE_ARN" \
    --instance-type ml.g5.xlarge \
    --region "$REGION" \
    --env SM_VLLM_MODEL=Qwen/Qwen3-0.6B \
    --env SM_VLLM_HOST=0.0.0.0 \
    --env SM_VLLM_TRUST_REMOTE_CODE=true \
    --env SM_VLLM_MAX_MODEL_LEN=4096

InService is not success — smoke-test before declaring victory

嵌入模型（TEI，通常部署在CPU）示例：

InService
only means the container answered 
/ping
. In MMS-based containers (HF Inference Toolkit) the Java front-end answers pings even while the Python worker crash-loops — an endpoint can be InService and serve nothing. Two checks, always:
  1. One real invocation.
    • Real-time: 
      invoke_endpoint.py
      (below) with a minimal payload; require an HTTP 200 with a sane body.
    • Async: upload one input to S3, call 
      invoke-endpoint-async
      , poll the output URI for a few minutes (see "Invoking async endpoints"). A result object = success; an object at the failure URI, or nothing appearing, = broken.
  2. Scan the endpoint logs for worker-crash markers — catches the crash-loop case even when the smoke request merely times out:
    bash
    aws logs filter-log-events \
    --log-group-name /aws/sagemaker/Endpoints/<endpoint-name> \
    --filter-pattern '?"Worker died" ?"Load model failed" ?"ImportError"' \
    --region <region> --max-items 5
Only report the deployment complete after both pass. If the log scan hits, surface the actual traceback from CloudWatch — not the InService status.
bash
python scripts/deploy.py \
    --model-name bge-large-embeddings \
    --image-uri "$IMAGE_URI" \
    --role-arn "$ROLE_ARN" \
    --instance-type ml.c6i.2xlarge \
    --region "$REGION" \
    --env HF_MODEL_ID=BAAI/bge-large-en-v1.5
注意：TEI部署不需要
--inference-ami-version
参数，该参数仅针对vLLM。TEI的环境变量也更简单（使用
HF_MODEL_ID
而非
SM_VLLM_*
，无需配置host或trust-remote-code）。
各参数来源说明：
参数来源
--image-uri
hf-cloud-serving-image-selection
— 智能体从AWS DLC目录页面读取
--inference-ami-version
hf-cloud-serving-image-selection
— 对于包含cu130+的vLLM标签是必填项
--role-arn
hf-cloud-sagemaker-iam-preflight
（
check_role.py
脚本）
--region
hf-cloud-aws-context-discovery
--instance-type
用户输入或规划器推荐
--env
模型专属配置；需查看
hf-cloud-serving-image-selection
获取必填的
SM_VLLM_*
变量
--model-s3-uri
可选参数 — 模型制品的S3路径；若从HF Hub加载则可省略
脚本会按顺序创建资源并处理错误，等待端点进入
InService
状态（最长等待30分钟），展示失败原因，配置自动扩缩容和告警，并打印包含销毁命令的部署摘要。脚本会在标准输出中输出包含端点/配置/模型名称的JSON数据，供后续脚本调用。
这些脚本随本技能一同提供。若安装副本缺少
scripts/
目录（部分工具仅在安装时复制SKILL.md），请从源码仓库获取脚本，而非根据本文描述重新实现。
冷启动预期：当模型从HF Hub加载时，下载操作会在端点启动后在容器内进行——通常需要5-15+分钟才能进入
InService
状态，这属于正常情况而非故障。
deploy.py
会等待30分钟；若你编写自定义等待代码，不要将超时时间设为15分钟。提前将权重预存到S3（使用
--model-s3-uri
参数）可缩短冷启动时间，并消除对HF Hub的依赖。

Testing a real-time endpoint

InService状态不代表部署成功——务必先执行冒烟测试

Once the endpoint is 
InService
, test it with the bundled helper. It is cross-platform and BOM-safe — use it instead of hand-writing a payload file and calling 
invoke-endpoint
directly:
bash
undefined
InService
仅表示容器能响应
/ping
请求。在基于MMS的容器（HF Inference Toolkit）中，Java前端会在Python Worker崩溃循环时仍响应ping请求——此时端点虽处于InService状态，但无法提供服务。必须执行以下两项检查：
  1. 执行一次真实调用
    • 实时端点：使用下方的
      invoke_endpoint.py
      脚本，传入最小有效载荷；要求返回HTTP 200状态码及合理响应体。
    • 异步端点：将一个输入文件上传到S3，调用
      invoke-endpoint-async
      ，轮询输出URI几分钟（详见“调用异步端点”部分）。若生成结果对象则表示成功；若失败URI下出现对象或无任何内容，则表示部署失败。
  2. 扫描端点日志查找Worker崩溃标记——即使冒烟请求仅超时，也能捕获崩溃循环情况：
    bash
    aws logs filter-log-events \
    --log-group-name /aws/sagemaker/Endpoints/<endpoint-name> \
    --filter-pattern '?"Worker died" ?"Load model failed" ?"ImportError"' \
    --region <region> --max-items 5
只有两项检查都通过后，才能报告部署完成。若日志扫描命中错误标记，需从CloudWatch提取实际回溯信息，而非仅报告InService状态。

macOS / Linux

测试实时端点

python3 scripts/invoke_endpoint.py
--endpoint-name <endpoint-name>
--payload '{"inputs": "Hello"}'
--region "$REGION"

```powershell
当端点进入
InService
状态后，可使用附带的辅助脚本进行测试。该脚本跨平台且无BOM问题——请使用此脚本，而非手动编写载荷文件并直接调用
invoke-endpoint
：
bash
undefined

Windows (PowerShell)

macOS / Linux

python scripts\invoke_endpoint.py 
    --endpoint-name <endpoint-name>
--payload-file payload.json ` --region $REGION

It accepts either `--payload '<json>'` (inline) or `--payload-file <path>`, validates JSON, writes the request body as plain UTF-8, invokes the endpoint, and prints the response body to stdout.
python3 scripts/invoke_endpoint.py
--endpoint-name <endpoint-name>
--payload '{"inputs": "Hello"}'
--region "$REGION"

```powershell

The UTF-8 BOM gotcha (Windows)

Windows (PowerShell)

If you write the request payload yourself on Windows, do not use 
Set-Content -Encoding UTF8
— depending on the PowerShell version it prepends a UTF-8 byte-order mark (BOM). SageMaker's JSON parser rejects a BOM with a 400 
ModelError
:
Unexpected UTF-8 BOM (decode using utf-8-sig): line 1 column 1 (char 0)
This is not a model, endpoint-health, or image problem — only the file encoding of the request body. 
invoke_endpoint.py
avoids it entirely (it even strips a BOM from a 
--payload-file
that already has one). If you must call the CLI directly, write the body as BOM-free UTF-8:
powershell
undefined
python scripts\invoke_endpoint.py 
    --endpoint-name <endpoint-name>
--payload-file payload.json ` --region $REGION

脚本支持`--payload '<json>'`（内联方式）或`--payload-file <path>`（文件方式），会验证JSON格式，将请求体以纯UTF-8格式写入，调用端点，并将响应体打印到标准输出。

BOM-free UTF-8 — use this

Windows平台的UTF-8 BOM陷阱

[System.IO.File]::WriteAllText((Resolve-Path "payload.json"), $json, [System.Text.UTF8Encoding]::new($false))
aws sagemaker-runtime invoke-endpoint 
    --endpoint-name <endpoint-name>
--content-type application/json 
    --body fileb://payload.json
--region $REGION ` response.json

**Fallback:** if any invocation fails with `Unexpected UTF-8 BOM`, rewrite the payload as BOM-free UTF-8 (or re-run via `invoke_endpoint.py`) and retry once before treating the endpoint or model as broken.
若你在Windows平台自行编写请求载荷，不要使用
Set-Content -Encoding UTF8
命令——根据PowerShell版本不同，该命令可能会在文件开头添加UTF-8字节顺序标记（BOM）。SageMaker的JSON解析器会拒绝带BOM的请求，返回400
ModelError
错误：
Unexpected UTF-8 BOM (decode using utf-8-sig): line 1 column 1 (char 0)
这不是模型、端点健康或镜像的问题——仅与请求体的文件编码有关。
invoke_endpoint.py
可完全避免此问题（即使
--payload-file
指定的文件已包含BOM，脚本也会自动移除）。若你必须直接调用CLI，请将请求体写入无BOM的UTF-8文件：
powershell
undefined

Invoking a generative reranker (vLLM)

无BOM的UTF-8格式——请使用此命令

Generative rerankers (Qwen3-Reranker etc. — routed to the HuggingFace vLLM DLC by 
hf-cloud-serving-image-selection
) are causal LMs scored by their first generated token, not chat models. Use the completions API with a raw 
prompt
, not the messages/chat API: chat templating does not reliably honor 
chat_template_kwargs
such as 
{"enable_thinking": false}
, and a wrong template silently returns near-identical scores for every query–document pair instead of erroring.
Payload shape (Qwen3-Reranker's expected format — substitute 
{query}
/ 
{document}
):
json
{
  "prompt": "<|im_start|>system\nJudge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be \"yes\" or \"no\".<|im_end|>\n<|im_start|>user\n<Instruct>: Given a web search query, retrieve relevant passages that answer the query\n<Query>: {query}\n<Document>: {document}<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "max_tokens": 1,
  "temperature": 0,
  "logprobs": 20
}
The trailing 
<|im_start|>assistant\n<think>\n\n</think>\n\n
suffix is load-bearing: it pre-fills an empty thinking block so the first generated token is the yes/no judgment. Score from the returned logprobs: 
P("yes") / (P("yes") + P("no"))
. Sanity check the endpoint with one relevant pair (expect >0.9) and one irrelevant pair (expect <0.05) — near-identical scores across pairs mean the prompt template is wrong, not that the model is broken.
The same rule generalizes: for any thinking-mode model where the prompt must be byte-exact, prefer the raw completions API over chat.
[System.IO.File]::WriteAllText((Resolve-Path "payload.json"), $json, [System.Text.UTF8Encoding]::new($false))
aws sagemaker-runtime invoke-endpoint 
    --endpoint-name <endpoint-name>
--content-type application/json 
    --body fileb://payload.json
--region $REGION ` response.json

** fallback方案**：若调用时出现`Unexpected UTF-8 BOM`错误，请将载荷重新写入无BOM的UTF-8文件（或通过`invoke_endpoint.py`重新调用），并重试一次，再判断端点或模型是否存在问题。

Picking the image URI

调用生成式重排模型（vLLM）

The agent reads the image URI from AWS's Deep Learning Containers catalog — pick the row that matches the model family (HuggingFace vLLM for LLMs, TEI for embeddings, etc.), substitute 
<region>
with the deployment region, and pass to 
deploy.py --image-uri
.
For vLLM images specifically (both 
huggingface-vllm
and the AWS 
vllm
fallback), also check the tag's CUDA version:
bash
undefined
生成式重排模型（如Qwen3-Reranker等——由
hf-cloud-serving-image-selection
路由到HuggingFace vLLM DLC）是因果语言模型，通过首个生成的token进行评分，而非对话模型。请使用带原始
prompt
的补全API，而非消息/对话API：对话模板无法可靠支持
chat_template_kwargs
（如
{"enable_thinking": false}
），错误的模板会静默返回所有查询-文档对几乎相同的分数，而非抛出错误。
载荷格式（Qwen3-Reranker预期格式——替换
{query}
/ 
{document}
）：
json
{
  "prompt": "<|im_start|>system\nJudge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be \"yes\" or \"no\".<|im_end|>\n<|im_start|>user\n<Instruct>: Given a web search query, retrieve relevant passages that answer the query\n<Query>: {query}\n<Document>: {document}<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "max_tokens": 1,
  "temperature": 0,
  "logprobs": 20
}
末尾的
<|im_start|>assistant\n<think>\n\n</think>\n\n
后缀至关重要：它预填充了一个空的思考块，使首个生成的token即为yes/no判断结果。从返回的logprobs计算分数：
P("yes") / (P("yes") + P("no"))
。请用一对相关的查询-文档对（预期分数>0.9）和一对不相关的查询-文档对（预期分数<0.05）验证端点——若所有对的分数几乎相同，则说明提示模板错误，而非模型故障。
此规则可推广：对于任何要求提示字节级精确的思考模式模型，优先使用原始补全API而非对话API。

Example: HuggingFace vLLM 0.21.0 from the catalog

选择镜像URI

IMAGE_URI="763104351884.dkr.ecr.eu-west-1.amazonaws.com/huggingface-vllm:0.21.0-transformers5.8.1-gpu-py312-cu130-ubuntu22.04"
智能体从AWS的深度学习容器目录读取镜像URI——选择与模型家族匹配的行（LLM选HuggingFace vLLM，嵌入模型选TEI等），将
<region>
替换为部署区域，然后传入
deploy.py --image-uri
参数。
针对vLLM镜像（包括
huggingface-vllm
和AWS的
vllm
备用镜像），还需检查标签的CUDA版本：
bash
undefined

cu130 tag → must pass --inference-ami-version

示例：从目录获取的HuggingFace vLLM 0.21.0镜像

python deploy.py --image-uri "$IMAGE_URI"
--inference-ami-version al2-ami-sagemaker-inference-gpu-3-1
...

For tags with `cu129` or lower, omit `--inference-ami-version`. See `hf-cloud-serving-image-selection` for the full vLLM AMI lookup table and the env-var requirements for each image family.
IMAGE_URI="763104351884.dkr.ecr.eu-west-1.amazonaws.com/huggingface-vllm:0.21.0-transformers5.8.1-gpu-py312-cu130-ubuntu22.04"

Async inference deployments

cu130标签 → 必须传入--inference-ami-version参数

For long-running inferences (>60s), large payloads, or workloads that are bursty/sparse enough to benefit from scale-to-zero, use 
deploy_async.py
instead of 
deploy.py
. Async genuinely supports 
MinCapacity=0
— real-time autoscaling can't.
bash
python scripts/deploy_async.py \
    --model-name flux-text-to-image \
    --image-uri "$IMAGE_URI" \
    --role-arn "$ROLE_ARN" \
    --instance-type ml.g5.2xlarge \
    --region "$REGION" \
    --output-s3-uri s3://my-bucket/async-output/ \
    --env HF_MODEL_ID=black-forest-labs/FLUX.1-dev
Required extras over 
deploy.py
:
  • --output-s3-uri
    — where async results land (results are not returned synchronously)
Optional async-specific flags:
  • --failure-s3-uri
    — separate path for failed invocations
  • --success-sns-topic
    , 
    --error-sns-topic
    — get notified when async results are ready or fail
  • --min-capacity 0
    (the default) — scale to zero between batches
  • --backlog-per-instance-target N
    — target queue depth per instance (default 5)
  • --max-concurrent-invocations-per-instance N
    — default 4
python deploy.py --image-uri "$IMAGE_URI"
--inference-ami-version al2-ami-sagemaker-inference-gpu-3-1
...

对于`cu129`或更低版本的标签，可省略`--inference-ami-version`参数。完整的vLLM AMI查找表及各镜像家族的环境变量要求，请查看`hf-cloud-serving-image-selection`。

How scale-to-zero works

异步推理部署

The async script registers two autoscaling policies on the variant:
  1. Target-tracking on 
    ApproximateBacklogSizePerInstance
    — handles ongoing scaling between min and max
  2. Step-scaling triggered by a 
    HasBacklogWithoutCapacity
    CloudWatch alarm — handles 
    0→1
    wake-from-zero
Both are needed. Target-tracking alone cannot transition from zero (it can't divide by zero instances), so without the step policy the endpoint comes up, scales to zero after the first batch, and never wakes again. The script wires this up automatically.
对于运行时间较长的推理（>60秒）、大载荷，或具有突发性/稀疏性可受益于缩容至零的工作负载，请使用
deploy_async.py
而非
deploy.py
。异步部署真正支持
MinCapacity=0
——实时自动扩缩容无法实现此功能。
bash
python scripts/deploy_async.py \
    --model-name flux-text-to-image \
    --image-uri "$IMAGE_URI" \
    --role-arn "$ROLE_ARN" \
    --instance-type ml.g5.2xlarge \
    --region "$REGION" \
    --output-s3-uri s3://my-bucket/async-output/ \
    --env HF_MODEL_ID=black-forest-labs/FLUX.1-dev
相较于
deploy.py
，异步部署需额外传入以下必填参数：
  • --output-s3-uri
    — 异步推理结果的存储位置（结果不会同步返回）
可选的异步专属参数：
  • --failure-s3-uri
    — 失败调用的单独存储路径
  • --success-sns-topic
    , 
    --error-sns-topic
    — 异步推理完成或失败时接收通知
  • --min-capacity 0
    （默认值） — 批次间隔期间缩容至零
  • --backlog-per-instance-target N
    — 单实例的目标队列深度（默认值为5）
  • --max-concurrent-invocations-per-instance N
    — 默认值为4

Async alarms

缩容至零的实现机制

The script creates three CloudWatch alarms:
  • ApproximateBacklogSize > 50
    — queue is building faster than capacity can drain it
  • InvocationsFailed > 5
    — repeated processing failures
  • HasBacklogWithoutCapacity
    — drives the wake-from-zero policy (not a notification alarm; its action is the step-scaling policy, not the SNS topic)
If you pass 
--sns-alarm-topic <arn>
, the first two notify on that topic. The wake alarm always points at the step policy.
异步脚本会为变体注册两个自动扩缩容策略：
  1. 目标追踪策略 — 基于
    ApproximateBacklogSizePerInstance
    指标，处理最小和最大容量之间的动态扩缩容
  2. 阶梯扩缩容策略 — 由
    HasBacklogWithoutCapacity
    CloudWatch告警触发，处理从0→1的唤醒逻辑
两者缺一不可。仅使用目标追踪策略无法从零容量状态过渡（无法除以零实例数），因此若缺少阶梯策略，端点启动后会在首个批次完成后缩容至零，且无法再次唤醒。脚本会自动完成这些配置。

Invoking async endpoints

异步告警

Async endpoints aren't called synchronously. You upload the input to S3, call 
invoke-endpoint-async
with the S3 input location, and SageMaker writes the result to your 
--output-s3-uri
when done:
bash
undefined
脚本会创建三个CloudWatch告警：
  • ApproximateBacklogSize > 50
    — 队列增长速度超过容量处理速度
  • InvocationsFailed > 5
    — 出现多次处理失败
  • HasBacklogWithoutCapacity
    — 驱动从零唤醒的策略（非通知告警；其关联动作是阶梯扩缩容策略，而非SNS主题）
若传入
--sns-alarm-topic <arn>
参数，前两个告警会向该主题发送通知。唤醒告警始终关联阶梯策略。

Upload your input first

调用异步端点

aws s3 cp input.json s3://my-input-bucket/job1/input.json
异步端点不支持同步调用。你需要将输入文件上传到S3，调用
invoke-endpoint-async
并传入S3输入位置，SageMaker会在处理完成后将结果写入你指定的
--output-s3-uri
：
bash
undefined

Invoke

先上传输入文件

aws sagemaker-runtime invoke-endpoint-async
--endpoint-name <endpoint-name>
--input-location s3://my-input-bucket/job1/input.json
--content-type application/json
--region <region>
aws s3 cp input.json s3://my-input-bucket/job1/input.json

Poll for the result at your output URI

调用端点

aws s3 cp s3://my-bucket/async-output/<inference-id>.out result.json

The same UTF-8 BOM caveat applies to the `input.json` you upload (see "The UTF-8 BOM gotcha" above) — if you build it on Windows, write it as BOM-free UTF-8 or the container's JSON parser will reject it.

Teardown works the same as real-time: `python3 scripts/teardown.py <endpoint-name>` (the teardown script discovers policies and alarms by name prefix, so it handles both deployment modes).
aws sagemaker-runtime invoke-endpoint-async
--endpoint-name <endpoint-name>
--input-location s3://my-input-bucket/job1/input.json
--content-type application/json
--region <region>

Defaults at a glance

轮询输出URI获取结果

SettingDefaultOverride
Initial instance count1
--initial-instance-count
Autoscaling min / max1 / 4
--min-capacity
, 
--max-capacity
Autoscaling target20 invocations/min/instance
--target-invocations-per-instance
Data capturedisabled (opt-in)
--enable-data-capture
CloudWatch alarms3 alarms
--no-alarms
SNS notificationnone (alarms created but won't notify)
--sns-alarm-topic <arn>
Environment tag
dev
--environment
InferenceAmiVersionnone (SageMaker default)
--inference-ami-version
(REQUIRED for vLLM CUDA 13+)
Not defaulted (user-specific input needed): VPC config, KMS key, multi-variant, async inference.
aws s3 cp s3://my-bucket/async-output/<inference-id>.out result.json

上传的`input.json`同样存在UTF-8 BOM陷阱（详见“Windows平台的UTF-8 BOM陷阱”部分）——若你在Windows平台创建该文件，请写入无BOM的UTF-8格式，否则容器的JSON解析器会拒绝该文件。

销毁操作与实时端点相同：`python3 scripts/teardown.py <endpoint-name>`（销毁脚本会通过名称前缀自动发现策略和告警，因此支持两种部署模式）。

Autoscaling target — tune by model type

默认配置概览

The default 
--target-invocations-per-instance 20
is conservative and tuned for LLM workloads where each request takes 1–5 seconds. For embedding deployments (TEI), each request is much faster (typically <100ms on CPU, <20ms on GPU), so a single instance can handle far more throughput. For embedding deployments, raise the target to 100–500 depending on instance and model size. The default of 20 will trigger autoscaling far too aggressively for embeddings and waste money.
A rule of thumb: target value ≈ 60 / (typical request latency in seconds). LLM at 3s latency → target 20. Embedding at 100ms → target 600. Generative rerankers sit in between — they generate a single token per request, so ~40–100 is a reasonable target.
设置项默认值覆盖方式
初始实例数1
--initial-instance-count
自动扩缩容最小/最大容量1 / 4
--min-capacity
, 
--max-capacity
自动扩缩容目标值20 次调用/分钟/实例
--target-invocations-per-instance
数据捕获禁用（需手动开启）
--enable-data-capture
CloudWatch告警3个告警
--no-alarms
SNS通知无（告警已创建但不会发送通知）
--sns-alarm-topic <arn>
环境标签
dev
--environment
InferenceAmiVersion无（使用SageMaker默认值）
--inference-ami-version
（vLLM CUDA 13+版本必填）
未设置默认值的项（需用户提供专属输入）：VPC配置、KMS密钥、多变体、异步推理。

Data capture + IAM gotcha

自动扩缩容目标值——按模型类型调整

If the user enables data capture, the execution role needs S3 write access to the capture prefix. The default URI (
s3://sagemaker-<region>-<account>/<endpoint>/data-capture/
) is typically a different bucket than the model artifact bucket. If 
hf-cloud-sagemaker-iam-preflight
scoped the inline policy narrowly to just the model bucket, capture writes fail silently — endpoint keeps serving but no data appears.
If the user reports "data capture isn't showing up", check the role's S3 access. Either widen the inline policy or pass 
--data-capture-s3-uri
pointing to a bucket the role can write.
默认的
--target-invocations-per-instance 20
是保守值，针对的是单次请求耗时1-5秒的LLM工作负载。对于嵌入模型（TEI），单次请求速度更快（CPU上通常<100ms，GPU上<20ms），因此单实例可处理更高吞吐量。对于嵌入模型部署，请将目标值提高到100-500，具体取决于实例和模型大小。默认值20会导致嵌入模型的自动扩缩容触发过于频繁，造成资源浪费。
经验法则：目标值 ≈ 60 / （典型请求延迟，单位为秒）。LLM延迟3秒 → 目标值20；嵌入模型延迟100ms → 目标值600；生成式重排模型介于两者之间——单次请求生成一个token，因此目标值设为40-100较为合理。

Teardown

数据捕获 + IAM陷阱

bash
python3 scripts/teardown.py <endpoint-name> <region>   # macOS / Linux
python  scripts\teardown.py <endpoint-name> <region>   # Windows
Deletes in safe order: alarms → autoscaling → endpoint (stops billing) → endpoint config → model. Idempotent.
Does not delete: the IAM execution role (might be shared), data capture S3 objects (user might want to keep), SNS topic, original model artifacts.
Always tell the user about the teardown command after the deployment summary. Users forget; endpoints accrue cost.
若用户启用数据捕获，执行角色需要对捕获前缀所在的S3桶拥有写入权限。默认URI（
s3://sagemaker-<region>-<account>/<endpoint>/data-capture/
）通常与模型制品桶不同。若
hf-cloud-sagemaker-iam-preflight
将内联策略的范围严格限定为模型桶，则数据捕获写入会静默失败——端点仍能提供服务，但不会生成任何数据。
若用户反馈“数据捕获未生成内容”，请检查角色的S3访问权限。可放宽内联策略的范围，或传入
--data-capture-s3-uri
参数指向角色拥有写入权限的桶。

When the deployment fails

销毁部署

CannotStartContainerError
+ no CloudWatch logs ever created — the InferenceAmiVersion problem. If the image tag contains 
cu130
or later and you didn't pass 
--inference-ami-version al2-ami-sagemaker-inference-gpu-3-1
, this is the cause. See 
hf-cloud-serving-image-selection
. Do NOT chase images, IAM roles, env vars, or instance types — the failure signature is identical for many other things but the cause here is the AMI.
"Failed to pass ping health check" — the container did start and produced logs, but 
/ping
isn't responding. Check CloudWatch at 
/aws/sagemaker/Endpoints/<endpoint-name>
. Usually: wrong image for model architecture, missing HF token, or OOM.
"Container failed to start" (with logs present) — entrypoint ran, then exited. Check CloudWatch. Common: missing required env vars (
SM_VLLM_MODEL
, 
SM_VLLM_HOST
, 
SM_VLLM_TRUST_REMOTE_CODE
), wrong 
ModelDataUrl
format, unreadable model artifacts.
ResourceLimitExceeded
 — no quota for the instance type in this region. Request increase or pick a different type (the planner should have checked quotas up front — see 
hf-cloud-sagemaker-deployment-planner
).
ImportError: libtorch_cuda.so: undefined symbol: ncclCommResume
in CloudWatch logs — known packaging defect in 
huggingface-pytorch-inference
GPU images (see "Known-broken images" in 
hf-cloud-serving-image-selection
). Inside the container, so no env var, AMI, instance type, or sibling tag fixes it. Switch to DJL Inference.
InService, but invocations time out / async outputs never appear — dead Python worker behind a live MMS front-end. Run the log scan from "InService is not success" above; the traceback in CloudWatch is the real error.
403 Forbidden
downloading weights from HF Hub during startup — the container's bundled 
huggingface_hub
predates HF's XET CDN auth. Add 
--env HF_HUB_ENABLE_HF_TRANSFER=0
, or pre-stage the weights in S3. Note: this can mask a deeper failure (the worker may still crash after the download succeeds) — re-check logs after fixing it.
Diagnostic rule: when failures look identical across multiple configurations (different images, roles, instance types) and no logs are ever produced, the cause is almost always below the container — host AMI, networking, account-level — not the deployment config. Stop iterating on config; check the AMI version and account state.
Don't retry blindly. The script prints the specific 
FailureReason
from 
describe-endpoint
— fix the root cause before retrying.
bash
python3 scripts/teardown.py <endpoint-name> <region>   # macOS / Linux
python  scripts\teardown.py <endpoint-name> <region>   # Windows
销毁顺序（安全可靠）：告警 → 自动扩缩容配置 → 端点（停止计费） → 端点配置 → 模型。该操作具有幂等性。
不会销毁的资源：IAM执行角色（可能被共享使用）、数据捕获的S3对象（用户可能需要保留）、SNS主题、原始模型制品。
部署摘要输出后，务必告知用户销毁命令。用户容易遗忘，端点会持续产生费用。
—

部署失败排查

—
CannotStartContainerError
+ 从未生成CloudWatch日志——这是InferenceAmiVersion问题。若镜像标签包含
cu130
或更高版本，但未传入
--inference-ami-version al2-ami-sagemaker-inference-gpu-3-1
参数，则会导致此问题。请查看
hf-cloud-serving-image-selection
。不要排查镜像、IAM角色、环境变量或实例类型——许多其他问题的失败特征与此相同，但此处的根本原因是AMI版本。
"Failed to pass ping health check"——容器已启动并生成日志，但
/ping
请求无响应。请查看CloudWatch的
/aws/sagemaker/Endpoints/<endpoint-name>
日志。通常原因：模型架构与镜像不匹配、缺少HF令牌、内存不足（OOM）。
"Container failed to start"（已生成日志）——入口脚本已执行，但随后退出。请查看CloudWatch日志。常见原因：缺少必填环境变量（
SM_VLLM_MODEL
、
SM_VLLM_HOST
、
SM_VLLM_TRUST_REMOTE_CODE
）、
ModelDataUrl
格式错误、模型制品无法读取。
ResourceLimitExceeded
——当前区域无该实例类型的配额。请申请配额提升或选择其他实例类型（规划器应提前检查配额——详见
hf-cloud-sagemaker-deployment-planner
）。
CloudWatch日志中出现
ImportError: libtorch_cuda.so: undefined symbol: ncclCommResume
——
huggingface-pytorch-inference
GPU镜像存在已知打包缺陷（详见
hf-cloud-serving-image-selection
中的“已知故障镜像”部分）。此问题发生在容器内部，无法通过环境变量、AMI、实例类型或其他标签修复。请切换到DJL Inference镜像。
处于InService状态，但调用超时 / 异步输出从未生成——Python Worker已崩溃，但MMS前端仍正常运行。请执行“InService状态不代表部署成功”部分的日志扫描；CloudWatch中的回溯信息才是真正的错误原因。
启动时从HF Hub下载权重出现
403 Forbidden
——容器内置的
huggingface_hub
版本早于HF的XET CDN认证机制。添加
--env HF_HUB_ENABLE_HF_TRANSFER=0
参数，或提前将权重预存到S3。注意：此操作可能会掩盖更深层的故障（Worker可能在下载成功后仍崩溃）——修复后请重新检查日志。
诊断规则：若故障在多种配置（不同镜像、角色、实例类型）下表现相同，且从未生成任何日志，则根本原因几乎总是在容器层面之下——主机AMI、网络、账户级问题，而非部署配置问题。停止迭代配置；请检查AMI版本和账户状态。
不要盲目重试。脚本会打印
describe-endpoint
返回的具体
FailureReason
——修复根本原因后再重试。