jetson-inference-mem-tune

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Jetson Inference Memory Tuning

Jetson推理内存调优

Recommends an inference runtime and the specific memory-related flags to pass to it, given the Jetson SKU/variant and the user's workload. Does not include quantization recipe selection — that lives in the model-benchmarking skill — but it does point at the precision floor each runtime can serve efficiently.

根据Jetson型号/SKU以及用户的工作负载，推荐推理运行时及需传入的特定内存相关参数。本内容不包含量化方案选择（该内容属于模型基准测试技能范畴），但会指出每个运行时可高效支持的最低精度。

Purpose

用途

Turn a live

jetson-memory-audit

snapshot into runtime and launch-flag recommendations for LLM/VLM serving on Jetson. Use this when the user needs to fit a model, reduce OOM risk, or switch to a lower-memory serving stack.

将实时的

jetson-memory-audit

快照转换为Jetson上LLM/VLM服务的运行时及启动参数建议。当用户需要适配模型、降低OOM风险或切换到低内存服务栈时使用本内容。

When to use

适用场景

"Which serving stack should I use on Orin Nano 8 GB to run a 7B model?"
"vLLM is OOMing — what should
```
--gpu-memory-utilization
```
and
```
--max-model-len
```
be?"
"Same model, less memory — can I switch from vLLM to llama.cpp?"
After
```
jetson-memory-audit
```
shows a model server is the top NvMap / PSS consumer.

“在Orin Nano 8 GB上运行7B模型应该使用哪个服务栈？”
“vLLM出现OOM问题——
```
--gpu-memory-utilization
```
和
```
--max-model-len
```
应该设为多少？”
“同一模型，想占用更少内存——我能从vLLM切换到llama.cpp吗？”
```
jetson-memory-audit
```
显示模型服务器是NvMap / PSS的顶级占用者之后。

Prerequisites

前置条件

Start with a current
```
jetson-memory-audit/scripts/audit.sh
```
JSON snapshot from the target Jetson.
Know the intended workload:
```
llm-server
```
,
```
vlm-server
```
,
```
embedding
```
, or
```
rag
```
.
If the user gives a desired free-memory target, pass it as
```
--target-mb
```
; otherwise let the script use SKU defaults.

从目标Jetson设备获取最新的
```
jetson-memory-audit/scripts/audit.sh
```
JSON快照。
明确预期工作负载：
```
llm-server
```
、
```
vlm-server
```
、
```
embedding
```
或
```
rag
```
。
如果用户指定了期望的空闲内存目标，传入
```
--target-mb
```
参数；否则让脚本使用SKU默认值。

Available Scripts

可用脚本

Script	Purpose	Arguments
`scripts/recommend.py`	Reads an audit JSON and emits runtime plus launch-flag recommendations.	`--audit PATH` , `--runtime` , `--workload` , `--target-mb` , `--human` .

If your agent runtime supports

run_script

, invoke

run_script("scripts/recommend.py", ["--audit", "/tmp/audit.json", "--runtime", "auto", "--workload", "llm-server"])

and summarize the returned JSON. Otherwise run it with

python3

from the repository root.

脚本	用途	参数
`scripts/recommend.py`	读取审计JSON并输出运行时及启动参数建议。	`--audit PATH` , `--runtime` , `--workload` , `--target-mb` , `--human` 。

如果你的Agent运行时支持

run_script

，调用

run_script("scripts/recommend.py", ["--audit", "/tmp/audit.json", "--runtime", "auto", "--workload", "llm-server"])

并总结返回的JSON。否则从仓库根目录使用

python3

运行该脚本。

Instructions

操作步骤

Run
```
jetson-memory-audit/scripts/audit.sh
```
to capture the device baseline.

Run

scripts/recommend.py --audit /tmp/audit.json --runtime auto --workload llm-server --target-mb 6000

to get a JSON of runtime + flag recommendations.

The agent presents the suggested runtime and the exact CLI flags. The user (or an outer agent) launches / restarts the server with those flags.
Re-run the audit to verify.

运行
```
jetson-memory-audit/scripts/audit.sh
```
捕获设备基线数据。

运行

scripts/recommend.py --audit /tmp/audit.json --runtime auto --workload llm-server --target-mb 6000

获取包含运行时+参数建议的JSON。

Agent展示推荐的运行时及具体CLI参数。用户（或外层Agent）使用这些参数启动/重启服务器。
重新运行审计以验证效果。

Expected workflow

预期工作流

Use

scripts/recommend.py

for the specific prompt and answer from the JSON it emits. If direct execution is blocked, run it as

python3 {baseDir}/scripts/recommend.py ...

For vLLM OOM prompts, run with

--runtime vllm --workload llm-server

and include concrete

--gpu-memory-utilization=<0.x>

and

--max-model-len=<number>

values from

launch_flags

For "lowest memory" or Orin Nano 8 GB prompts, run with
```
--runtime auto --workload llm-server
```
; prefer the runtime in the JSON and explicitly mention the GGUF / 4-bit tradeoff when it selects
```
llama-cpp
```
.
For SGLang prompts, run with
```
--runtime sglang
```
and quote
```
--mem-fraction-static
```
,
```
--max-running-requests
```
, and any context/KV-cache note.
For "switch from vLLM to llama.cpp" prompts, run with
```
--runtime llama-cpp
```
and quote
```
-ngl
```
,
```
-c
```
, and
```
--no-mmap
```
.

针对特定提示使用

scripts/recommend.py

，并根据其输出的JSON给出答案。如果无法直接执行，运行

python3 {baseDir}/scripts/recommend.py ...

。

对于vLLM OOM相关提示，使用

--runtime vllm --workload llm-server

运行脚本，并从

launch_flags

中提取具体的

--gpu-memory-utilization=<0.x>

和

--max-model-len=<number>

值。

对于“最低内存占用”或Orin Nano 8 GB相关提示，使用
```
--runtime auto --workload llm-server
```
运行脚本；优先选择JSON中的运行时，当选择
```
llama-cpp
```
时需明确提及GGUF / 4-bit的权衡。
对于SGLang相关提示，使用
```
--runtime sglang
```
运行脚本，并引用
```
--mem-fraction-static
```
、
```
--max-running-requests
```
及任何上下文/KV缓存说明。
对于“从vLLM切换到llama.cpp”相关提示，使用
```
--runtime llama-cpp
```
运行脚本，并引用
```
-ngl
```
、
```
-c
```
和
```
--no-mmap
```
参数。

Limitations

局限性

Recommendations are only as fresh as the audit JSON. Re-run
```
jetson-memory-audit
```
after stopping services, changing power mode, or restarting model servers.
The script estimates memory pressure from SKU defaults and audit totals; model-specific KV-cache, quantization, and tokenizer behavior can still require benchmarking.
This skill emits flags only. It does not start, stop, or restart model servers.

建议的时效性与审计JSON一致。停止服务、更改电源模式或重启模型服务器后，需重新运行
```
jetson-memory-audit
```
。
脚本根据SKU默认值和审计总量估算内存压力；但模型特定的KV缓存、量化和分词器行为仍可能需要基准测试。
本技能仅输出参数，不启动、停止或重启模型服务器。

Error handling

错误处理

Exit
```
2
```
: the audit JSON could not be read, parsed, or did not contain valid numeric memory fields. Ask the user to rerun
```
jetson-memory-audit/scripts/audit.sh
```
.
Exit
```
3
```
: unsupported runtime or workload request. Re-run with one of the
```
--runtime
```
and
```
--workload
```
values listed in
```
scripts/recommend.py --help
```
.
Empty or missing
```
launch_flags
```
: do not invent fallback flags. Report the script failure and ask for a fresh audit or a supported runtime.

退出码
```
2
```
：无法读取、解析审计JSON，或JSON中不包含有效的数值内存字段。请用户重新运行
```
jetson-memory-audit/scripts/audit.sh
```
。
退出码
```
3
```
：请求的运行时或工作负载不被支持。使用
```
scripts/recommend.py --help
```
中列出的
```
--runtime
```
和
```
--workload
```
值重新运行。
```
launch_flags
```
为空或缺失：不要自行生成备用参数。报告脚本失败，并要求用户提供新的审计数据或使用受支持的运行时。

Output contract for

recommend.py

recommend.py

输出约定

json

{
  "sku": "orin-nx",
  "variant": "orin-nx-16gb",
  "mem_total_gb": 16,
  "runtime": "vllm",
  "rationale": "Highest throughput at this memory budget given continuous batching + paged attention.",
  "launch_flags": [
    "--gpu-memory-utilization=0.55",
    "--max-model-len=4096",
    "--max-num-seqs=8",
    "--enable-prefix-caching"
  ],
  "alternatives": [
    { "runtime": "llama-cpp", "rationale": "Lower memory floor with GGUF Q4_K_M.", "launch_flags": ["-ngl 28", "-c 4096", "--no-mmap"] }
  ],
  "notes": ["Lower --gpu-memory-utilization further if you also run a small VLM alongside."]
}

json

{
  "sku": "orin-nx",
  "variant": "orin-nx-16gb",
  "mem_total_gb": 16,
  "runtime": "vllm",
  "rationale": "Highest throughput at this memory budget given continuous batching + paged attention.",
  "launch_flags": [
    "--gpu-memory-utilization=0.55",
    "--max-model-len=4096",
    "--max-num-seqs=8",
    "--enable-prefix-caching"
  ],
  "alternatives": [
    { "runtime": "llama-cpp", "rationale": "Lower memory floor with GGUF Q4_K_M.", "launch_flags": ["-ngl 28", "-c 4096", "--no-mmap"] }
  ],
  "notes": ["Lower --gpu-memory-utilization further if you also run a small VLM alongside."]
}

Runtimes covered

覆盖的运行时

Runtime	Best for	Key memory knobs	Preferred install path
llama.cpp	Tightest budget; GGUF; Orin Nano-class	`-ngl` , `-c` , `--mlock` , `--no-mmap`	`ghcr.io/nvidia-ai-iot/llama_cpp:latest-jetson-{orin,thor}`
vLLM	High-throughput serving with continuous batching	`--gpu-memory-utilization` , `--max-model-len` , `--max-num-seqs` , `--enable-prefix-caching`	Thor and Orin JetPack 7.2 / L4T r39+: upstream vLLM 0.20+ ( `vllm/vllm-openai` ) container or validated native vLLM 0.20+. Older Orin: NVIDIA-AI-IOT image
SGLang	Programmable workflows (RAG, tool use, structured output)	`--mem-fraction-static` , `--mem-fraction-dynamic` , `--max-running-requests`	Thor: NVIDIA SGLang 26.01 ( `nvcr.io/nvidia/sglang:26.01-py3` , SGLang 0.5.5.post2). Orin: JetPack-matched environment
TensorRT Edge-LLM	NVIDIA-tuned production serving	Build profile per SKU; paged-KV; KV reuse	Vendor docs for the target JetPack

For Orin JetPack 7.2 / L4T r39+, upstream vLLM 0.20+ is supported. For older Orin releases, prefer NVIDIA-AI-IOT prebuilt vLLM images where available because they ship the matching CUDA/cuDNN/TensorRT stack for JetPack. For Thor, prefer upstream vLLM 0.20+ (
vllm/vllm-openai
) or a validated native vLLM 0.20+ install; for SGLang use NVIDIA SGLang 26.01 (
nvcr.io/nvidia/sglang:26.01-py3
, SGLang 0.5.5.post2) or newer NVIDIA SGLang release notes that explicitly list Jetson Thor support. Do not force an Orin-specific Jetson container path on Thor, and do not assume native upstream SGLang support on Orin.

运行时	适用场景	关键内存参数	推荐安装路径
llama.cpp	内存预算紧张场景；GGUF格式模型；Orin Nano级设备	`-ngl` , `-c` , `--mlock` , `--no-mmap`	`ghcr.io/nvidia-ai-iot/llama_cpp:latest-jetson-{orin,thor}`
vLLM	支持连续批处理的高吞吐量服务场景	`--gpu-memory-utilization` , `--max-model-len` , `--max-num-seqs` , `--enable-prefix-caching`	Thor和Orin JetPack 7.2 / L4T r39+：上游vLLM 0.20+（ `vllm/vllm-openai` ）容器或经过验证的原生vLLM 0.20+。旧版Orin：NVIDIA-AI-IOT镜像
SGLang	可编程工作流（RAG、工具调用、结构化输出）场景	`--mem-fraction-static` , `--mem-fraction-dynamic` , `--max-running-requests`	Thor：NVIDIA SGLang 26.01（ `nvcr.io/nvidia/sglang:26.01-py3` ，SGLang 0.5.5.post2）。Orin：匹配JetPack的环境
TensorRT Edge-LLM	NVIDIA调优的生产级服务场景	针对每个SKU构建配置文件；分页KV；KV复用	目标JetPack的厂商文档

对于Orin JetPack 7.2 / L4T r39+，支持上游vLLM 0.20+。对于旧版Orin版本，优先使用NVIDIA-AI-IOT预构建的vLLM镜像（如果可用），因为它们附带匹配JetPack的CUDA/cuDNN/TensorRT栈。对于Thor，优先使用上游vLLM 0.20+（
vllm/vllm-openai
）或经过验证的原生vLLM 0.20+安装；对于SGLang，使用NVIDIA SGLang 26.01（
nvcr.io/nvidia/sglang:26.01-py3
，SGLang 0.5.5.post2）或明确列出Jetson Thor支持的更新版NVIDIA SGLang。不要在Thor上强制使用Orin特定的Jetson容器路径，也不要假设Orin支持原生上游SGLang。

Quantization recommendations

量化建议

Use runtime-specific quantization names. vLLM and SGLang usually consume Hugging Face checkpoints such as W4A16, AWQ, GPTQ, FP16, or NVFP4. llama.cpp and Ollama consume GGUF models, so recommend INT4/Q4_K_M-style GGUF instead.

Runtime family	Jetson family	First choice	Fallback
vLLM / SGLang	Thor	NVFP4 when the model/runtime supports it	W4A16
vLLM / SGLang	Orin Nano / NX	W4A16	AWQ or GPTQ 4-bit
vLLM / SGLang	AGX Orin	W4A16	AWQ or GPTQ 4-bit
llama.cpp / Ollama	Orin and Thor	GGUF INT4 / Q4_K_M	Smaller INT4 GGUF model if memory is tight

Do not describe GGUF Q4_K_M as W4A16/AWQ/GPTQ. Do not compare Thor NVFP4 results with Orin W4A16 results unless the output includes a

quant

field.

使用运行时特定的量化名称。vLLM和SGLang通常使用Hugging Face checkpoint，如W4A16、AWQ、GPTQ、FP16或NVFP4。llama.cpp和Ollama使用GGUF模型，因此推荐INT4/Q4_K_M风格的GGUF模型。

运行时家族	Jetson家族	首选方案	备选方案
vLLM / SGLang	Thor	模型/运行时支持的情况下使用NVFP4	W4A16
vLLM / SGLang	Orin Nano / NX	W4A16	AWQ或GPTQ 4-bit
vLLM / SGLang	AGX Orin	W4A16	AWQ或GPTQ 4-bit
llama.cpp / Ollama	Orin和Thor	GGUF INT4 / Q4_K_M	内存紧张时使用更小的INT4 GGUF模型

不要将GGUF Q4_K_M描述为W4A16/AWQ/GPTQ。除非输出包含

quant

字段，否则不要将Thor NVFP4结果与Orin W4A16结果进行比较。

Runtime command guidance

运行时命令指南

Use

recommend.py

as the source of truth for memory knobs, then place its

launch_flags

into the matching serving command. Keep the command guidance in this skill instead of separate small reference files so agents ingest one complete instruction set.

For vLLM on Orin with JetPack 7.2 / L4T r39+, use upstream vLLM 0.20+ (

vllm/vllm-openai:latest

). On older Orin releases, use the NVIDIA-AI-IOT image:

bash

docker run --rm -it --runtime nvidia --network host --name vllm \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  -e HF_TOKEN="$HF_TOKEN" \
  ghcr.io/nvidia-ai-iot/vllm:latest-jetson-orin \
  vllm serve <hf-model-id-or-local-path> \
    --host 0.0.0.0 \
    --port 8000 \
    --gpu-memory-utilization 0.60 \
    --max-model-len 4096 \
    --max-num-seqs 8 \
    --enable-prefix-caching

For vLLM on Thor, use upstream vLLM 0.20+ (

vllm/vllm-openai:latest

) unless host-native vLLM 0.20+ is already installed and validated:

bash

docker run --rm -it --runtime nvidia --network host --ipc host --name vllm \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  -e HF_TOKEN="$HF_TOKEN" \
  vllm/vllm-openai:latest \
  vllm serve <hf-model-id-or-local-path> \
    --host 0.0.0.0 \
    --port 8000 \
    --gpu-memory-utilization 0.75 \
    --max-model-len 8192 \
    --max-num-seqs 32 \
    --enable-prefix-caching

Thor vLLM note: do not judge Thor support from pre-0.20 vLLM results; upstream vLLM support starts at vLLM 0.20+.

For SGLang on Thor, use NVIDIA SGLang 26.01 (

nvcr.io/nvidia/sglang:26.01-py3

). NVIDIA SGLang 26.01 contains SGLang

0.5.5.post2

and explicitly lists Jetson Thor support. Avoid judging Thor support from older prerelease SGLang results. Avoid recommending

gpt-oss

or FP8 paths on Thor unless newer NVIDIA SGLang release notes say those known issues are fixed.

bash

docker run --rm -it --runtime nvidia --network host --ipc host --name sglang \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  -e HF_TOKEN="$HF_TOKEN" \
  nvcr.io/nvidia/sglang:26.01-py3 \
  python3 -m sglang.launch_server \
    --model-path <hf-model-id-or-local-path> \
    --host 0.0.0.0 \
    --port 8000 \
    --mem-fraction-static 0.60 \
    --max-running-requests 8

For llama.cpp, use the NVIDIA-AI-IOT llama.cpp image when available, or the

llama-server

binary from a JetPack-matched build. Start with GGUF INT4 / Q4_K_M on both Orin and Thor; choose a smaller INT4 GGUF model if the audit shows tight memory.

bash

docker run --rm -it --runtime nvidia --network host --name llama-cpp \
  -v "$PWD/models:/models:ro" \
  ghcr.io/nvidia-ai-iot/llama_cpp:latest-jetson-<orin-or-thor> \
  llama-server \
    -m /models/<model>.gguf \
    --host 0.0.0.0 \
    --port 8000 \
    -ngl 28 \
    -c 4096 \
    --no-mmap \
    --flash-attn

以

recommend.py

作为内存参数的权威来源，然后将其

launch_flags

放入对应的服务命令中。将命令指南保留在本技能中，而不是单独的参考文件，以便Agent获取完整的指令集。

对于JetPack 7.2 / L4T r39+的Orin设备，使用上游vLLM 0.20+（

vllm/vllm-openai:latest

）。对于旧版Orin版本，使用NVIDIA-AI-IOT镜像：

bash

docker run --rm -it --runtime nvidia --network host --name vllm \\
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \\
  -e HF_TOKEN="$HF_TOKEN" \\
  ghcr.io/nvidia-ai-iot/vllm:latest-jetson-orin \\
  vllm serve <hf-model-id-or-local-path> \\
    --host 0.0.0.0 \\
    --port 8000 \\
    --gpu-memory-utilization 0.60 \\
    --max-model-len 4096 \\
    --max-num-seqs 8 \\
    --enable-prefix-caching

对于Thor设备上的vLLM，使用上游vLLM 0.20+（

vllm/vllm-openai:latest

），除非已安装并验证了原生vLLM 0.20+：

bash

docker run --rm -it --runtime nvidia --network host --ipc host --name vllm \\
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \\
  -e HF_TOKEN="$HF_TOKEN" \\
  vllm/vllm-openai:latest \\
  vllm serve <hf-model-id-or-local-path> \\
    --host 0.0.0.0 \\
    --port 8000 \\
    --gpu-memory-utilization 0.75 \\
    --max-model-len 8192 \\
    --max-num-seqs 32 \\
    --enable-prefix-caching

Thor vLLM注意事项：不要根据vLLM 0.20之前的版本判断Thor支持情况；上游vLLM从0.20+开始支持Thor。

对于Thor设备上的SGLang，使用NVIDIA SGLang 26.01（

nvcr.io/nvidia/sglang:26.01-py3

）。NVIDIA SGLang 26.01包含SGLang

0.5.5.post2

，并明确列出支持Jetson Thor。不要根据旧版预发布SGLang结果判断Thor支持情况。除非更新的NVIDIA SGLang发行说明表明已知问题已修复，否则避免在Thor上推荐

gpt-oss

或FP8路径。

bash

docker run --rm -it --runtime nvidia --network host --ipc host --name sglang \\
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \\
  -e HF_TOKEN="$HF_TOKEN" \\
  nvcr.io/nvidia/sglang:26.01-py3 \\
  python3 -m sglang.launch_server \\
    --model-path <hf-model-id-or-local-path> \\
    --host 0.0.0.0 \\
    --port 8000 \\
    --mem-fraction-static 0.60 \\
    --max-running-requests 8

对于llama.cpp，优先使用可用的NVIDIA-AI-IOT llama.cpp镜像，或匹配JetPack构建的

llama-server

二进制文件。在Orin和Thor上均从GGUF INT4 / Q4_K_M开始；如果审计显示内存紧张，选择更小的INT4 GGUF模型。

bash

docker run --rm -it --runtime nvidia --network host --name llama-cpp \\
  -v "$PWD/models:/models:ro" \\
  ghcr.io/nvidia-ai-iot/llama_cpp:latest-jetson-<orin-or-thor> \\
  llama-server \\
    -m /models/<model>.gguf \\
    --host 0.0.0.0 \\
    --port 8000 \\
    -ngl 28 \\
    -c 4096 \\
    --no-mmap \\
    --flash-attn

Procedure (the script encodes this)

流程（脚本已编码实现）

Pick the lightest runtime that satisfies the user's required features (continuous batching? structured generation? CPU offload?).
Pick the lowest precision that meets the user's accuracy bar (model-benchmarking skill).
Sweep the runtime's memory knobs (start with
```
gpu-memory-utilization
```
for vLLM,
```
n-gpu-layers
```
and
```
ctx-size
```
for llama.cpp) to find the minimum footprint that sustains target throughput.
Re-measure with
```
jetson-memory-audit
```
.

选择满足用户所需功能的最轻量运行时（是否支持连续批处理？结构化生成？CPU卸载？）。
选择满足用户精度要求的最低精度（模型基准测试技能）。
遍历运行时的内存参数（vLLM从
```
gpu-memory-utilization
```
开始，llama.cpp从
```
n-gpu-layers
```
和
```
ctx-size
```
开始），找到维持目标吞吐量的最小内存占用。
使用
```
jetson-memory-audit
```
重新测量。

Safety

安全说明

Read-only. The skill never starts, stops, or restarts a model server. It emits flags; the user (or an outer orchestration agent) is responsible for invoking the runtime.

只读操作。本技能从不启动、停止或重启模型服务器。仅输出参数；用户（或外层编排Agent）负责调用运行时。",

jetson-inference-mem-tune

Original

Translation

Jetson Inference Memory Tuning

Jetson推理内存调优

Purpose

用途

When to use

适用场景

Prerequisites

前置条件

Available Scripts

可用脚本

Instructions

操作步骤

Expected workflow

预期工作流

Limitations

局限性

Error handling

错误处理

Output contract for
`recommend.py`

`recommend.py`
输出约定

Runtimes covered

覆盖的运行时

Quantization recommendations

量化建议

Runtime command guidance

运行时命令指南

Procedure (the script encodes this)

流程（脚本已编码实现）

Safety

安全说明

jetson-inference-mem-tune

Original

Translation

Jetson Inference Memory Tuning

Jetson推理内存调优

Purpose

用途

When to use

适用场景

Prerequisites

前置条件

Available Scripts

可用脚本

Instructions

操作步骤

Expected workflow

预期工作流

Limitations

局限性

Error handling

错误处理

Output contract for recommend.py

recommend.py输出约定

Runtimes covered

覆盖的运行时

Quantization recommendations

量化建议

Runtime command guidance

运行时命令指南

Procedure (the script encodes this)

流程（脚本已编码实现）

Safety

安全说明

Output contract for
`recommend.py`

`recommend.py`
输出约定