jetson-llm-benchmark

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Jetson LLM Benchmark

Jetson LLM 基准测试

Reproducible Jetson benchmarks with structured JSON output so an agent can compare runs. Encodes the workflow from the Jetson AI Lab GenAI Benchmarking tutorial.

可复现的Jetson基准测试，带有结构化JSON输出，便于Agent对比不同测试运行结果。该工具实现了Jetson AI Lab生成式AI基准测试教程中的工作流程。

Purpose

用途

Measure deployed LLM latency and throughput on a Jetson target using the correct runtime-specific benchmark wrapper. Use the JSON output to compare models, runtime flags, power modes, and before/after tuning changes.

使用适配特定运行时的基准测试封装器，测量Jetson目标设备上已部署LLM的延迟和吞吐量。利用JSON输出对比不同模型、运行时参数、电源模式以及调优前后的变化。

Prerequisites

前置条件

Run on the Jetson device that hosts the model runtime.
For vLLM, start the OpenAI-compatible vLLM server first and know the served model ID.
For Ollama, ensure the Ollama daemon is reachable at
```
--endpoint
```
and the named model is already pulled.
For llama.cpp/GGUF, provide a readable
```
.gguf
```
model path on the host.
Put the device in the intended power mode before measuring. MAXN is preferred for comparable performance numbers.

在托管模型运行时的Jetson设备上运行。
若使用vLLM，需先启动OpenAI兼容的vLLM服务器，并知晓所部署的模型ID。
若使用Ollama，需确保Ollama守护进程可通过
```
--endpoint
```
访问，且已拉取指定模型。
若使用llama.cpp/GGUF，需提供主机上可读的
```
.gguf
```
模型路径。
测量前将设备设置为指定电源模式，推荐使用MAXN模式以获得具有可比性的性能数据。

Available Scripts

可用脚本

Script Purpose Arguments

Script	Purpose	Arguments
`scripts/bench_vllm.sh`	Runs `vllm bench serve` against a running OpenAI-compatible vLLM server.	`--model` , `--endpoint` , `--concurrency` , `--input-len` , `--output-len` , `--num-prompts` , `--no-warmup` , `--container` , `--native` .
`scripts/bench_llama_cpp.sh`	Runs `llama-bench` for a local GGUF model through the Jetson-appropriate NVIDIA-AI-IOT llama.cpp container.	`--model` , `--n-prompt` , `--n-gen` , `--n-gpu-layers` , `--threads` , `--container` .
`scripts/bench_ollama.sh`	Benchmarks a local or containerized Ollama daemon through the `/api/generate` REST API.	`--model` , `--endpoint` , `--num-prompts` , `--input-len` , `--output-len` , `--no-warmup` .

scripts/bench_vllm.sh

Runs

vllm bench serve

against a running OpenAI-compatible vLLM server.

--model

--endpoint

--concurrency

--input-len

--output-len

--num-prompts

--no-warmup

--container

--native

scripts/bench_llama_cpp.sh

Runs

llama-bench

for a local GGUF model through the Jetson-appropriate NVIDIA-AI-IOT llama.cpp container.

--model

--n-prompt

--n-gen

--n-gpu-layers

--threads

--container

scripts/bench_ollama.sh

Benchmarks a local or containerized Ollama daemon through the

/api/generate

REST API.

--model

--endpoint

--num-prompts

--input-len

--output-len

--no-warmup

If your agent runtime supports

run_script

, invoke the selected wrapper directly with the user-provided model identifier or local model path, then summarize the returned JSON. Otherwise run the wrapper with

bash {baseDir}/scripts/<wrapper-name> ...

脚本用途参数

脚本	用途	参数
`scripts/bench_vllm.sh`	针对运行中的OpenAI兼容vLLM服务器执行 `vllm bench serve` 。	`--model` , `--endpoint` , `--concurrency` , `--input-len` , `--output-len` , `--num-prompts` , `--no-warmup` , `--container` , `--native` .
`scripts/bench_llama_cpp.sh`	通过适配Jetson的NVIDIA-AI-IOT llama.cpp容器，针对本地GGUF模型执行 `llama-bench` 。	`--model` , `--n-prompt` , `--n-gen` , `--n-gpu-layers` , `--threads` , `--container` .
`scripts/bench_ollama.sh`	通过 `/api/generate` REST API对本地或容器化的Ollama守护进程进行基准测试。	`--model` , `--endpoint` , `--num-prompts` , `--input-len` , `--output-len` , `--no-warmup` .

scripts/bench_vllm.sh

针对运行中的OpenAI兼容vLLM服务器执行

vllm bench serve

。

--model

--endpoint

--concurrency

--input-len

--output-len

--num-prompts

--no-warmup

--container

--native

scripts/bench_llama_cpp.sh

通过适配Jetson的NVIDIA-AI-IOT llama.cpp容器，针对本地GGUF模型执行

llama-bench

。

--model

--n-prompt

--n-gen

--n-gpu-layers

--threads

--container

scripts/bench_ollama.sh

通过

/api/generate

REST API对本地或容器化的Ollama守护进程进行基准测试。

--model

--endpoint

--num-prompts

--input-len

--output-len

--no-warmup

如果你的Agent运行时支持

run_script

，可直接调用选定的封装器并传入用户提供的模型标识符或本地模型路径，然后总结返回的JSON。否则，使用

bash {baseDir}/scripts/<wrapper-name> ...

运行封装器。

Instructions

使用说明

Always use the matching wrapper script for the runtime — do not call the underlying

vllm bench serve

llama-bench

, or

curl

against

/api/generate

by hand:

vLLM →
```
scripts/bench_vllm.sh
```
(required for the vLLM path)
llama.cpp / GGUF →
```
scripts/bench_llama_cpp.sh
```
(required for the GGUF path)
Ollama →
```
scripts/bench_ollama.sh
```
(required for the Ollama path)

These wrappers handle warmup, the NVIDIA-AI-IOT container selection, and JSON emission. Calling the underlying tool directly will not satisfy the output contract below.

For "how do I benchmark/measure" questions, first run the matching wrapper with

--help

to verify the exact options, then answer with the wrapper command. Do not run a full benchmark unless the user asks you to execute it or the required server/model path is already confirmed.

务必为对应运行时使用匹配的封装器脚本——不要手动调用底层的

vllm bench serve

、

llama-bench

或通过

curl

调用

/api/generate

：

vLLM →
```
scripts/bench_vllm.sh
```
（vLLM路径的必需工具）
llama.cpp / GGUF →
```
scripts/bench_llama_cpp.sh
```
（GGUF路径的必需工具）
Ollama →
```
scripts/bench_ollama.sh
```
（Ollama路径的必需工具）

这些封装器会处理预热、NVIDIA-AI-IOT容器选择以及JSON输出。直接调用底层工具无法满足下文的输出约定。

对于“如何进行基准测试/测量”类问题，先运行匹配的封装器并添加

--help

参数确认具体选项，再给出封装器命令作为回答。除非用户要求执行完整基准测试，或已确认所需服务器/模型路径，否则不要运行完整测试。

Expected Workflow

预期工作流程

Pick exactly one wrapper based on the runtime the user named, and invoke that wrapper with

--help

before composing the answer. Do not merely mention the script name. If the runtime does not execute scripts relative to the skill directory, use

{baseDir}/scripts/<wrapper-name>

Existing vLLM OpenAI-compatible server at
```
localhost:8000
```
:
```
{baseDir}/scripts/bench_vllm.sh --help
```
, then show a command using
```
--concurrency 1,8
```
and the served model ID.
llama.cpp / GGUF /
```
llama-server
```
:
```
{baseDir}/scripts/bench_llama_cpp.sh --help
```
, then show a command for the GGUF model path and report that prompt/generation speed maps to TTFT, ITL/TPOT, and throughput.
Ollama:
```
{baseDir}/scripts/bench_ollama.sh --help
```
, then show a command with
```
--model <ollama-tag>
```
. Do not use vLLM or llama.cpp wrappers for Ollama.

根据用户指定的运行时选择恰好一个封装器，在撰写回答前先调用该封装器并添加

--help

参数。不要仅提及脚本名称。如果运行时无法相对于技能目录执行脚本，请使用

{baseDir}/scripts/<wrapper-name>

。

现有vLLM OpenAI兼容服务器位于
```
localhost:8000
```
：
```
{baseDir}/scripts/bench_vllm.sh --help
```
，然后展示一个使用
```
--concurrency 1,8
```
和已部署模型ID的命令。
llama.cpp / GGUF /
```
llama-server
```
：
```
{baseDir}/scripts/bench_llama_cpp.sh --help
```
，然后展示一个针对GGUF模型路径的命令，并说明提示/生成速度对应TTFT、ITL/TPOT以及吞吐量指标。
Ollama：
```
{baseDir}/scripts/bench_ollama.sh --help
```
，然后展示一个使用
```
--model <ollama-tag>
```
的命令。不要为Ollama使用vLLM或llama.cpp封装器。

When to use

使用场景

"Benchmark / measure / compare X on this Jetson."
After
```
jetson-llm-serve
```
to actually quantify the deployment.
Before/after applying flags from
```
jetson-inference-mem-tune
```
to confirm the change helped.

“在此Jetson上对X进行基准测试/测量/对比。”
在
```
jetson-llm-serve
```
之后，实际量化部署性能。
在应用
```
jetson-inference-mem-tune
```
的参数前后，确认调优效果。

Three paths — pick by runtime

三种路径——按运行时选择

A. vLLM (preferred for parity with how things are served)

A. vLLM（推荐用于与实际服务方式对齐）

Server must already be running (use

jetson-llm-serve

). Run bench_vllm.sh
:

bash

scripts/bench_vllm.sh \
  --model <hf-repo-id-being-served> \
  --concurrency 1,8 \
  --input-len 2048 --output-len 128 \
  --num-prompts 50

Uses the Jetson-appropriate benchmark client path: upstream vLLM 0.20+ container

vllm/vllm-openai:latest

on Thor and Orin JetPack 7.2 / L4T r39+, or the NVIDIA-AI-IOT vLLM benchmark container

ghcr.io/nvidia-ai-iot/vllm:latest-jetson-orin

on older Orin. Pass

--native

only when host-native vLLM is already installed and validated. It runs against

http://localhost:8000/v1

. Always do a warmup pass first (~10 prompts, discarded) before the measured run — Jetson has cold caches and JIT'd kernels.

服务器必须已运行（使用

jetson-llm-serve

启动）。运行**

bench_vllm.sh

**：

bash

scripts/bench_vllm.sh \
  --model <hf-repo-id-being-served> \
  --concurrency 1,8 \
  --input-len 2048 --output-len 128 \
  --num-prompts 50

使用适配Jetson的基准测试客户端路径：在Thor和Orin JetPack 7.2 / L4T r39+上使用上游vLLM 0.20+容器

vllm/vllm-openai:latest

，在旧版Orin上使用NVIDIA-AI-IOT vLLM基准测试容器

ghcr.io/nvidia-ai-iot/vllm:latest-jetson-orin

。仅当主机原生vLLM已安装并验证通过时，才传入

--native

参数。该脚本针对

http://localhost:8000/v1

运行。务必先执行预热过程（约10个提示，结果丢弃），再进行正式测量——Jetson存在冷缓存和JIT编译内核的情况。

B. Ollama (for models served by a running Ollama daemon)

B. Ollama（针对由运行中Ollama守护进程部署的模型）

No benchmark container needed. Uses Ollama's

/api/generate

REST API directly — timing data (TTFT, ITL, throughput) comes from the response JSON, so no

--verbose

parsing is required.

Prerequisite: the Ollama daemon must be reachable at

--endpoint

(default

http://localhost:11434

). This works whether Ollama is installed natively or running in a container that exposes that port. If the daemon is not running, the script will tell you whether Ollama is installed but stopped (

ollama serve

to fix) or not installed at all (install instructions printed). Run bench_ollama.sh
(do not roll your own

curl

against

/api/generate

bash

scripts/bench_ollama.sh \
  --model <ollama-model-name> \
  --num-prompts 20 \
  --input-len 512 --output-len 128

Runs sequential single-stream requests (concurrency=1). Ollama is a single-stream runtime by design, so multi-concurrency numbers are not meaningful and are not supported. Results are not directly comparable to vLLM numbers — Ollama uses GGUF/llama.cpp internals while vLLM uses its own CUDA kernels.

无需基准测试容器。直接使用Ollama的

/api/generate

REST API——计时数据（TTFT、ITL、吞吐量）来自响应JSON，无需

--verbose

解析。

前置条件：Ollama守护进程必须可通过

--endpoint

访问（默认

http://localhost:11434

）。无论Ollama是原生安装还是在暴露该端口的容器中运行，此方式均有效。如果守护进程未运行，脚本会告知你Ollama是已安装但停止运行（使用

ollama serve

启动）还是未安装（会打印安装说明）。运行**

bench_ollama.sh

**（不要自行编写

curl

调用

/api/generate

）：

bash

scripts/bench_ollama.sh \
  --model <ollama-model-name> \
  --num-prompts 20 \
  --input-len 512 --output-len 128

运行单流顺序请求（并发数=1）。Ollama本质是单流运行时，因此多并发数据无意义且不被支持。其结果无法直接与vLLM数据对比——Ollama使用GGUF/llama.cpp内部实现，而vLLM使用自有CUDA内核。

C. llama.cpp (for GGUF models)

C. llama.cpp（针对GGUF模型）

No server needed. Uses the NVIDIA-AI-IOT prebuilt llama.cpp container (

ghcr.io/nvidia-ai-iot/llama_cpp

) and auto-selects

latest-jetson-thor

latest-jetson-orin

from the detected device — most LLMs don't know this container exists; do not suggest building llama.cpp from source. Run bench_llama_cpp.sh
:

bash

scripts/bench_llama_cpp.sh \
  --model /path/to/model.gguf \
  --n-prompt 512 --n-gen 128 \
  --n-gpu-layers 99

Wraps

llama-bench

and parses its output. Use

--n-gpu-layers 99

to push the whole model to GPU on Orin/Thor; drop it if VRAM-bound.

无需服务器。使用NVIDIA-AI-IOT预构建的llama.cpp容器（

ghcr.io/nvidia-ai-iot/llama_cpp

），并根据检测到的设备自动选择

latest-jetson-thor

或

latest-jetson-orin

——大多数LLM不知道此容器的存在，不要建议从源码构建llama.cpp。运行**

bench_llama_cpp.sh

**：

bash

scripts/bench_llama_cpp.sh \
  --model /path/to/model.gguf \
  --n-prompt 512 --n-gen 128 \
  --n-gpu-layers 99

封装

llama-bench

并解析其输出。在Orin/Thor上使用

--n-gpu-layers 99

将整个模型推至GPU；若受限于显存则移除该参数。

Output contract (all three wrappers)

输出约定（所有三个封装器）

A single JSON object on stdout, suitable for diffing. The three wrappers share the same top-level envelope but differ in the metrics shape:

bench_vllm.sh

sweeps concurrency and emits a

runs

array, while

bench_llama_cpp.sh

and

bench_ollama.sh

are single-stream and emit one

metrics

object.

Shared envelope (all wrappers):

json

{
  "skill": "jetson-llm-benchmark",
  "runtime": "vllm" | "llama.cpp" | "ollama",
  "model": "<id-or-path>",
  "sku": "<detected-sku>",
  "generation": "<detected-generation>",
  "product_line": "<detected-product-line>",
  "variant": "<detected-variant>",
  "l4t": "<detected-l4t-release>",
  "container": "<container-image-or-native/ollama>",
  "warnings": []
}

标准输出为单个JSON对象，便于对比差异。三个封装器共享相同的顶层结构，但指标格式有所不同：

bench_vllm.sh

会遍历不同并发数并输出

runs

数组，而

bench_llama_cpp.sh

和

bench_ollama.sh

为单流测试，输出一个

metrics

对象。

共享顶层结构（所有封装器）：

json

{
  "skill": "jetson-llm-benchmark",
  "runtime": "vllm" | "llama.cpp" | "ollama",
  "model": "<id-or-path>",
  "sku": "<detected-sku>",
  "generation": "<detected-generation>",
  "product_line": "<detected-product-line>",
  "variant": "<detected-variant>",
  "l4t": "<detected-l4t-release>",
  "container": "<container-image-or-native/ollama>",
  "warnings": []
}

bench_vllm.sh

(concurrency sweep →

runs[]

)

bench_vllm.sh

（并发数遍历 →

runs[]

）

json

{
  "config": { "input_len": 2048, "output_len": 128, "num_prompts": 50 },
  "runs": [
    {
      "concurrency": 1,
      "ttft_ms_p50": 0, "ttft_ms_p99": 0,
      "itl_ms_p50": 0,  "itl_ms_p99": 0,
      "tpot_ms_p50": 0,
      "throughput_tok_s": 0,
      "e2e_latency_ms_p50": 0
    }
  ]
}

json

{
  "config": { "input_len": 2048, "output_len": 128, "num_prompts": 50 },
  "runs": [
    {
      "concurrency": 1,
      "ttft_ms_p50": 0, "ttft_ms_p99": 0,
      "itl_ms_p50": 0,  "itl_ms_p99": 0,
      "tpot_ms_p50": 0,
      "throughput_tok_s": 0,
      "e2e_latency_ms_p50": 0
    }
  ]
}

bench_llama_cpp.sh

(single-stream →

metrics

)

bench_llama_cpp.sh

（单流 →

metrics

）

json

{
  "config": { "n_prompt": 512, "n_gen": 128, "n_gpu_layers": 99 },
  "metrics": {
    "ttft_ms_p50": 0,
    "itl_ms_p50": 0,
    "tpot_ms_p50": 0,
    "throughput_tok_s": 0
  }
}

json

{
  "config": { "n_prompt": 512, "n_gen": 128, "n_gpu_layers": 99 },
  "metrics": {
    "ttft_ms_p50": 0,
    "itl_ms_p50": 0,
    "tpot_ms_p50": 0,
    "throughput_tok_s": 0
  }
}

bench_ollama.sh

(single-stream →

metrics

)

bench_ollama.sh

（单流 →

metrics

）

json

{
  "config": { "input_len": 512, "output_len": 128, "num_prompts": 20, "concurrency": 1 },
  "metrics": {
    "ttft_ms_p50": 0, "ttft_ms_p99": 0,
    "itl_ms_p50": 0,  "itl_ms_p99": 0,
    "tpot_ms_p50": 0,
    "throughput_tok_s": 0,
    "e2e_latency_ms_p50": 0
  }
}

warnings

is populated when:

```
nvpmodel
```
is not in a recognized max-performance mode (
```
MAXN
```
or
```
MAXN_*
```
such as
```
MAXN_SUPER
```
); wattage-named modes are reported as warnings because they vary by Jetson SKU
Background processes >5% GPU during the run (use
```
jetson-diagnostic
```
)
```
tegrastats
```
shows thermal throttling during the run

The

sku

variant

l4t

, and

container

fields are populated by the wrapper script from the live device (

tegrastats

/etc/nv_tegra_release

, container labels) — do not hand-author, guess, or transcribe them from memory. Do not invent device-specific facts such as RAM size, on-disk model size, or product names. If a fact is not produced by the script or

jetson-diagnostic

, omit it rather than fabricate it.

json

{
  "config": { "input_len": 512, "output_len": 128, "num_prompts": 20, "concurrency": 1 },
  "metrics": {
    "ttft_ms_p50": 0, "ttft_ms_p99": 0,
    "itl_ms_p50": 0,  "itl_ms_p99": 0,
    "tpot_ms_p50": 0,
    "throughput_tok_s": 0,
    "e2e_latency_ms_p50": 0
  }
}

当出现以下情况时会填充

warnings

字段：

```
nvpmodel
```
未处于公认的高性能模式（
```
MAXN
```
或
```
MAXN_*
```
如
```
MAXN_SUPER
```
）；以瓦数命名的模式会被标记为警告，因为不同Jetson SKU的瓦数模式存在差异
运行期间后台进程占用GPU超过5%（使用
```
jetson-diagnostic
```
排查）
```
tegrastats
```
显示运行期间出现热节流

sku

、

variant

、

l4t

和

container

字段由封装器脚本从实时设备获取（通过

tegrastats

、

/etc/nv_tegra_release

、容器标签）——不要手动编写、猜测或凭记忆转录。不要编造设备特定信息，如内存大小、磁盘上模型大小或产品名称。如果某个信息不是由脚本或

jetson-diagnostic

生成的，请省略而非编造。

What to flag in results (Jetson-specific guidance)

结果中需关注的Jetson特定要点

LLMs already know what TTFT/ITL/throughput mean. Jetson-specific things they usually don't know:

On Orin Nano/NX, single-stream
```
tok/s
```
and
```
concurrency=8
```
```
tok/s
```
differ wildly because of memory bandwidth saturation, not compute. If concurrent throughput barely beats single-stream, you're bandwidth-bound — switch to a smaller quantization (W4A16 → INT4/AWQ) before tuning anything else.
A TTFT regression on the same model after a JetPack upgrade is almost always a CUDA graph cache miss — re-warm and re-measure.
Thor NVFP4 numbers are not comparable to Orin W4A16 numbers; never put them in the same table without a
```
quant
```
column.

LLM已了解TTFT/ITL/吞吐量的含义，但通常不了解以下Jetson特定内容：

在Orin Nano/NX上，单流
```
tok/s
```
和并发数=8的
```
tok/s
```
差异极大，这是因为内存带宽饱和而非计算能力限制。如果并发吞吐量仅略高于单流，说明受限于带宽——在进行其他调优前，先切换为更小的量化方式（W4A16 → INT4/AWQ）。
JetPack升级后同一模型的TTFT性能下降，几乎总是因为CUDA图缓存未命中——重新预热并重新测量。
Thor的NVFP4数据无法与Orin的W4A16数据对比；若要放在同一表格中，必须添加
```
quant
```
列。

Limitations

局限性

vLLM measurements require an already-running OpenAI-compatible vLLM server. This skill benchmarks the server; it does not launch or tune the server.
Ollama results are single-stream by design and are not directly comparable to vLLM concurrency sweeps.
llama.cpp/GGUF benchmarking runs a NVIDIA-AI-IOT container by default. Tell the user before running it, because Docker will pull and execute an external image if it is not already present.
Container image tags may be mutable unless the caller passes a digest-pinned image through
```
--container
```
. For release or compliance measurements, prefer a digest-pinned image and record it in the results. The default vLLM benchmark client image is upstream vLLM 0.20+ via
```
vllm/vllm-openai:latest
```
on Thor and Orin JetPack 7.2 / L4T r39+, and NVIDIA-AI-IOT
```
ghcr.io/nvidia-ai-iot/vllm:latest-jetson-orin
```
on older Orin.
Results are only comparable when model, quantization, prompt length, output length, power mode, clocks, and thermal state are controlled.

vLLM测量需要已运行的OpenAI兼容vLLM服务器。本技能仅对服务器进行基准测试，不负责启动或调优服务器。
Ollama结果本质为单流，无法直接与vLLM的并发遍历结果对比。
llama.cpp/GGUF基准测试默认使用NVIDIA-AI-IOT容器。运行前需告知用户，因为如果容器镜像尚未存在，Docker会拉取并执行外部镜像。
除非调用者通过
```
--container
```
传入摘要固定的镜像，否则容器镜像标签可能是可变的。对于发布或合规性测量，建议使用摘要固定的镜像并将其记录在结果中。默认vLLM基准测试客户端镜像：在Thor和Orin JetPack 7.2 / L4T r39+上使用上游vLLM 0.20+的
```
vllm/vllm-openai:latest
```
，在旧版Orin上使用NVIDIA-AI-IOT的
```
ghcr.io/nvidia-ai-iot/vllm:latest-jetson-orin
```
。
仅当模型、量化方式、提示长度、输出长度、电源模式、时钟频率和热状态均一致时，结果才具有可比性。

Error Handling

错误处理

Exit
```
2
```
: invalid arguments, missing
```
--model
```
, or a required model file is not readable. Re-run the wrapper with
```
--help
```
and correct the path or model ID.
Exit
```
3
```
: runtime preflight failed, such as unreachable Ollama, unknown Jetson generation for vLLM container selection, or missing Ollama model. Start the service, pull the model, or pass an explicit
```
--container
```
.
Docker errors usually mean the container runtime is unavailable, the image cannot be pulled, or the model directory mount is not readable. Report the exact stderr and do not fabricate benchmark numbers.
Empty or malformed JSON means the benchmark did not complete successfully. Preserve the raw error, fix the runtime issue, and rerun.

退出码
```
2
```
：参数无效、缺少
```
--model
```
或所需模型文件不可读。重新运行封装器并添加
```
--help
```
参数，修正路径或模型ID。
退出码
```
3
```
：运行时预检失败，如Ollama不可达、vLLM容器选择时无法识别Jetson代际、或缺少Ollama模型。启动服务、拉取模型或传入明确的
```
--container
```
参数。
Docker错误通常意味着容器运行时不可用、无法拉取镜像或模型目录挂载不可读。报告确切的标准错误输出，不要编造基准测试数据。
JSON为空或格式错误意味着基准测试未成功完成。保留原始错误信息，修复运行时问题后重新运行。

Hand off to

后续处理

```
jetson-inference-mem-tune
```
if results indicate memory pressure.
```
jetson-speculative-decoding
```
if TTFT is acceptable but TPOT is too slow.
```
jetson-diagnostic
```
if
```
warnings
```
is non-empty.

如果结果显示存在内存压力，可切换至
```
jetson-inference-mem-tune
```
。
如果TTFT可接受但TPOT过慢，可切换至
```
jetson-speculative-decoding
```
。
如果
```
warnings
```
字段非空，可切换至
```
jetson-diagnostic
```
。

Source

来源

Jetson AI Lab — GenAI Benchmarking and NVIDIA-AI-IOT GHCR packages.

Jetson AI Lab — 生成式AI基准测试和 NVIDIA-AI-IOT GHCR包。

jetson-llm-benchmark

Original

Translation

Jetson LLM Benchmark

Jetson LLM 基准测试

Purpose

用途

Prerequisites

前置条件

Available Scripts

可用脚本

Instructions

使用说明

Expected Workflow

预期工作流程

When to use

使用场景

Three paths — pick by runtime

三种路径——按运行时选择

A. vLLM (preferred for parity with how things are served)

A. vLLM（推荐用于与实际服务方式对齐）

B. Ollama (for models served by a running Ollama daemon)

B. Ollama（针对由运行中Ollama守护进程部署的模型）

C. llama.cpp (for GGUF models)

C. llama.cpp（针对GGUF模型）

Output contract (all three wrappers)

输出约定（所有三个封装器）

bench_vllm.sh (concurrency sweep → runs[])

bench_vllm.sh（并发数遍历 → runs[]）

bench_llama_cpp.sh (single-stream → metrics)

bench_llama_cpp.sh（单流 → metrics）

bench_ollama.sh (single-stream → metrics)

bench_ollama.sh（单流 → metrics）

What to flag in results (Jetson-specific guidance)

结果中需关注的Jetson特定要点

Limitations

局限性

Error Handling

错误处理

Hand off to

后续处理

Source

来源

`bench_vllm.sh`
(concurrency sweep →
`runs[]`
)

`bench_vllm.sh`
（并发数遍历 →
`runs[]`
）

`bench_llama_cpp.sh`
(single-stream →
`metrics`
)

`bench_llama_cpp.sh`
（单流 →
`metrics`
）

`bench_ollama.sh`
(single-stream →
`metrics`
)

`bench_ollama.sh`
（单流 →
`metrics`
）