jetson-speculative-decoding

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Jetson Speculative Decoding (vLLM)

Jetson 推测解码（vLLM）

Speculative decoding lets a small "draft" model propose tokens that the target model verifies in a single forward pass, reducing per-token latency. On Jetson, the win/loss is dominated by VRAM headroom, not by the draft quality. This skill encodes the parts an LLM won't already know.

推测解码让一个小型"草稿"模型先预测token，再由目标模型在单次前向传播中验证，从而降低每个token的延迟。在Jetson平台上，效果好坏主要取决于显存余量，而非草稿模型的质量。本技能涵盖了大语言模型本身不会知晓的细节部分。

Purpose

目的

Tune an existing Jetson vLLM deployment for faster token generation by appending the right

--speculative-config

and validating whether it improves single-stream decode speed.

通过添加正确的

--speculative-config

参数并验证是否提升了单流解码速度，优化现有的Jetson vLLM部署，加快token生成速度。

When to use

适用场景

TPOT/ITL is the bottleneck (TTFT is fine, output is just slow).
Workload is single-stream or low-concurrency (≤2). Speculation usually loses at high concurrency.
Jetson family is Thor or AGX Orin. Do not suggest EAGLE-3 on Orin Nano/NX — there is rarely enough VRAM headroom to host both target and draft, and you'll OOM at startup.

TPOT/ITL是瓶颈（TTFT表现良好，但输出速度较慢）。
工作负载为单流或低并发（≤2）。推测解码在高并发场景下通常效果不佳。
Jetson系列为Thor或AGX Orin。请勿在Orin Nano/NX上使用EAGLE-3——这类设备几乎没有足够的显存同时容纳目标模型和草稿模型，启动时会出现内存不足（OOM）问题。

When NOT to use

不适用场景

High-concurrency serving (≥8): batched decode usually beats speculation; the draft model just steals VRAM.
Models without a published EAGLE-3 head — do not train one ad-hoc as a "fix".
After applying
```
jetson-inference-mem-tune
```
flags that already pushed
```
--gpu-memory-utilization
```
near the ceiling. Free at least ~2 GB first.

高并发服务（≥8）：批量解码通常优于推测解码；草稿模型只会占用显存资源。
没有已发布的EAGLE-3头部的模型——不要临时训练一个来"修复"问题。
已应用
```
jetson-inference-mem-tune
```
参数，且
```
--gpu-memory-utilization
```
已接近上限的情况。请先释放至少约2GB的显存。

Prerequisites

前置条件

A working vLLM server recipe from
```
jetson-llm-serve
```
.
Enough memory headroom for the draft model or EAGLE-3 head in addition to the target model.
A benchmark baseline from
```
jetson-llm-benchmark
```
before enabling speculation.
A target model with a compatible EAGLE-3 head, or a small same-family draft model for the fallback path.

拥有来自
```
jetson-llm-serve
```
的可用vLLM服务器配置方案。
除目标模型外，有足够的显存余量容纳草稿模型或EAGLE-3头部。
在启用推测解码前，通过
```
jetson-llm-benchmark
```
获取基准测试数据。
目标模型需具备兼容的EAGLE-3头部，或者有同系列的小型草稿模型作为备选方案。

Instructions

操作步骤

Append

--speculative-config

to the

vllm serve

command shown in

jetson-llm-serve

EAGLE-3 (preferred when a head is published for the target model):

bash

--speculative-config '{
  "method": "eagle3",
  "model": "<eagle3-head-repo-id>",
  "num_speculative_tokens": 5,
  "draft_tensor_parallel_size": 1
}'

Draft-model (fallback — pair a small same-family model):

bash

--speculative-config '{
  "method": "draft_model",
  "model": "<small-draft-model-repo-id>",
  "num_speculative_tokens": 4,
  "draft_tensor_parallel_size": 1
}'

在

jetson-llm-serve

提供的

vllm serve

命令后添加

--speculative-config

参数。

EAGLE-3（当目标模型有已发布的头部时优先使用）：

bash

--speculative-config '{
  "method": "eagle3",
  "model": "<eagle3-head-repo-id>",
  "num_speculative_tokens": 5,
  "draft_tensor_parallel_size": 1
}'

草稿模型（备选方案——搭配同系列的小型模型）：

bash

--speculative-config '{
  "method": "draft_model",
  "model": "<small-draft-model-repo-id>",
  "num_speculative_tokens": 4,
  "draft_tensor_parallel_size": 1
}'

Jetson-specific tuning rules

Jetson专属调优规则

```
num_speculative_tokens
```
: start at 5 on Thor, 3 on AGX Orin. Higher values pay off only if the draft acceptance rate is >0.6.
Always pair with the same vLLM runtime path used by
```
jetson-llm-serve
```
: upstream vLLM 0.20+ (
```
vllm/vllm-openai:latest
```
) or validated native vLLM 0.20+ on Thor, upstream vLLM 0.20+ on Orin JetPack 7.2 / L4T r39+, or the NVIDIA-AI-IOT vLLM image on older Orin. Do not use an Orin NVIDIA-AI-IOT vLLM image on Thor. Older runtimes may lack EAGLE-3 or the current
```
--speculative-config
```
shape.
Drop
```
--gpu-memory-utilization
```
by ~0.05 vs the non-speculative baseline to give the draft model headroom.

```
num_speculative_tokens
```
: start at 5 on Thor, 3 on AGX Orin. Higher values pay off only if the draft acceptance rate is >0.6.
Always pair with the same vLLM runtime path used by
```
jetson-llm-serve
```
: upstream vLLM 0.20+ (
```
vllm/vllm-openai:latest
```
) or validated native vLLM 0.20+ on Thor, upstream vLLM 0.20+ on Orin JetPack 7.2 / L4T r39+, or the NVIDIA-AI-IOT vLLM image on older Orin. Do not use an Orin NVIDIA-AI-IOT vLLM image on Thor. Older runtimes may lack EAGLE-3 or the current
```
--speculative-config
```
shape.
Drop
```
--gpu-memory-utilization
```
by ~0.05 vs the non-speculative baseline to give the draft model headroom.

How to verify it actually helped

如何验证效果提升

Run
```
jetson-llm-benchmark
```
(vLLM path) at
```
--concurrency 1
```
before and after enabling speculation.
Acceptance: target ≥30% improvement in
```
throughput_tok_s
```
and ≥20% drop in
```
tpot_ms_p50
```
at concurrency 1.
If improvement is <10%, or
```
throughput_tok_s
```
regresses at concurrency 8, disable speculation. The draft model is costing more than it returns.

Run
```
jetson-llm-benchmark
```
(vLLM path) at
```
--concurrency 1
```
before and after enabling speculation.
Acceptance: target ≥30% improvement in
```
throughput_tok_s
```
and ≥20% drop in
```
tpot_ms_p50
```
at concurrency 1.
If improvement is <10%, or
```
throughput_tok_s
```
regresses at concurrency 8, disable speculation. The draft model is costing more than it returns.

Limitations

Speculative decoding improves decode-heavy workloads; it does not reduce TTFT-dominated latency.
High concurrency can erase the benefit because continuous batching already keeps the GPU busy.
Orin Nano/NX usually lack enough memory headroom for both target and draft models.
Acceptance rate and draft overhead are model-specific, so benchmark before and after instead of assuming a speedup.

Speculative decoding improves decode-heavy workloads; it does not reduce TTFT-dominated latency.
High concurrency can erase the benefit because continuous batching already keeps the GPU busy.
Orin Nano/NX usually lack enough memory headroom for both target and draft models.
Acceptance rate and draft overhead are model-specific, so benchmark before and after instead of assuming a speedup.

Error handling

If vLLM rejects
```
--speculative-config
```
, verify that Thor and Orin JetPack 7.2 / L4T r39+ are using vLLM 0.20+ and that older Orin is using a JetPack-matched NVIDIA-AI-IOT vLLM image; then switch back to the non-speculative serving command if the runtime still rejects it.
If startup OOMs, lower
```
--gpu-memory-utilization
```
, use a smaller draft, or disable speculation and hand off to
```
jetson-inference-mem-tune
```
.
If benchmark throughput regresses, remove
```
--speculative-config
```
; a bad draft path is worse than no speculation.

If vLLM rejects
```
--speculative-config
```
, verify that Thor and Orin JetPack 7.2 / L4T r39+ are using vLLM 0.20+ and that older Orin is using a JetPack-matched NVIDIA-AI-IOT vLLM image; then switch back to the non-speculative serving command if the runtime still rejects it.
If startup OOMs, lower
```
--gpu-memory-utilization
```
, use a smaller draft, or disable speculation and hand off to
```
jetson-inference-mem-tune
```
.
If benchmark throughput regresses, remove
```
--speculative-config
```
; a bad draft path is worse than no speculation.

Hand off to

```
jetson-llm-benchmark
```
to quantify the change.
```
jetson-inference-mem-tune
```
if startup OOMs after enabling speculation.

```
jetson-llm-benchmark
```
to quantify the change.
```
jetson-inference-mem-tune
```
if startup OOMs after enabling speculation.

Source

vLLM speculative decoding docs and the Jetson AI Lab GenAI tutorial.