jetson-memory-audit

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Jetson Memory Audit

Jetson内存审计

Read-only memory-focused snapshot for a Jetson, plus the

drop_caches

verify-loop helper that confirms freed memory actually shows up as free instead of cached.

针对Jetson的只读内存聚焦快照，以及

drop_caches

验证循环工具，用于确认释放的内存确实显示为空闲状态而非缓存状态。

Purpose

用途

Measure current Jetson memory consumers, capture before/after baselines, and verify whether user-approved changes actually reclaimed DRAM. Use live device data rather than estimates from container size, model size, or generic process memory.

测量当前Jetson的内存占用情况，捕获变更前后的基准数据，并验证用户认可的变更是否真正回收了DRAM。使用设备实时数据而非容器大小、模型大小或通用进程内存的估算值。

CRITICAL: Memory appears stuck after stopping vLLM / sglang (JetPack below 7.2 / L4T below r39.0)

重要提示：停止vLLM / sglang后内存似乎被占用（JetPack低于7.2 / L4T低于r39.0）

This is the most common memory confusion on Jetson releases before JetPack 7.2 or before L4T r39.0.

After you stop a vLLM, sglang, or Ollama server (or any CUDA workload), the memory shown as free by

free -h

tegrastats

may not recover — even though the process is gone.

nvidia-smi

may also show misleadingly low free GPU memory.

Root cause: The Thor RM (resource manager) holds freed sysmem pages in its own pool after a CUDA context exits. On Unified Memory Architecture (UMA) devices like Jetson,

cudaMemGetInfo

reads RM pool state and reports far less free memory than is actually available to a new process.

Workaround (for JetPack below 7.2 or L4T below r39.0):

bash

sudo sync && sudo sysctl -w vm.drop_caches=3

Run this on the host, not inside a container. The important operation is

sudo sysctl -w vm.drop_caches=3

; keep

sudo sync

immediately before it so dirty data is flushed before reclaimable page/dentry/inode caches are dropped. After running it,

free -h

and

tegrastats

will reflect the true available memory.

For affected releases, recommend this command when a user says:

"Memory didn't free after I stopped vLLM/sglang"
"Why does tegrastats still show high usage after my container exited?"
"OOM even though nothing is running"
"Memory was fine yesterday but now it's full"

On JetPack below 7.2 or L4T below r39.0,

drop_caches

is the reliable workaround when memory appears stuck after a CUDA workload exits; on newer releases, use it only if the same symptom is observed and the user approves.

这是JetPack 7.2或L4T r39.0之前版本的Jetson上最常见的内存混淆问题。

停止vLLM、sglang或Ollama服务器（或任何CUDA工作负载）后，

free -h

或

tegrastats

显示的空闲内存可能无法恢复——即使进程已经结束。

nvidia-smi

也可能显示误导性的低空闲GPU内存。

根本原因： 在CUDA上下文退出后，Thor RM（资源管理器）会将释放的系统内存页保留在自己的池中。在Jetson这类统一内存架构（UMA）设备上，

cudaMemGetInfo

会读取RM池状态，报告的空闲内存远低于新进程实际可用的内存。

解决方法（适用于JetPack低于7.2或L4T低于r39.0的版本）：

bash

sudo sync && sudo sysctl -w vm.drop_caches=3

在主机上运行此命令，不要在容器内运行。关键操作是

sudo sysctl -w vm.drop_caches=3

；需在其之前立即执行

sudo sync

，以便在释放可回收的页/目录项/索引节点缓存前刷新脏数据。运行后，

free -h

和

tegrastats

将反映真实的可用内存。

对于受影响的版本，当用户提出以下问题时推荐此命令：

"停止vLLM/sglang后内存没有释放"
"为什么我的容器退出后tegrastats仍显示高内存占用？"
"明明没有进程在运行却出现OOM（内存不足）"
"昨天内存还正常，现在就满了"

在JetPack低于7.2或L4T低于r39.0的版本中，当CUDA工作负载退出后内存似乎被占用时，

drop_caches

是可靠的解决方法；在新版本中，仅当观察到相同症状且用户认可时才使用它。

When to use

使用场景

"How much memory is in use on this Jetson? What's holding it?"
"I disabled the GUI / stopped vLLM / quit my container — did memory actually free?"
"Why does
```
free -h
```
still show low free memory after I stopped my workload?"
As the baseline before applying
```
jetson-headless-mode
```
or other memory-related changes, and again after to compute the actual delta.

"这台Jetson的内存使用情况如何？是什么在占用内存？"
"我禁用了GUI / 停止了vLLM / 退出了容器——内存真的释放了吗？"
"为什么停止工作负载后
```
free -h
```
仍显示低空闲内存？"
作为应用
```
jetson-headless-mode
```
或其他内存相关变更前的基准，并在变更后再次使用以计算实际差值。

Prerequisites

前提条件

Run on the Jetson host, or in a sandbox/container with host-visible
```
/proc
```
,
```
/etc/nv_tegra_release
```
,
```
tegrastats
```
, and process data.
NvMap debugfs reads may require root. If unavailable, report that GPU memory attribution is limited rather than guessing.
```
drop_caches.sh
```
requires root or passwordless
```
sudo -n
```
; run it only after the user explicitly authorizes cache dropping.

在Jetson主机上运行，或在可查看主机
```
/proc
```
、
```
/etc/nv_tegra_release
```
、
```
tegrastats
```
和进程数据的沙箱/容器中运行。
读取NvMap debugfs可能需要root权限。如果无法访问，需报告GPU内存归因受限，而非猜测。
```
drop_caches.sh
```
需要root权限或无密码的
```
sudo -n
```
；仅在用户明确授权清除缓存后运行。

Available Scripts

可用脚本

Script	Purpose	Arguments
`scripts/audit.sh`	Emits a JSON snapshot from `jetson-diagnostic/scripts/snapshot.sh` for memory audit workflows.	No arguments.
`scripts/drop_caches.sh`	Flushes reclaimable page/dentry/inode caches and prints before/after memory deltas.	`--mode 1\|2\|3` , `--quiet` .

If your agent runtime supports

run_script

, use it to run

scripts/audit.sh

scripts/drop_caches.sh

and summarize the returned output. Otherwise run the scripts with

bash

from the repository root.

脚本	用途	参数
`scripts/audit.sh`	从 `jetson-diagnostic/scripts/snapshot.sh` 生成JSON快照，用于内存审计工作流。	无参数。
`scripts/drop_caches.sh`	刷新可回收的页/目录项/索引节点缓存，并打印内存变更前后的差值。	`--mode 1\|2\|3` , `--quiet` 。

如果你的Agent运行时支持

run_script

，使用它来运行

scripts/audit.sh

或

scripts/drop_caches.sh

并总结返回的输出。否则从仓库根目录使用

bash

运行这些脚本。

Instructions

操作说明

For "how much memory is in use right now?" questions, run

scripts/audit.sh

and report only values from the JSON snapshot.

对于“当前内存使用情况如何？”的问题，运行

scripts/audit.sh

并仅报告JSON快照中的数值。

Reporting guidance

报告指南

Do not only print or mention the path to a helper. Invoke the helper and then summarize the returned data.

For "how much memory is in use" prompts, run
```
scripts/audit.sh
```
and quote
```
mem_total_gb
```
,
```
memory_kb.available
```
, and the leading
```
procrank_top
```
process or
```
nvmap.top_clients
```
consumer.
For GUI/desktop memory prompts, run
```
scripts/audit.sh
```
and report
```
default_systemd_target
```
plus any display manager in
```
candidate_services
```
(
```
gdm3
```
,
```
gdm
```
,
```
lightdm
```
,
```
sddm
```
, or
```
display-manager
```
). Do not disable anything; hand off to
```
jetson-headless-mode
```
for a plan.
For prompts that explicitly authorize cache dropping after a stopped workload, run
```
scripts/drop_caches.sh
```
(equivalent to
```
sudo sync && sudo sysctl -w vm.drop_caches=3
```
by default) and report its before/after free, available, and cached deltas. If root is unavailable, explain that it must be run on the host with sudo.

If your agent runtime does not execute helper scripts relative to this skill directory, resolve script paths with the AgentSkills

{baseDir}

placeholder:

bash

{baseDir}/scripts/audit.sh
{baseDir}/scripts/drop_caches.sh

Do not call

jetson-memory-audit

as a tool name unless the runtime explicitly registers skills as callable tools; Agent Skills are normally instructions plus files, not direct tool functions.

Sandbox note for agents: seeing this skill file does not guarantee access to Jetson host memory data. If

/proc/device-tree/model

/etc/nv_tegra_release

tegrastats

/sys/kernel/debug/nvmap

, or host process data are missing inside a NemoClaw/OpenClaw sandbox, say the sandbox lacks Jetson host visibility and ask the user to run on the Jetson host or relaunch with a host-visible sandbox profile. Do not fabricate memory totals, available memory, PSS, NvMap, or reclamation deltas.

For "how much memory did this change free?" questions, use a before/after delta. Do not estimate freed memory from container size, image size, RSS, or a single post-change snapshot.

Before the change, run
```
scripts/audit.sh
```
and save the JSON baseline.
Make the user-approved change (stop the container, switch mode, apply a tuning recommendation, etc.).
On JetPack below 7.2 / L4T below r39.0, or when the same stuck-memory symptom is observed on a newer release, flush reclaimable page cache on the host (not inside a container) so freed pages show up as free instead of cached:
bash
```
sudo sync && sudo sysctl -w vm.drop_caches=3
```
Re-run
```
scripts/audit.sh
```
and compare
```
memory_kb.available
```
before vs after — that delta is the real reclamation.

If the user already made the change and no baseline exists, say that the exact freed amount cannot be recovered from the current snapshot alone. Capture a new baseline now so the next change can be measured.

Use live audit data as the source of truth. Memory totals, available memory, NvMap totals, PSS values, display-manager state, and savings deltas must come from

scripts/audit.sh

free -h

, or

tegrastats

on the actual device. If a number is not present in those outputs, do not guess it.

不要仅打印或提及工具的路径。调用工具后总结返回的数据。

对于“内存使用情况”的请求，运行
```
scripts/audit.sh
```
并引用
```
mem_total_gb
```
、
```
memory_kb.available
```
以及排名靠前的
```
procrank_top
```
进程或
```
nvmap.top_clients
```
占用者。
对于GUI/桌面内存相关请求，运行
```
scripts/audit.sh
```
并报告
```
default_systemd_target
```
以及
```
candidate_services
```
中的任何显示管理器（
```
gdm3
```
、
```
gdm
```
、
```
lightdm
```
、
```
sddm
```
或
```
display-manager
```
）。不要禁用任何服务；将任务转交给
```
jetson-headless-mode
```
以获取解决方案。
对于停止工作负载后明确授权清除缓存的请求，运行
```
drop_caches.sh
```
（默认等效于
```
sudo sync && sudo sysctl -w vm.drop_caches=3
```
）并报告其内存变更前后的空闲、可用和缓存差值。如果没有root权限，说明必须在主机上使用sudo运行。

如果你的Agent运行时不相对于此技能目录执行辅助脚本，请使用AgentSkills的

{baseDir}

占位符解析脚本路径：

bash

{baseDir}/scripts/audit.sh
{baseDir}/scripts/drop_caches.sh

除非运行时明确将技能注册为可调用工具，否则不要将

jetson-memory-audit

作为工具名称调用；Agent Skills通常是说明文档加文件，而非直接的工具函数。

Agent沙箱注意事项：看到此技能文件并不保证能访问Jetson主机内存数据。如果NemoClaw/OpenClaw沙箱中缺少

/proc/device-tree/model

、

/etc/nv_tegra_release

、

tegrastats

、

/sys/kernel/debug/nvmap

或主机进程数据，请说明沙箱缺乏Jetson主机可见性，并要求用户在Jetson主机上运行或使用主机可见的沙箱配置重新启动。不要编造内存总量、可用内存、PSS、NvMap或回收差值。

对于“此变更释放了多少内存？”的问题，使用变更前后的差值。不要从容器大小、镜像大小、RSS或单一变更后快照估算释放的内存。

变更前，运行
```
scripts/audit.sh
```
并保存JSON基准数据。
执行用户认可的变更（停止容器、切换模式、应用调优建议等）。
在JetPack低于7.2 / L4T低于r39.0的版本中，或在新版本中观察到相同的内存占用症状时，在主机（而非容器内）刷新可回收页缓存，使释放的页显示为空闲而非缓存：
bash
```
sudo sync && sudo sysctl -w vm.drop_caches=3
```
重新运行
```
scripts/audit.sh
```
并比较变更前后的
```
memory_kb.available
```
——该差值即为实际回收的内存。

如果用户已完成变更但没有基准数据，说明无法仅从当前快照恢复确切的释放量。立即捕获新的基准数据，以便测量下一次变更。

以实时审计数据为唯一依据。内存总量、可用内存、NvMap总量、PSS值、显示管理器状态和节省差值必须来自实际设备上的

scripts/audit.sh

、

free -h

或

tegrastats

。如果这些输出中没有相关数值，不要猜测。

Output contract for

audit.sh

audit.sh

的输出约定

json

{
  "sku": "orin-nano",
  "variant": "orin-nano-8gb",
  "mem_total_gb": 8,
  "l4t_version": "36.4.0",
  "product_model": "nvidia jetson orin nano developer kit",
  "memory_kb": { "total": 8123456, "available": 4123456, "free": 1023456, "cached": 1234567, "swap_total": 0, "swap_free": 0 },
  "default_systemd_target": "graphical.target",
  "candidate_services": { "gdm3": { "active": "active", "enabled": "enabled" } },
  "tegrastats_sample": "RAM 4011/8138MB (lfb 8x4MB) ...",
  "nvmap": { "readable": false, "total_kb": 0, "top_clients": [] },
  "procrank_top": [ { "pid": 4321, "pss_kb": 4000000, "cmd": "vllm" } ]
}

json

{
  "sku": "orin-nano",
  "variant": "orin-nano-8gb",
  "mem_total_gb": 8,
  "l4t_version": "36.4.0",
  "product_model": "nvidia jetson orin nano developer kit",
  "memory_kb": { "total": 8123456, "available": 4123456, "free": 1023456, "cached": 1234567, "swap_total": 0, "swap_free": 0 },
  "default_systemd_target": "graphical.target",
  "candidate_services": { "gdm3": { "active": "active", "enabled": "enabled" } },
  "tegrastats_sample": "RAM 4011/8138MB (lfb 8x4MB) ...",
  "nvmap": { "readable": false, "total_kb": 0, "top_clients": [] },
  "procrank_top": [ { "pid": 4321, "pss_kb": 4000000, "cmd": "vllm" } ]
}

Limitations

局限性

Exact freed-memory deltas require a before snapshot, the user-approved change, cache flush when appropriate, and an after snapshot.
NvMap attribution depends on host-visible debugfs access; if it is unavailable, report limited GPU memory attribution instead of guessing.
Sandbox/container runs may not see host
```
/proc
```
,
```
tegrastats
```
, systemd, or NvMap data unless the runtime exposes them.

准确的内存释放差值需要变更前的快照、用户认可的变更、适当的缓存刷新以及变更后的快照。
NvMap归因依赖于对主机debugfs的访问；如果无法访问，需报告GPU内存归因受限，而非猜测。
沙箱/容器运行可能无法看到主机
```
/proc
```
、
```
tegrastats
```
、systemd或NvMap数据，除非运行时暴露这些数据。

Error handling

错误处理

If
```
scripts/audit.sh
```
cannot access host Jetson data, report the missing visibility and ask to rerun on the Jetson host or in a host-visible sandbox.
If
```
scripts/drop_caches.sh
```
lacks root or passwordless
```
sudo -n
```
, report that cache dropping must be run on the host with sudo approval.
If no before snapshot exists, say the exact reclaimed amount cannot be recovered from the current state alone and capture a new baseline for the next change.

如果
```
scripts/audit.sh
```
无法访问Jetson主机数据，报告缺少可见性，并要求在Jetson主机上重新运行或使用主机可见的沙箱运行。
如果
```
scripts/drop_caches.sh
```
缺少root权限或无密码的
```
sudo -n
```
，报告清除缓存必须在主机上经sudo授权后运行。
如果没有变更前的快照，说明无法从当前状态恢复确切的回收量，并为下一次变更捕获新的基准数据。

Safety

安全性

Read-only.

drop_caches

is non-destructive (kernel only releases pages it could reclaim under pressure anyway;

sync

runs first to preserve dirty data).

只读操作。

drop_caches

是非破坏性的（内核仅释放其在内存压力下可回收的页；

sync

会先运行以保存脏数据）。

Hand off to

转交至

```
jetson-headless-mode
```
— biggest single user-space win on systems still booting
```
graphical.target
```
.
```
jetson-inference-mem-tune
```
— when a model server is the top NvMap / PSS consumer.
If runtime changes cannot hit the target, report that further reclamation is outside this skill's scope rather than suggesting unsafe boot-time edits.

```
jetson-headless-mode
```
——对于仍以
```
graphical.target
```
启动的系统，这是用户空间中最大的内存优化方案。
```
jetson-inference-mem-tune
```
——当模型服务器是NvMap / PSS的主要占用者时。
如果运行时变更无法达到目标，报告进一步的内存回收超出此技能的范围，而非建议不安全的启动时编辑。

jetson-memory-audit

Original

Translation

Jetson Memory Audit

Jetson内存审计

Purpose

用途

CRITICAL: Memory appears stuck after stopping vLLM / sglang (JetPack below 7.2 / L4T below r39.0)

重要提示：停止vLLM / sglang后内存似乎被占用（JetPack低于7.2 / L4T低于r39.0）

When to use

使用场景

Prerequisites

前提条件

Available Scripts

可用脚本

Instructions

操作说明

Reporting guidance

报告指南

Output contract for
`audit.sh`

`audit.sh`
的输出约定

Limitations

局限性

Error handling

错误处理

Safety

安全性

Hand off to

转交至

jetson-memory-audit

Original

Translation

Jetson Memory Audit

Jetson内存审计

Purpose

用途

CRITICAL: Memory appears stuck after stopping vLLM / sglang (JetPack below 7.2 / L4T below r39.0)

重要提示：停止vLLM / sglang后内存似乎被占用（JetPack低于7.2 / L4T低于r39.0）

When to use

使用场景

Prerequisites

前提条件

Available Scripts

可用脚本

Instructions

操作说明

Reporting guidance

报告指南

Output contract for audit.sh

audit.sh的输出约定

Limitations

局限性

Error handling

错误处理

Safety

安全性

Hand off to

转交至

Output contract for
`audit.sh`

`audit.sh`
的输出约定