deepstream-dev

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

DeepStream Development Skill

DeepStream开发技能指南

When this skill is active, ALWAYS read the relevant reference documents before generating code. Do NOT rely on memory - the reference documents contain critical details about exact property names, correct API usage, and common pitfalls.

当此技能激活时，生成代码前务必阅读相关参考文档。不要依赖记忆——参考文档包含了关于精确属性名称、正确API用法和常见陷阱的关键细节。

SDK and Architecture Quick Reference

SDK与架构快速参考

DeepStream SDK 9.0 Version Requirements

DeepStream SDK 9.0版本要求

GStreamer: 1.24.2
NVIDIA Driver: 590+
CUDA: 13.1
TensorRT: 10.14.1.48
Platforms: Ubuntu 24.04 (x86_64 and ARM64/Jetson)

GStreamer: 1.24.2
NVIDIA驱动: 590+
CUDA: 13.1
TensorRT: 10.14.1.48
平台: Ubuntu 24.04（x86_64及ARM64/Jetson）

Typical Pipeline Flow

典型流水线流程

Source → Stream Muxer → Inference → [Tracker] → OSD → Renderer

Components in

[brackets]

are optional -- only add them when the user explicitly requests them.

Stage	Role	Key Element(s)	Required?
Source	Input from files, RTSP, cameras	`nvurisrcbin` (preferred), `nvmultiurisrcbin` , `filesrc`	Yes
Stream Muxer	Batches streams for inference	`nvstreammux`	Yes
Inference	TensorRT model execution	`nvinfer` , `nvinferserver`	Yes
Tracker	Multi-object tracking across frames	`nvtracker`	Only if requested
OSD	Draws bounding boxes, labels, overlays	`nvosdbin`	Yes (for visualization)
Renderer	Display or save output	`nveglglessink` , `nv3dsink` , `filesink`	Yes

Source → Stream Muxer → Inference → [Tracker] → OSD → Renderer

方括号

[]

中的组件为可选——仅在用户明确要求时添加。

阶段	作用	核心元素	是否必需？
Source	从文件、RTSP、摄像头输入	`nvurisrcbin` （推荐）、 `nvmultiurisrcbin` 、 `filesrc`	是
Stream Muxer	批量处理流以进行推理	`nvstreammux`	是
Inference	TensorRT模型执行	`nvinfer` 、 `nvinferserver`	是
Tracker	跨帧多目标追踪	`nvtracker`	仅在用户要求时添加
OSD	绘制边界框、标签、叠加层	`nvosdbin`	是（用于可视化）
Renderer	显示或保存输出	`nveglglessink` 、 `nv3dsink` 、 `filesink`	是

Memory Model

内存模型

DeepStream uses NVIDIA Video Memory Manager (NVMM) for zero-copy GPU buffer transfers. Caps strings use

memory:NVMM

to indicate GPU memory (e.g.,

video/x-raw(memory:NVMM), format=NV12

DeepStream使用NVIDIA视频内存管理器（NVMM）实现零拷贝GPU缓冲区传输。Caps字符串使用

memory:NVMM

表示GPU内存（例如：

video/x-raw(memory:NVMM), format=NV12

）。

Critical Rules

关键规则

Only Add Requested Components: Do NOT add pipeline elements the user did not ask for.
- Tracker (
  nvtracker
  ): Only add when the user explicitly requests tracking or object IDs across frames
- Secondary GIEs: Only add when the user requests classification or attribute extraction
- Analytics (
  nvdsanalytics
  ): Only add when the user requests line crossing, ROI counting, etc.
- Message broker (
  nvmsgbroker
  /
  nvmsgconv
  ): Only add when the user requests Kafka/cloud messaging
- When in doubt, build the minimal working pipeline and let the user ask for additions
Default to
nvurisrcbin
for Sources: When the user says "camera", "stream", "video", or provides a file path:
- Always use
```
nvurisrcbin
```
  -- it handles RTSP, HTTP, and local files (
```
file://
```
  ) transparently
- Only use
```
filesrc
```
  +
```
qtdemux
```
  + parser when the user explicitly needs raw file source control
- For RTSP/live sources, also set
```
live-source=1
```
  on
```
nvstreammux
```
  and
```
sync=0
```
  on the sink
- Convert local paths to URI:
```
"file://" + os.path.abspath(path)
```
Metadata Iteration: Use
```
.frame_items
```
and
```
.object_items
```
(returns iterators, NOT lists)
- NEVER use
```
len()
```
  on these - iterate to count
- Iterator can only be consumed once

Request Pad Syntax: Use

"sink_%u"

template, NEVER literal pad names

python

pipeline.link(("decoder", "mux"), ("", "sink_%u"))  # CORRECT
# pipeline.link(("decoder", "mux"), ("", "sink_0"))  # WRONG - will fail

Platform Detection for Sinks:

python

import platform
sink_type = "nv3dsink" if platform.processor() == "aarch64" else "nveglglessink"

Buffer Cloning: Always clone buffers for async processing
python
```
tensor = buffer.extract(0).clone()  # CRITICAL
```
Queue Types:
- ```
queue.Queue
```
  → Use with
```
threading.Thread
```
- ```
multiprocessing.Queue
```
  → Use with
```
multiprocessing.Process
```
- Using wrong type causes silent data loss!
nvinfer Config Format:
- YAML: Use
```
property:
```
  section (NOT
```
model:
```
  ),
```
key: value
```
  with space after colon
- INI: Use
```
[property]
```
  section,
```
key=value
```
  with equals sign
- Section MUST be named
```
property
```
nvmsgbroker is a SINK: Cannot have downstream elements - use
```
tee
```
to split pipeline

ALL Sinks Need async=0 for Tee Splits or Dynamic Sources: CRITICAL for state transitions

python

# When using tee splits OR dynamic sources, ALL sinks MUST have async=0
pipeline.add("nveglglessink", "sink", {
    "sync": 0, "qos": 0,
    "async": 0  # CRITICAL - prevents state transition deadlock
})

Symptom if missing: Pipeline stays in PAUSED state, no video displays.

Built-in Probe Attachment:
```
measure_fps_probe
```
can only be attached to processing elements (e.g.,
```
nvinfer
```
,
```
nvosdbin
```
), NOT to sink elements. Attaching to a sink raises
```
RuntimeError: Probe failure
```
.
Dynamic ONNX Models Require
infer-dims
: When the ONNX model has dynamic input shapes (e.g., exported with
```
dynamic=True
```
in Ultralytics YOLO, or with dynamic batch/height/width axes), you MUST add
```
infer-dims=C;H;W
```
to the nvinfer config. Without it, TensorRT sees
```
-1
```
for dynamic dimensions and fails with
```
setDimensions: Error Code 3
```
. Common values:
- YOLO models (640 input):
```
infer-dims=3;640;640
```
- Models with 416 input:
```
infer-dims=3;416;416
```
- Models with 1280 input:
```
infer-dims=3;1280;1280
```
Ultralytics YOLO Output Format Depends on Model Generation — newer models (v10+/v26+) output post-NMS results; older models (v8/v11) output raw pre-NMS tensors. The custom parser and
```
cluster-mode
```
must match the actual output:

Model generation	Output tensor shape	Fields	`cluster-mode`
v8 / v11	`[batch, 84, 8400]`	`[features(4+80), anchors]` — raw cx/cy/w/h + class scores, no NMS	`2` (NMS)
v10 / v26+	`[batch, 300, 6]`	`[max_det, (x1,y1,x2,y2,conf,cls)]` — already post-NMS, pixel coords	`4` (none)

How to identify at runtime: log

inferDims.d[0]

and

inferDims.d[1]

inside the custom parser.

```
d={84, 8400}
```
→ pre-NMS (v8/v11 style)
```
d={300, 6}
```
→ post-NMS (v10/v26+ style)

Symptom of mismatch: If

cluster-mode: 2

is used with a post-NMS

[N, 6]

output, bounding boxes appear shifted by 45° or 135° from the actual objects (DeepStream's NMS incorrectly re-processes already-final coordinates). If you see tilted or rotated boxes, also check the OBB /

rotation_angle

note in

references/nvinfer_config.md

: for non-OBB models, value-initialize

NvDsInferObjectDetectionInfo

with

obj{}

and keep

rotation_angle = 0

; plain

NvDsInferObjectDetectionInfo obj;

leaves fields uninitialized.

Virtual Environment Must Include pyservicemaker:
```
pyservicemaker
```
is installed system-wide but is NOT accessible from a standard Python virtual environment. When a task requires a venv (e.g., for model download/conversion pip dependencies), always install
pyservicemaker
and
pyyaml
inside the venv. The venv setup in generated code and README must always include:
bash
```
python3 -m venv venv
source venv/bin/activate
pip install /opt/nvidia/deepstream/deepstream/service-maker/python/pyservicemaker*.whl pyyaml
pip install -r requirements.txt  # other dependencies
```
Symptom if missing:
```
ModuleNotFoundError: No module named 'pyservicemaker'
```
when running the app inside the venv.

仅添加用户请求的组件：不要添加用户未要求的流水线元素。
- Tracker (
  nvtracker
  )：仅在用户明确要求追踪或跨帧目标ID时添加
- Secondary GIEs：仅在用户要求分类或属性提取时添加
- Analytics (
  nvdsanalytics
  )：仅在用户要求越线检测、ROI计数等功能时添加
- 消息中间件 (
  nvmsgbroker
  /
  nvmsgconv
  )：仅在用户要求Kafka/云消息传递时添加
- 如有疑问，先构建最小可用流水线，让用户提出额外需求
默认使用
nvurisrcbin
作为源：当用户提到“摄像头”、“流”、“视频”或提供文件路径时：
- 始终使用
```
nvurisrcbin
```
  ——它可透明处理RTSP、HTTP和本地文件（
```
file://
```
  ）
- 仅在用户明确需要原始文件源控制时使用
```
filesrc
```
  +
```
qtdemux
```
  + 解析器
- 对于RTSP/实时源，需在
```
nvstreammux
```
  上设置
```
live-source=1
```
  ，在sink上设置
```
sync=0
```
- 将本地路径转换为URI：
```
"file://" + os.path.abspath(path)
```
元数据迭代：使用
```
.frame_items
```
和
```
.object_items
```
（返回迭代器，而非列表）
- 切勿对这些对象使用
```
len()
```
  ——通过迭代来计数
- 迭代器仅能被消费一次

请求Pad语法：使用

"sink_%u"

模板，切勿使用字面Pad名称

python

pipeline.link(("decoder", "mux"), ("", "sink_%u"))  # 正确
# pipeline.link(("decoder", "mux"), ("", "sink_0"))  # 错误 - 会失败

针对Sink的平台检测:

python

import platform
sink_type = "nv3dsink" if platform.processor() == "aarch64" else "nveglglessink"

缓冲区克隆：异步处理时务必克隆缓冲区
python
```
tensor = buffer.extract(0).clone()  # 关键步骤
```
队列类型:
- ```
queue.Queue
```
  → 与
```
threading.Thread
```
  配合使用
- ```
multiprocessing.Queue
```
  → 与
```
multiprocessing.Process
```
  配合使用
- 使用错误类型会导致静默数据丢失！
nvinfer配置格式:
- YAML：使用
```
property:
```
  节（而非
```
model:
```
  ），采用
```
key: value
```
  格式（冒号后需加空格）
- INI：使用
```
[property]
```
  节，采用
```
key=value
```
  格式（使用等号）
- 节名称必须为
```
property
```
nvmsgbroker是Sink：不能有下游元素——使用
```
tee
```
拆分流水线

所有Sink在Tee拆分或动态源场景下需设置async=0：这对状态转换至关重要

python

# 当使用tee拆分或动态源时，所有Sink必须设置async=0
pipeline.add("nveglglessink", "sink", {
    "sync": 0, "qos": 0,
    "async": 0  # 关键 - 防止状态转换死锁
})

缺失时的症状：流水线停留在PAUSED状态，无视频显示。

内置探针挂载限制：
```
measure_fps_probe
```
仅能挂载到处理元素（如
```
nvinfer
```
、
```
nvosdbin
```
），不能挂载到sink元素。挂载到sink会引发
```
RuntimeError: Probe failure
```
。
动态ONNX模型需要
infer-dims
：当ONNX模型具有动态输入形状时（例如，在Ultralytics YOLO中使用
```
dynamic=True
```
导出，或具有动态批次/高度/宽度轴），必须在nvinfer配置中添加
```
infer-dims=C;H;W
```
。否则，TensorRT会将动态维度识别为
```
-1
```
，并因
```
setDimensions: Error Code 3
```
失败。常见值：
- YOLO模型（640输入）：
```
infer-dims=3;640;640
```
- 416输入模型：
```
infer-dims=3;416;416
```
- 1280输入模型：
```
infer-dims=3;1280;1280
```
Ultralytics YOLO输出格式取决于模型版本 —— 新版本模型（v10+/v26+）输出NMS后结果；旧版本模型（v8/v11）输出原始NMS前张量。自定义解析器和
```
cluster-mode
```
必须与实际输出匹配：

模型版本	输出张量形状	字段	`cluster-mode`
v8 / v11	`[batch, 84, 8400]`	`[features(4+80), anchors]` —— 原始cx/cy/w/h + 类别分数，未经过NMS	`2` （NMS）
v10 / v26+	`[batch, 300, 6]`	`[max_det, (x1,y1,x2,y2,conf,cls)]` —— 已完成NMS，像素坐标	`4` （无）

运行时识别方法：在自定义解析器中记录

inferDims.d[0]

和

inferDims.d[1]

。

```
d={84, 8400}
```
→ NMS前（v8/v11风格）
```
d={300, 6}
```
→ NMS后（v10/v26+风格）

不匹配时的症状：如果对NMS后的

[N, 6]

输出使用

cluster-mode: 2

，边界框会相对于实际对象偏移45°或135°（DeepStream的NMS错误地重新处理已最终确定的坐标）。如果看到倾斜或旋转的框，还需检查

references/nvinfer_config.md

中的OBB /

rotation_angle

说明：对于非OBB模型，需用

obj{}

初始化

NvDsInferObjectDetectionInfo

并保持

rotation_angle = 0

；直接声明

NvDsInferObjectDetectionInfo obj;

会导致字段未初始化。

虚拟环境必须包含pyservicemaker：
```
pyservicemaker
```
是系统级安装的，但无法从标准Python虚拟环境访问。当任务需要虚拟环境时（例如，用于模型下载/转换的pip依赖），务必在虚拟环境中安装
pyservicemaker
和
pyyaml
。生成代码和README中的虚拟环境设置必须包含：
bash
```
python3 -m venv venv
source venv/bin/activate
pip install /opt/nvidia/deepstream/deepstream/service-maker/python/pyservicemaker*.whl pyyaml
pip install -r requirements.txt  # 其他依赖
```
缺失时的症状：在虚拟环境中运行应用时出现
```
ModuleNotFoundError: No module named 'pyservicemaker'
```
。

Key Paths (DeepStream 9.0)

关键路径（DeepStream 9.0）

Models:

/opt/nvidia/deepstream/deepstream/samples/models/

Primary Detector:

/opt/nvidia/deepstream/deepstream/samples/models/Primary_Detector/resnet18_trafficcamnet_pruned.onnx

Tracker lib:

/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so

Kafka lib:

/opt/nvidia/deepstream/deepstream/lib/libnvds_kafka_proto.so

Sample configs:

/opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/

模型：

/opt/nvidia/deepstream/deepstream/samples/models/

主检测器：

/opt/nvidia/deepstream/deepstream/samples/models/Primary_Detector/resnet18_trafficcamnet_pruned.onnx

追踪器库：

/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so

Kafka库：

/opt/nvidia/deepstream/deepstream/lib/libnvds_kafka_proto.so

示例配置：

/opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/

Reference Documents

参考文档

IMPORTANT: Always read these documents for complete details. Do NOT generate code from memory.

Document	Use When
references/gstreamer_plugins.md	Looking up plugin properties, ALL properties listed
references/service_maker_api.md	Using Pipeline/Flow API, metadata access, probes, EventMessageUserMetadata
references/use_cases_pipelines.md	Building pipelines: simple playback, multi-inference, cascaded GIE
references/kafka_messaging.md	Kafka/message broker setup, nvmsgconv/nvmsgbroker config, msg2p-newapi
references/best_practices.md	Design patterns, common pitfalls, anti-patterns
references/buffer_apis.md	BufferProvider/Feeder (injection), BufferRetriever/Receiver (extraction)
references/media_extractor_advanced.md	MediaExtractor, MediaChunk, FrameSampler
references/utilities_config.md	PerfMonitor, EngineFileMonitor, SourceConfig, SensorInfo, SmartRecordConfig
references/nvinfer_config.md	nvinfer config file format, ALL parameters
references/tracker_config.md	nvtracker config, NvDCF/IOU/DeepSORT/NvSORT
references/troubleshooting.md	Error messages and solutions
references/rest_api_dynamic.md	REST API, dynamic source add/remove, nvmultiurisrcbin
references/metamux_config.md	nvdsmetamux config, parallel multi-model inference, metadata merging, source ID filtering
references/docker_containers.md	Docker images, Dockerfile examples, pyservicemaker install, container run commands

重要提示：如需完整细节，请务必阅读这些文档。不要仅凭记忆生成代码。

文档	使用场景
references/gstreamer_plugins.md	查询插件属性，所有属性均已列出
references/service_maker_api.md	使用Pipeline/Flow API、元数据访问、探针、EventMessageUserMetadata
references/use_cases_pipelines.md	构建流水线：简单播放、多推理、级联GIE
references/kafka_messaging.md	Kafka/消息中间件设置、nvmsgconv/nvmsgbroker配置、msg2p-newapi
references/best_practices.md	设计模式、常见陷阱、反模式
references/buffer_apis.md	BufferProvider/Feeder（注入）、BufferRetriever/Receiver（提取）
references/media_extractor_advanced.md	MediaExtractor、MediaChunk、FrameSampler
references/utilities_config.md	PerfMonitor、EngineFileMonitor、SourceConfig、SensorInfo、SmartRecordConfig
references/nvinfer_config.md	nvinfer配置文件格式、所有参数
references/tracker_config.md	nvtracker配置、NvDCF/IOU/DeepSORT/NvSORT
references/troubleshooting.md	错误消息及解决方案
references/rest_api_dynamic.md	REST API、动态源添加/移除、nvmultiurisrcbin
references/metamux_config.md	nvdsmetamux配置、并行多模型推理、元数据合并、源ID过滤
references/docker_containers.md	Docker镜像、Dockerfile示例、pyservicemaker安装、容器运行命令

Quick Error Reference

快速错误参考

Error	Solution
`iterator has no len()`	Iterate to count, don't use `len()`
`pad template not found`	Use `"sink_%u"` not `"sink_0"`
Queue data loss	Use `multiprocessing.Queue` with `Process`
Config parse failed	Use `property:` not `model:` in YAML
`is-classifier` deprecation warning	Use `network-type: 1` instead of `is-classifier: 1` for classifiers; omit both for detectors
`min-boxes` unknown key warning	Use `minBoxes` (camelCase) in `class-attrs-*` sections, not `min-boxes`
Secondary GIE inactive	Set `process-mode: 2` , check `operate-on-gie-id`
Tee/dynamic source stuck PAUSED	Set `async: 0` on ALL sink elements
RTSP no data/reconnecting	Test URL with ffplay, check credentials
`RuntimeError: Probe failure`	`measure_fps_probe` cannot attach to sink elements; use `nvinfer` or `nvosdbin` instead
`setDimensions` negative dims / engine build failed	Add `infer-dims=C;H;W` for dynamic ONNX models (e.g., `infer-dims=3;640;640` )
`No module named 'pyservicemaker'` in venv	`pip install /opt/nvidia/deepstream/deepstream/service-maker/python/pyservicemaker*.whl pyyaml` inside the venv
`AttributeError: object has no attribute 'obj_label'`	Use `obj_meta.label` not `obj_meta.obj_label` in pyservicemaker (C API name differs from Python binding)

错误	解决方案
`iterator has no len()`	通过迭代计数，不要使用 `len()`
`pad template not found`	使用 `"sink_%u"` 而非 `"sink_0"`
队列数据丢失	将 `multiprocessing.Queue` 与 `Process` 配合使用
配置解析失败	在YAML中使用 `property:` 而非 `model:`
`is-classifier` 弃用警告	对于分类器，使用 `network-type: 1` 替代 `is-classifier: 1` ；检测器则两者都省略
`min-boxes` 未知键警告	在 `class-attrs-*` 节中使用 `minBoxes` （驼峰式），而非 `min-boxes`
Secondary GIE未激活	设置 `process-mode: 2` ，检查 `operate-on-gie-id`
Tee/动态源停留在PAUSED状态	在所有sink元素上设置 `async: 0`
RTSP无数据/重连	使用ffplay测试URL，检查凭据
`RuntimeError: Probe failure`	`measure_fps_probe` 无法挂载到sink元素；改用 `nvinfer` 或 `nvosdbin`
`setDimensions` 负维度/引擎构建失败	为动态ONNX模型添加 `infer-dims=C;H;W` （例如： `infer-dims=3;640;640` ）
虚拟环境中出现 `No module named 'pyservicemaker'`	在虚拟环境中执行 `pip install /opt/nvidia/deepstream/deepstream/service-maker/python/pyservicemaker*.whl pyyaml`
`AttributeError: object has no attribute 'obj_label'`	在pyservicemaker中使用 `obj_meta.label` 而非 `obj_meta.obj_label` （C API名称与Python绑定不同）