deepstream-run-mv3dt

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Skill: Run DeepStream MV3DT

技能:运行DeepStream MV3DT

When to Use This Skill

何时使用本技能

Activate this skill when the user wants to set up, run, verify, or debug the DeepStream Multi-View 3D Tracking reference app. Typical prompts:
  • "set up MV3DT DeepStream"
  • "run the 4-camera MV3DT sample"
  • "run the 12-camera MV3DT sample"
  • "run MV3DT on my synchronized MP4s"
  • "I have videos but no calibration; calibrate and run MV3DT"
  • "show the BEV visualizer or Kafka metadata"
  • "stop MV3DT" / "clean up MV3DT" / "tear down MV3DT"
Do not use this skill for single-view 3D tracking, generic DeepStream app development, or live-stream onboarding unless the user explicitly maps that work to this repo's MV3DT pipeline.
当用户想要设置、运行、验证或调试DeepStream多视图3D追踪参考应用时,激活本技能。典型请求示例:
  • "设置MV3DT DeepStream"
  • "运行4相机MV3DT示例"
  • "运行12相机MV3DT示例"
  • "在我的同步MP4文件上运行MV3DT"
  • "我有视频但没有校准信息;完成校准并运行MV3DT"
  • "展示BEV可视化工具或Kafka元数据"
  • "停止MV3DT" / "清理MV3DT" / "关闭MV3DT"
请勿将本技能用于单视图3D追踪、通用DeepStream应用开发或直播入门操作,除非用户明确将相关工作映射到本仓库的MV3DT流水线。

Examples

示例场景

  • "Deploy the DeepStream MV3DT 4-camera sample and show the OSD and BEV windows."
  • "Run the MV3DT 12-camera sample headlessly with RTDETR and save videos."
  • "Run MV3DT on synchronized MP4s under /data/mv3dt-demo using PeopleNetTransformer."
  • "My custom MV3DT videos do not have calibration; use AutoMagicCalib, then run MV3DT."
  • "部署DeepStream MV3DT 4相机示例并展示OSD和BEV窗口。"
  • "以无头模式运行MV3DT 12相机示例,使用RTDETR并保存视频。"
  • "在/data/mv3dt-demo下的同步MP4文件上运行MV3DT,使用PeopleNetTransformer。"
  • "我的自定义MV3DT视频没有校准信息;使用AutoMagicCalib完成校准后运行MV3DT。"

Overview

概述

Operate the Multi-View 3D Tracking reference app in
DeepStream/src/apps/reference_apps/deepstream-tracker-3d-multi-view
using the DeepStream Container path. The skill supports setup, shipped sample runs, custom synchronized MP4 datasets, calibration handoff to AutoMagicCalib, display/headless execution, OSD/BEV outputs, and Kafka metadata inspection.
通过DeepStream容器路径,操作位于
DeepStream/src/apps/reference_apps/deepstream-tracker-3d-multi-view
中的多视图3D追踪参考应用。本技能支持环境设置、官方示例运行、自定义同步MP4数据集处理、将校准任务委托给AutoMagicCalib、显示/无头模式执行、OSD/BEV输出以及Kafka元数据查看。

Prerequisites

前置条件

  • MV3DT reference app directory on disk under
    DeepStream/src/apps/reference_apps/deepstream-tracker-3d-multi-view
    ; if it is absent, ask before cloning the public DeepStream repo
  • Docker with NVIDIA GPU support
  • DeepStream container image access
  • MV3DT sample datasets, models, custom parser libraries, Kafka, Mosquitto, and
    mv3dt_venv
    prepared by the repo setup script
  • Working X11/VNC display for live OSD/BEV windows, or the saved-output headless path for tiled DeepStream MP4 plus Kafka-derived BEV MP4
  • 磁盘上存在MV3DT参考应用目录,路径为
    DeepStream/src/apps/reference_apps/deepstream-tracker-3d-multi-view
    ;若目录不存在,需先询问用户是否同意克隆公开的DeepStream仓库
  • 支持NVIDIA GPU的Docker环境
  • 可访问DeepStream容器镜像
  • 仓库设置脚本已准备好MV3DT示例数据集、模型、自定义解析库、Kafka、Mosquitto和
    mv3dt_venv
    虚拟环境
  • 可用的X11/VNC显示环境(用于实时OSD/BEV窗口),或采用无头模式保存输出(生成拼接后的DeepStream MP4文件以及基于Kafka的BEV MP4文件)

Instructions

操作步骤

Step 0: Resolve MV3DT App Checkout

步骤0:确认MV3DT应用目录

The skill can be installed outside the DeepStream repo. Resolve
REPO_ROOT
to the MV3DT app directory, not necessarily the Git top-level directory.
bash
MV3DT_APP_SUBDIR="src/apps/reference_apps/deepstream-tracker-3d-multi-view"

is_mv3dt_app_dir() {
  test -f "$1/README.md" || return 1
  test -d "$1/config_templates" || return 1
  test -d "$1/scripts" || return 1
  grep -q "Multi-View 3D Tracking" "$1/README.md"
}

GIT_TOP="$(git rev-parse --show-toplevel 2>/dev/null || true)"
CANDIDATES=()
if [ -n "${MV3DT_REPO_ROOT:-}" ]; then CANDIDATES+=("${MV3DT_REPO_ROOT}"); fi
if [ -n "${DEEPSTREAM_REPO_ROOT:-}" ]; then CANDIDATES+=("${DEEPSTREAM_REPO_ROOT}/${MV3DT_APP_SUBDIR}"); fi
CANDIDATES+=("${PWD}")
if [ -n "${GIT_TOP}" ]; then
  CANDIDATES+=("${GIT_TOP}" "${GIT_TOP}/${MV3DT_APP_SUBDIR}")
fi
CANDIDATES+=("${HOME}/DeepStream/${MV3DT_APP_SUBDIR}" "${HOME}/deepstream/${MV3DT_APP_SUBDIR}")

REPO_ROOT=""
for candidate in "${CANDIDATES[@]}"; do
  if [ -n "$candidate" ] && is_mv3dt_app_dir "$candidate"; then
    REPO_ROOT="$(cd "$candidate" && pwd)"
    break
  fi
done

if [ -z "${REPO_ROOT}" ]; then
  cat <<'EOF'
ERROR: MV3DT reference app directory was not found.
Set MV3DT_REPO_ROOT to an existing deepstream-tracker-3d-multi-view app directory, or ask the user to approve cloning the public DeepStream repo and then run:

  DEEPSTREAM_REPO_ROOT="${DEEPSTREAM_REPO_ROOT:-$HOME/DeepStream}"
  git clone https://github.com/NVIDIA/DeepStream.git "$DEEPSTREAM_REPO_ROOT"
  export MV3DT_REPO_ROOT="$DEEPSTREAM_REPO_ROOT/src/apps/reference_apps/deepstream-tracker-3d-multi-view"

Do not clone silently.
EOF
  exit 1
fi

cd "${REPO_ROOT}"
export REPO_ROOT MV3DT_REPO_ROOT="${REPO_ROOT}"
If the app directory cannot be resolved, ask the user for an existing checkout path or for approval to clone
https://github.com/NVIDIA/DeepStream
. Do not clone silently.
本技能可安装在DeepStream仓库外部。需将
REPO_ROOT
解析为MV3DT应用目录,而非Git顶层目录。
bash
MV3DT_APP_SUBDIR="src/apps/reference_apps/deepstream-tracker-3d-multi-view"

is_mv3dt_app_dir() {
  test -f "$1/README.md" || return 1
  test -d "$1/config_templates" || return 1
  test -d "$1/scripts" || return 1
  grep -q "Multi-View 3D Tracking" "$1/README.md"
}

GIT_TOP="$(git rev-parse --show-toplevel 2>/dev/null || true)"
CANDIDATES=()
if [ -n "${MV3DT_REPO_ROOT:-}" ]; then CANDIDATES+=("${MV3DT_REPO_ROOT}"); fi
if [ -n "${DEEPSTREAM_REPO_ROOT:-}" ]; then CANDIDATES+=("${DEEPSTREAM_REPO_ROOT}/${MV3DT_APP_SUBDIR}"); fi
CANDIDATES+=("${PWD}")
if [ -n "${GIT_TOP}" ]; then
  CANDIDATES+=("${GIT_TOP}" "${GIT_TOP}/${MV3DT_APP_SUBDIR}")
fi
CANDIDATES+=("${HOME}/DeepStream/${MV3DT_APP_SUBDIR}" "${HOME}/deepstream/${MV3DT_APP_SUBDIR}")

REPO_ROOT=""
for candidate in "${CANDIDATES[@]}"; do
  if [ -n "$candidate" ] && is_mv3dt_app_dir "$candidate"; then
    REPO_ROOT="$(cd "$candidate" && pwd)"
    break
  fi
done

if [ -z "${REPO_ROOT}" ]; then
  cat <<'EOF'
ERROR: MV3DT reference app directory was not found.
Set MV3DT_REPO_ROOT to an existing deepstream-tracker-3d-multi-view app directory, or ask the user to approve cloning the public DeepStream repo and then run:

  DEEPSTREAM_REPO_ROOT="${DEEPSTREAM_REPO_ROOT:-$HOME/DeepStream}"
  git clone https://github.com/NVIDIA/DeepStream.git "$DEEPSTREAM_REPO_ROOT"
  export MV3DT_REPO_ROOT="$DEEPSTREAM_REPO_ROOT/src/apps/reference_apps/deepstream-tracker-3d-multi-view"

Do not clone silently.
EOF
  exit 1
fi

cd "${REPO_ROOT}"
export REPO_ROOT MV3DT_REPO_ROOT="${REPO_ROOT}"
若无法解析应用目录,请询问用户现有目录路径或是否同意克隆
https://github.com/NVIDIA/DeepStream
仓库。请勿静默克隆。

Step 1: Select The Primary Workflow

步骤1:选择主工作流

Load exactly one primary reference for the user's current request:
User intentReference
Install, prepare, or verify prerequisites
references/setup.md
Run bundled 4-camera or 12-camera sample
references/sample-run.md
Run custom synchronized MP4s
references/custom-dataset.md
Missing custom calibration
references/amc-calibration-handoff.md
, then return to
references/custom-dataset.md
View OSD, BEV, screenshots, recordings, or Kafka metadata
references/visualization-metadata.md
Stop a run, clean generated artifacts, or stop prerequisite services
references/setup.md
If setup, datasets, models, Kafka, Mosquitto, Docker GPU runtime, or the Python venv are missing, load
references/setup.md
before continuing to the user's original workflow.
When
sample-run.md
or
custom-dataset.md
needs to regenerate DeepStream configs, load
references/generate-configs.md
as the canonical shared config-generation reference rather than duplicating the shell logic.
根据用户当前请求,加载恰好一个主参考文档:
用户意图参考文档
安装、准备或验证前置条件
references/setup.md
运行内置的4相机或12相机示例
references/sample-run.md
运行自定义同步MP4文件
references/custom-dataset.md
缺少自定义校准信息
references/amc-calibration-handoff.md
,之后返回
references/custom-dataset.md
查看OSD、BEV、截图、录制文件或Kafka元数据
references/visualization-metadata.md
停止运行、清理生成的工件或停止前置服务
references/setup.md
若缺少环境设置、数据集、模型、Kafka、Mosquitto、Docker GPU运行时或Python虚拟环境,需先加载
references/setup.md
,再继续执行用户原本的工作流。
sample-run.md
custom-dataset.md
需要重新生成DeepStream配置时,加载
references/generate-configs.md
作为标准的共享配置生成参考文档,而非重复编写Shell逻辑。

Step 2: Follow The Run Stages

步骤2:遵循运行阶段

For every run, use this stage order:
StageAction
ValidateCheck app directory, prerequisites, dataset shape, display/headless mode, Docker GPU support, and output-directory writability.
PrepareState the selected sample/custom dataset, detector, run mode, output directory, expected output surfaces, and the Docker security decision for
--privileged --net=host
before launch.
ExecuteGenerate configs, start offline BEV capture first in headless mode or when saved BEV MP4 is explicitly requested, then run DeepStream or delegate missing calibration to AutoMagicCalib.
VerifyConfirm functional readiness with
App run successful
, fresh MP4 artifacts when file output is enabled, Kafka offsets/messages, and BEV message/frame counts when BEV MP4 capture runs.
ReportSummarize selected options, generated files, artifact paths, file sizes/counts, and any skipped or failed output surface; after a default 4-camera sample run, mention that the 12-camera sample is also available as a follow-up.
每次运行需按照以下阶段顺序执行:
阶段操作
验证检查应用目录、前置条件、数据集格式、显示/无头模式、Docker GPU支持以及输出目录的可写性。
准备在启动前,说明所选的示例/自定义数据集、检测器、运行模式、输出目录、预期输出界面,以及Docker使用
--privileged --net=host
的安全决策。
执行生成配置文件;若为无头模式或明确要求保存BEV MP4文件,先启动离线BEV捕获,再运行DeepStream或将缺失的校准任务委托给AutoMagicCalib。
验证通过
App run successful
提示、启用文件输出时生成的新MP4工件、Kafka偏移量/消息、以及BEV MP4捕获运行时的BEV消息/帧数量,确认功能就绪。
报告总结所选选项、生成的文件、工件路径、文件大小/数量,以及任何跳过或失败的输出界面;默认4相机示例运行完成后,提及12相机示例也可作为后续操作。

Step 3: Apply Defaults Explicitly

步骤3:明确应用默认设置

  • Runtime path: DeepStream Container.
  • Display path: if a working X11/VNC display is available, use the repo quick-start path with OSD and BEV windows. The OSD window does not save MP4 by default; when the user explicitly asks to save output in display mode, regenerate configs with both
    --enable-osd
    and
    --enable-file-output
    plus
    --enable-msg-broker
    .
  • Headless path: if no working display is available, use saved outputs by default. Generate configs with
    --enable-file-output
    and
    --enable-msg-broker
    ; start offline BEV capture before DeepStream so the run produces both the tiled DeepStream MP4 and the Kafka-derived BEV MP4.
  • Sample dataset: support both shipped 4-camera and 12-camera datasets. If the user does not specify a sample, run the 4-camera sample first, then mention in the final report that the 12-camera sample is also available and can be run next.
  • Detector:
    PeopleNetTransformer
    unless the user asks for
    RTDETR
    or
    PeopleNet2.6.3
    ; carry the selected
    DETECTOR_MODEL
    through config generation and AMC
    camInfo
    modelInfo normalization. For custom calibration handoff, ask the user to choose the AutoMagicCalib detector instead of silently defaulting.
  • Custom data source: synchronized MP4 files. Live-stream handling is outside this first-release skill.
  • Calibration handoff: use standalone AutoMagicCalib skills instead of duplicating their setup or API workflow.
  • 运行路径:DeepStream容器。
  • 显示模式:若有可用的X11/VNC显示环境,使用仓库快速启动路径,显示OSD和BEV窗口。默认情况下OSD窗口不保存MP4文件;当用户明确要求在显示模式下保存输出时,需重新生成配置,同时启用
    --enable-osd
    --enable-file-output
    --enable-msg-broker
  • 无头模式:若无可用显示环境,默认使用保存输出模式。生成配置时启用
    --enable-file-output
    --enable-msg-broker
    ;在运行DeepStream前启动离线BEV捕获,确保运行同时生成拼接后的DeepStream MP4文件和基于Kafka的BEV MP4文件。
  • 示例数据集:支持内置的4相机和12相机数据集。若用户未指定示例,先运行4相机示例,然后在最终报告中提及12相机示例也可用,可后续运行。
  • 检测器:默认使用
    PeopleNetTransformer
    ,除非用户要求使用
    RTDETR
    PeopleNet2.6.3
    ;将所选的
    DETECTOR_MODEL
    贯穿配置生成和AMC
    camInfo
    模型信息标准化流程。对于自定义校准委托,需询问用户选择AutoMagicCalib检测器,而非静默使用默认值。
  • 自定义数据源:同步MP4文件。直播流处理不在本初始版本技能范围内。
  • 校准委托:使用独立的AutoMagicCalib技能,而非重复其设置或API工作流。

Step 4: Preserve Idempotency And User Data

步骤4:保持幂等性并保护用户数据

  • Readiness checks are safe to rerun.
  • Setup may install packages, pull containers, download models, and start services; ask first.
  • Config generation and DeepStream runs update generated files under
    EXPERIMENT_DIR
    ; record
    RUN_STARTED_AT
    and do not report old artifacts as current-run success.
  • Normal teardown stops only current run processes and leaves Kafka, Mosquitto, models, datasets, and generated artifacts in place unless the user explicitly asks to stop services or delete files.
  • Custom datasets are user data. Copy missing calibration-format variants by default, and ask before renaming, overwriting, clearing, or deleting dataset files.
  • 就绪检查可安全重复执行。
  • 环境设置可能涉及安装包、拉取容器、下载模型和启动服务;执行前需先询问用户。
  • 配置生成和DeepStream运行会更新
    EXPERIMENT_DIR
    下的生成文件;记录
    RUN_STARTED_AT
    ,勿将旧工件报告为当前运行的成功结果。
  • 正常关闭仅停止当前运行进程,保留Kafka、Mosquitto、模型、数据集和生成的工件,除非用户明确要求停止服务或删除文件。
  • 自定义数据集属于用户数据。默认复制缺失的校准格式变体,重命名、覆盖、清除或删除数据集文件前需先询问用户。

Success Criteria

成功标准

  • Prerequisite checks pass or the missing prerequisite is reported with a narrow next step.
  • DeepStream run eventually prints
    App run successful
    .
  • Display mode shows the DeepStream OSD grid and live BEV visualizer by default.
  • Headless mode produces a fresh
    ${EXPERIMENT_DIR}/outVideos/tiled_display_raw.mp4
    and attempts BEV MP4 capture by default.
  • Kafka topic
    mv3dt
    receives current-run protobuf metadata when message broker output is enabled.
  • BEV MP4 is reported as successful only when the separate BEV capture process produced nonzero messages and frames.
  • 前置条件检查通过,或报告缺失的前置条件并给出明确的下一步操作。
  • DeepStream运行最终输出
    App run successful
  • 显示模式默认显示DeepStream OSD网格和实时BEV可视化工具。
  • 无头模式默认生成新的
    ${EXPERIMENT_DIR}/outVideos/tiled_display_raw.mp4
    文件,并尝试捕获BEV MP4文件。
  • 启用消息代理输出时,Kafka主题
    mv3dt
    接收当前运行的protobuf元数据。
  • 仅当独立的BEV捕获进程生成了非零数量的消息和帧时,才报告BEV MP4生成成功。

Key Output

关键输出

  • Generated configs:
    ${EXPERIMENT_DIR}/config_deepstream.txt
    ,
    ${EXPERIMENT_DIR}/config_tracker.yml
    ,
    ${EXPERIMENT_DIR}/config_msgconv.txt
  • Saved DeepStream tiled MP4 when file output is enabled:
    ${EXPERIMENT_DIR}/outVideos/tiled_display_raw.mp4
  • Saved BEV MP4 when offline capture is used:
    ${EXPERIMENT_DIR}/bev_outputs/trajectory_video_<timestamp>.mp4
  • Kafka topic:
    mv3dt
  • Sample output roots:
    ${REPO_ROOT}/experiments/deepstream/4cam
    and
    ${REPO_ROOT}/experiments/deepstream/12cam
  • 生成的配置文件:
    ${EXPERIMENT_DIR}/config_deepstream.txt
    ${EXPERIMENT_DIR}/config_tracker.yml
    ${EXPERIMENT_DIR}/config_msgconv.txt
  • 启用文件输出时保存的DeepStream拼接MP4文件:
    ${EXPERIMENT_DIR}/outVideos/tiled_display_raw.mp4
  • 使用离线捕获时保存的BEV MP4文件:
    ${EXPERIMENT_DIR}/bev_outputs/trajectory_video_<timestamp>.mp4
  • Kafka主题:
    mv3dt
  • 示例输出根目录:
    ${REPO_ROOT}/experiments/deepstream/4cam
    ${REPO_ROOT}/experiments/deepstream/12cam

Troubleshooting

故障排除

IssueFirst action
Setup prerequisites missingLoad
references/setup.md
and run the check-only path before setup.
Docker cannot access GPUFix NVIDIA Container Toolkit or Docker runtime before launching samples.
Display window missingCheck
DISPLAY
and
/tmp/.X11-unix
; use the headless saved-output path when no display is available.
DeepStream MP4 missingConfirm configs were generated with
--enable-file-output
and verify the artifact is newer than
RUN_STARTED_AT
.
BEV MP4 missing or zero messagesStart offline BEV capture with
--from-end
before DeepStream, use a long enough
--first-message-timeout
, and verify Kafka offsets move during the run.
Kafka client shows no messagesRegenerate configs with
--enable-msg-broker
and verify topic
mv3dt
exists.
Custom dataset lacks calibrationLoad
references/amc-calibration-handoff.md
; ask detector/settings choices before delegating.
Generated files are root-ownedReport the ownership issue and ask before applying a narrow generated-directory permission fix.
问题首要操作
缺少前置条件加载
references/setup.md
,在执行设置前先运行检查流程。
Docker无法访问GPU在启动示例前修复NVIDIA Container Toolkit或Docker运行时。
显示窗口缺失检查
DISPLAY
/tmp/.X11-unix
;若无可用显示环境,使用无头模式保存输出。
DeepStream MP4文件缺失确认生成配置时启用了
--enable-file-output
,并验证工件的时间晚于
RUN_STARTED_AT
BEV MP4文件缺失或消息数量为零在运行DeepStream前使用
--from-end
启动离线BEV捕获,设置足够长的
--first-message-timeout
,并验证运行期间Kafka偏移量有变化。
Kafka客户端无消息显示重新生成配置并启用
--enable-msg-broker
,验证主题
mv3dt
存在。
自定义数据集缺少校准信息加载
references/amc-calibration-handoff.md
;委托前询问用户检测器/设置选项。
生成文件为root权限所有报告权限问题,在应用针对性的生成目录权限修复前需先询问用户。

Safety Notes

安全注意事项

  • Ask before commands that use
    sudo
    , install packages, pull containers, download models, start or stop services, change host display access, overwrite dataset files, or clear generated state.
  • Treat deleting Kafka, Mosquitto, models, datasets, or experiment outputs as destructive cleanup. Show the exact targets and get explicit confirmation before removing anything.
  • Before running
    docker run --privileged --net=host
    , explicitly state that the container gets broad host, device, network, and mounted-repo access, then get user approval.
  • Do not silently clone repositories, change host permissions outside the repo, rename or delete user datasets, or report old artifacts as current-run success.
  • Treat custom videos, calibration, saved visualizations, and tracking metadata as potentially sensitive local data. Keep outputs local unless the user explicitly asks to move or share them.
  • If permission fixes are needed for generated outputs, propose the narrowest generated-directory-only fix and ask first; never recommend broad world-writable recursive permission changes.
  • 执行使用
    sudo
    、安装包、拉取容器、下载模型、启动或停止服务、更改主机显示权限、覆盖数据集文件或清除生成状态的命令前,需先询问用户。
  • 将删除Kafka、Mosquitto、模型、数据集或实验输出视为破坏性清理操作。执行前需显示明确的目标并获得用户的明确确认。
  • 运行
    docker run --privileged --net=host
    前,需明确说明容器将获得广泛的主机、设备、网络和挂载仓库访问权限,然后获得用户批准。
  • 请勿静默克隆仓库、更改仓库外的主机权限、重命名或删除用户数据集,或将旧工件报告为当前运行的成功结果。
  • 将自定义视频、校准信息、保存的可视化结果和追踪元数据视为潜在敏感的本地数据。除非用户明确要求移动或共享,否则保持输出本地化。
  • 若需修复生成输出的权限问题,建议仅针对生成目录进行最小范围的修复,并先询问用户;绝不推荐广泛的全局可写递归权限更改。

Related Skills

相关技能

  • amc-setup-calibration-stack
    - Launch the standalone AutoMagicCalib stack when calibration is needed.
  • amc-run-video-calibration
    - Generate calibration from synchronized local MP4s before returning to MV3DT.
<!-- signing marker -->
  • amc-setup-calibration-stack
    - 当需要校准时,启动独立的AutoMagicCalib栈。
  • amc-run-video-calibration
    - 从同步本地MP4文件生成校准信息,之后返回MV3DT操作。
<!-- signing marker -->