Skill: Run DeepStream MV3DT

When to Use This Skill

• "set up MV3DT DeepStream"
• "run the 4-camera MV3DT sample"
• "run the 12-camera MV3DT sample"
• "run MV3DT on my synchronized MP4s"
• "I have videos but no calibration; calibrate and run MV3DT"
• "show the BEV visualizer or Kafka metadata"
• "stop MV3DT" / "clean up MV3DT" / "tear down MV3DT"

Examples

• "Deploy the DeepStream MV3DT 4-camera sample and show the OSD and BEV windows."
• "Run the MV3DT 12-camera sample headlessly with RTDETR and save videos."
• "Run MV3DT on synchronized MP4s under /data/mv3dt-demo using PeopleNetTransformer."
• "My custom MV3DT videos do not have calibration; use AutoMagicCalib, then run MV3DT."

Overview

DeepStream/src/apps/reference_apps/deepstream-tracker-3d-multi-view

Prerequisites

• MV3DT reference app directory on disk under

  DeepStream/src/apps/reference_apps/deepstream-tracker-3d-multi-view

  ; if it is absent, ask before cloning the public DeepStream repo
• Docker with NVIDIA GPU support
• DeepStream container image access
• MV3DT sample datasets, models, custom parser libraries, Kafka, Mosquitto, and

  mv3dt_venv

  prepared by the repo setup script
• Working X11/VNC display for live OSD/BEV windows, or the saved-output headless path for tiled DeepStream MP4 plus Kafka-derived BEV MP4

Instructions

Step 0: Resolve MV3DT App Checkout

REPO_ROOT

MV3DT_APP_SUBDIR="src/apps/reference_apps/deepstream-tracker-3d-multi-view"

is_mv3dt_app_dir() {
  test -f "$1/README.md" || return 1
  test -d "$1/config_templates" || return 1
  test -d "$1/scripts" || return 1
  grep -q "Multi-View 3D Tracking" "$1/README.md"
}

GIT_TOP="$(git rev-parse --show-toplevel 2>/dev/null || true)"
CANDIDATES=()
if [ -n "${MV3DT_REPO_ROOT:-}" ]; then CANDIDATES+=("${MV3DT_REPO_ROOT}"); fi
if [ -n "${DEEPSTREAM_REPO_ROOT:-}" ]; then CANDIDATES+=("${DEEPSTREAM_REPO_ROOT}/${MV3DT_APP_SUBDIR}"); fi
CANDIDATES+=("${PWD}")
if [ -n "${GIT_TOP}" ]; then
  CANDIDATES+=("${GIT_TOP}" "${GIT_TOP}/${MV3DT_APP_SUBDIR}")
fi
CANDIDATES+=("${HOME}/DeepStream/${MV3DT_APP_SUBDIR}" "${HOME}/deepstream/${MV3DT_APP_SUBDIR}")

REPO_ROOT=""
for candidate in "${CANDIDATES[@]}"; do
  if [ -n "$candidate" ] && is_mv3dt_app_dir "$candidate"; then
    REPO_ROOT="$(cd "$candidate" && pwd)"
    break
  fi
done

if [ -z "${REPO_ROOT}" ]; then
  cat <<'EOF'
ERROR: MV3DT reference app directory was not found.
Set MV3DT_REPO_ROOT to an existing deepstream-tracker-3d-multi-view app directory, or ask the user to approve cloning the public DeepStream repo and then run:

  DEEPSTREAM_REPO_ROOT="${DEEPSTREAM_REPO_ROOT:-$HOME/DeepStream}"
  git clone https://github.com/NVIDIA/DeepStream.git "$DEEPSTREAM_REPO_ROOT"
  export MV3DT_REPO_ROOT="$DEEPSTREAM_REPO_ROOT/src/apps/reference_apps/deepstream-tracker-3d-multi-view"

Do not clone silently.
EOF
  exit 1
fi

cd "${REPO_ROOT}"
export REPO_ROOT MV3DT_REPO_ROOT="${REPO_ROOT}"

https://github.com/NVIDIA/DeepStream

Step 1: Select The Primary Workflow

references/setup.md

references/sample-run.md

references/custom-dataset.md

references/amc-calibration-handoff.md

references/custom-dataset.md

references/visualization-metadata.md

references/setup.md

references/setup.md

sample-run.md

custom-dataset.md

references/generate-configs.md

Step 2: Follow The Run Stages

--privileged --net=host

App run successful

Step 3: Apply Defaults Explicitly

• Runtime path: DeepStream Container.
• Display path: if a working X11/VNC display is available, use the repo quick-start path with OSD and BEV windows. The OSD window does not save MP4 by default; when the user explicitly asks to save output in display mode, regenerate configs with both

  --enable-osd

  and

  --enable-file-output

  plus

  --enable-msg-broker

  .
• Headless path: if no working display is available, use saved outputs by default. Generate configs with

  --enable-file-output

  and

  --enable-msg-broker

  ; start offline BEV capture before DeepStream so the run produces both the tiled DeepStream MP4 and the Kafka-derived BEV MP4.
• Sample dataset: support both shipped 4-camera and 12-camera datasets. If the user does not specify a sample, run the 4-camera sample first, then mention in the final report that the 12-camera sample is also available and can be run next.
• Detector:

  PeopleNetTransformer

  unless the user asks for

  RTDETR

  or

  PeopleNet2.6.3

  ; carry the selected

  DETECTOR_MODEL

  through config generation and AMC

  camInfo

  modelInfo normalization. For custom calibration handoff, ask the user to choose the AutoMagicCalib detector instead of silently defaulting.
• Custom data source: synchronized MP4 files. Live-stream handling is outside this first-release skill.
• Calibration handoff: use standalone AutoMagicCalib skills instead of duplicating their setup or API workflow.

Step 4: Preserve Idempotency And User Data

• Readiness checks are safe to rerun.
• Setup may install packages, pull containers, download models, and start services; ask first.
• Config generation and DeepStream runs update generated files under

  EXPERIMENT_DIR

  ; record

  RUN_STARTED_AT

  and do not report old artifacts as current-run success.
• Normal teardown stops only current run processes and leaves Kafka, Mosquitto, models, datasets, and generated artifacts in place unless the user explicitly asks to stop services or delete files.
• Custom datasets are user data. Copy missing calibration-format variants by default, and ask before renaming, overwriting, clearing, or deleting dataset files.

Success Criteria

• Prerequisite checks pass or the missing prerequisite is reported with a narrow next step.
• DeepStream run eventually prints

  App run successful

  .
• Display mode shows the DeepStream OSD grid and live BEV visualizer by default.
• Headless mode produces a fresh

  ${EXPERIMENT_DIR}/outVideos/tiled_display_raw.mp4

  and attempts BEV MP4 capture by default.
• Kafka topic

  mv3dt

  receives current-run protobuf metadata when message broker output is enabled.
• BEV MP4 is reported as successful only when the separate BEV capture process produced nonzero messages and frames.

Key Output

• Generated configs:

  ${EXPERIMENT_DIR}/config_deepstream.txt

  ,

  ${EXPERIMENT_DIR}/config_tracker.yml

  ,

  ${EXPERIMENT_DIR}/config_msgconv.txt

• Saved DeepStream tiled MP4 when file output is enabled:

  ${EXPERIMENT_DIR}/outVideos/tiled_display_raw.mp4

• Saved BEV MP4 when offline capture is used:

  ${EXPERIMENT_DIR}/bev_outputs/trajectory_video_<timestamp>.mp4

• Kafka topic:

  mv3dt

• Sample output roots:

  ${REPO_ROOT}/experiments/deepstream/4cam

  and

  ${REPO_ROOT}/experiments/deepstream/12cam

Troubleshooting

references/setup.md

DISPLAY

/tmp/.X11-unix

--enable-file-output

RUN_STARTED_AT

--from-end

--first-message-timeout

--enable-msg-broker

mv3dt

references/amc-calibration-handoff.md

Safety Notes

• Ask before commands that use

  sudo

  , install packages, pull containers, download models, start or stop services, change host display access, overwrite dataset files, or clear generated state.
• Treat deleting Kafka, Mosquitto, models, datasets, or experiment outputs as destructive cleanup. Show the exact targets and get explicit confirmation before removing anything.
• Before running

  docker run --privileged --net=host

  , explicitly state that the container gets broad host, device, network, and mounted-repo access, then get user approval.
• Do not silently clone repositories, change host permissions outside the repo, rename or delete user datasets, or report old artifacts as current-run success.
• Treat custom videos, calibration, saved visualizations, and tracking metadata as potentially sensitive local data. Keep outputs local unless the user explicitly asks to move or share them.
• If permission fixes are needed for generated outputs, propose the narrowest generated-directory-only fix and ask first; never recommend broad world-writable recursive permission changes.

Related Skills

• amc-setup-calibration-stack

  - Launch the standalone AutoMagicCalib stack when calibration is needed.

• amc-run-video-calibration

  - Generate calibration from synchronized local MP4s before returning to MV3DT.