tao-train-nvpanoptix3d

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

NVPanoptix3D

NVPanoptix3D for panoptic 3D scene reconstruction from posed RGB images. Produces 3D panoptic segmentation (semantic, instance, and panoptic masks) with occupancy completion. Built on VGGT backbone with Mask2Former-style head and 3D frustum reconstruction.

Uses 2D and 3D stage checkpoints. Set train.checkpoint_2d and train.checkpoint_3d for staged initialization.

NVPanoptix3D 用于基于带位姿的RGB图像进行全景3D场景重建。可生成带有占据补全的3D全景分割结果（包含语义、实例及全景掩码）。该模型基于VGGT骨干网络构建，搭配Mask2Former风格的头部模块与3D视锥体重建功能。

使用2D和3D阶段检查点。设置train.checkpoint_2d和train.checkpoint_3d用于阶段式初始化。

Dataclass Schemas

数据类模式

Generated TAO Core schemas are packaged in

schemas/<action>.schema.json

, with

schemas/manifest.json

listing available actions. Each generated schema also emits

references/spec_template_<action>.yaml

from the schema top-level

default

field. AutoML enablement is declared at the model layer in

references/skill_info.yaml

via

automl_enabled

. Runnable AutoML still requires

schemas/train.schema.json

and

references/spec_template_train.yaml

to exist and parse. Use the packaged train schema for

automl_default_parameters

automl_disabled_parameters

, defaults, min/max bounds, enums, option weights, math conditions, dependencies, and popular parameters. Do not expect

~/tao-core

at runtime; maintainers regenerate schemas/templates before packaging the skill bank.

生成的TAO Core模式打包在

schemas/<action>.schema.json

中，

schemas/manifest.json

列出了可用的操作。每个生成的模式还会从模式顶层的

default

字段生成

references/spec_template_<action>.yaml

。AutoML支持在

references/skill_info.yaml

的模型层通过

automl_enabled

声明。可运行的AutoML仍要求

schemas/train.schema.json

和

references/spec_template_train.yaml

存在且可解析。使用打包的训练模式来配置

automl_default_parameters

、

automl_disabled_parameters

、默认值、最小/最大边界、枚举值、选项权重、数学条件、依赖关系以及常用参数。运行时不要依赖

~/tao-core

；维护人员在打包技能库前会重新生成模式/模板。

Train Action Policy

训练操作策略

This model is AutoML-enabled at the model layer. Before handling any train-stage request, read

references/skill_info.yaml

and resolve the run override from either an explicit

automl_policy

value or the user's workflow request. Treat phrases like "turn off AutoML", "disable AutoML", "no HPO", or "plain training" as

automl_policy: off

for this run only; otherwise default to

auto

. When

automl_policy: auto

automl_enabled: true

, and both

schemas/train.schema.json

and

references/spec_template_train.yaml

are packaged, route the train action through

tao-skill-bank:tao-run-automl

by default with this model's

skill_dir

. Preserve workflow/application overrides for datasets, specs, output directories, GPU/platform settings, parent checkpoints, and

automl_policy

. Use direct model training only when

automl_policy: off

or the packaged train schema/template is missing; in the missing-schema case, report that AutoML is enabled but not runnable for this model until schemas are generated.

Non-train actions such as

evaluate

inference

export

, and deploy flows stay in this model skill. The per-run

automl_policy

override does not change model metadata.

该模型在模型层支持AutoML。处理任何训练阶段请求前，需读取

references/skill_info.yaml

，并通过显式的

automl_policy

值或用户的工作流请求来确定运行覆盖项。将“turn off AutoML”、“disable AutoML”、“no HPO”或“plain training”这类短语视为本次运行的

automl_policy: off

；否则默认设为

auto

。当

automl_policy: auto

、

automl_enabled: true

，且

schemas/train.schema.json

和

references/spec_template_train.yaml

均已打包时，默认将训练操作通过

tao-skill-bank:tao-run-automl

路由，并传入该模型的

skill_dir

。保留数据集、规格、输出目录、GPU/平台设置、父检查点以及

automl_policy

的工作流/应用覆盖项。仅当

automl_policy: off

或打包的训练模式/模板缺失时，才使用直接模型训练；若模式缺失，需报告该模型已启用AutoML但无法运行，需先生成模式。

非训练操作（如

evaluate

、

inference

、

export

以及部署流程）仍保留在该模型技能中。每次运行的

automl_policy

覆盖项不会更改模型元数据。

Training Requirements

训练要求

Dataset type: nvpanoptix3d
Formats: front3d, matterport
Monitoring metric: kpi

数据集类型: nvpanoptix3d
格式: front3d, matterport
监控指标: kpi

Per-Action Dataset Requirements

各操作的数据集要求

Action	Spec Key	Source	Files	List?
evaluate	dataset.frustum_mask_path	eval_dataset	meta/frustum_mask.npz	No
evaluate	dataset.label_map	eval_dataset	meta/colormap.json	No
evaluate	dataset.val.json_path	eval_dataset	meta/val.json	No
evaluate	dataset.val.base_dir	eval_dataset		No
evaluate	dataset.test.json_path	inference_dataset	meta/test.json	No
evaluate	dataset.test.base_dir	inference_dataset		No
inference	dataset.frustum_mask_path	inference_dataset	meta/frustum_mask.npz	No
inference	dataset.label_map	inference_dataset	meta/colormap.json	No
inference	inference.images_dir	inference_dataset	images.tar.gz	No
train	dataset.frustum_mask_path	train_datasets	meta/frustum_mask.npz	No
train	dataset.label_map	train_datasets	meta/colormap.json	No
train	dataset.train.json_path	train_datasets	meta/train.json	No
train	dataset.train.base_dir	train_datasets		No
train	dataset.val.json_path	eval_dataset	meta/val.json	No
train	dataset.val.base_dir	eval_dataset		No
train	dataset.test.json_path	inference_dataset	meta/test.json	No
train	dataset.test.base_dir	inference_dataset		No

操作	规格键	来源	文件	是否为列表？
evaluate	dataset.frustum_mask_path	eval_dataset	meta/frustum_mask.npz	否
evaluate	dataset.label_map	eval_dataset	meta/colormap.json	否
evaluate	dataset.val.json_path	eval_dataset	meta/val.json	否
evaluate	dataset.val.base_dir	eval_dataset		否
evaluate	dataset.test.json_path	inference_dataset	meta/test.json	否
evaluate	dataset.test.base_dir	inference_dataset		否
inference	dataset.frustum_mask_path	inference_dataset	meta/frustum_mask.npz	否
inference	dataset.label_map	inference_dataset	meta/colormap.json	否
inference	inference.images_dir	inference_dataset	images.tar.gz	否
train	dataset.frustum_mask_path	train_datasets	meta/frustum_mask.npz	否
train	dataset.label_map	train_datasets	meta/colormap.json	否
train	dataset.train.json_path	train_datasets	meta/train.json	否
train	dataset.train.base_dir	train_datasets		否
train	dataset.val.json_path	eval_dataset	meta/val.json	否
train	dataset.val.base_dir	eval_dataset		否
train	dataset.test.json_path	inference_dataset	meta/test.json	否
train	dataset.test.base_dir	inference_dataset		否

Typical Spec Overrides

典型规格覆盖项

Data source overrides are mandatory for every action — the agent MUST construct data source paths from the Per-Action Dataset Requirements table above and include them in

spec_overrides

python

S3_TRAIN = "s3://bucket/data/train"
S3_EVAL = "s3://bucket/data/eval"

train (mandatory data sources):

python

{
    "train.num_epochs": 10,
    "train.checkpoint_interval": 10,
    "train.validation_interval": 10,
    "train.num_gpus": 1,
    "dataset.enable_3d": True,
    "model.sem_seg_head.num_classes": 13,
    "dataset.frustum_mask_path": f"{S3_TRAIN}/meta/frustum_mask.npz",
    "dataset.label_map": f"{S3_TRAIN}/meta/colormap.json",
    "dataset.train.json_path": f"{S3_TRAIN}/meta/train.json",
    "dataset.train.base_dir": f"{S3_TRAIN}",
    "dataset.val.json_path": f"{S3_EVAL}/meta/val.json",
    "dataset.val.base_dir": f"{S3_EVAL}",
    "dataset.test.json_path": f"{S3_EVAL}/meta/test.json",
    "dataset.test.base_dir": f"{S3_EVAL}",
}

evaluate (mandatory data sources):

python

{
    "dataset.enable_3d": True,
    "dataset.frustum_mask_path": f"{S3_EVAL}/meta/frustum_mask.npz",
    "dataset.label_map": f"{S3_EVAL}/meta/colormap.json",
    "dataset.val.json_path": f"{S3_EVAL}/meta/val.json",
    "dataset.val.base_dir": f"{S3_EVAL}",
    "dataset.test.json_path": f"{S3_EVAL}/meta/test.json",
    "dataset.test.base_dir": f"{S3_EVAL}",
}

inference (mandatory data sources):

python

{
    "dataset.enable_3d": True,
    "dataset.frustum_mask_path": f"{S3_EVAL}/meta/frustum_mask.npz",
    "dataset.label_map": f"{S3_EVAL}/meta/colormap.json",
    "inference.images_dir": f"{S3_EVAL}/images.tar.gz",
}

数据源覆盖项对每个操作都是必需的——代理必须根据上述“各操作的数据集要求”表格构建数据源路径，并将其包含在

spec_overrides

中。

python

S3_TRAIN = "s3://bucket/data/train"
S3_EVAL = "s3://bucket/data/eval"

训练（必需数据源）:

python

{
    "train.num_epochs": 10,
    "train.checkpoint_interval": 10,
    "train.validation_interval": 10,
    "train.num_gpus": 1,
    "dataset.enable_3d": True,
    "model.sem_seg_head.num_classes": 13,
    "dataset.frustum_mask_path": f"{S3_TRAIN}/meta/frustum_mask.npz",
    "dataset.label_map": f"{S3_TRAIN}/meta/colormap.json",
    "dataset.train.json_path": f"{S3_TRAIN}/meta/train.json",
    "dataset.train.base_dir": f"{S3_TRAIN}",
    "dataset.val.json_path": f"{S3_EVAL}/meta/val.json",
    "dataset.val.base_dir": f"{S3_EVAL}",
    "dataset.test.json_path": f"{S3_EVAL}/meta/test.json",
    "dataset.test.base_dir": f"{S3_EVAL}",
}

评估（必需数据源）:

python

{
    "dataset.enable_3d": True,
    "dataset.frustum_mask_path": f"{S3_EVAL}/meta/frustum_mask.npz",
    "dataset.label_map": f"{S3_EVAL}/meta/colormap.json",
    "dataset.val.json_path": f"{S3_EVAL}/meta/val.json",
    "dataset.val.base_dir": f"{S3_EVAL}",
    "dataset.test.json_path": f"{S3_EVAL}/meta/test.json",
    "dataset.test.base_dir": f"{S3_EVAL}",
}

推理（必需数据源）:

python

{
    "dataset.enable_3d": True,
    "dataset.frustum_mask_path": f"{S3_EVAL}/meta/frustum_mask.npz",
    "dataset.label_map": f"{S3_EVAL}/meta/colormap.json",
    "inference.images_dir": f"{S3_EVAL}/images.tar.gz",
}

Eval Dataset

评估数据集

Optional. Val/test splits configured via dataset.val and dataset.test paths.

可选。验证/测试集拆分通过dataset.val和dataset.test路径配置。

Important Parameters

重要参数

model.sem_seg_head.num_classes: Number of semantic classes. Default 13.
model.mode: Prediction mode. Options: panoptic, instance, semantic. Default panoptic.
model.backbone_type: Backbone. Default vggt (only option in schema).
model.mask_former.num_object_queries: Object queries. Default 100.
model.mask_former.dec_layers: Decoder layers. Default 10.
model.frustum3d.truncation: 3D frustum truncation. Default 3.
model.frustum3d.panoptic_weight: Panoptic loss weight. Default 25.
model.frustum3d.completion_weights: Completion loss weights. Default [50, 25, 10].
dataset.name: Dataset name. Options: front3d, matterport, synthetic_hospital, synthetic_warehouse.
dataset.downsample_factor: Image downsample factor. Default 1 (Front3D), 2 (Matterport).
dataset.target_size: Target image size. Default [320, 240].
dataset.depth_min: Min depth. Default 0.4 meters.
dataset.depth_max: Max depth. Default 6.0 meters.
train.lr: Learning rate. Default 2e-4. backbone_multiplier=0.1.
train.lr_scheduler: Options: MultiStep, Warmuppoly. Milestones [88, 96].
train.precision: Options: fp16, fp32. Default fp16.
train.distributed_strategy: Options: ddp, fsdp. activation_checkpoint=True by default.
train.clip_grad_norm: Gradient clipping norm. Default 0.1.
export.onnx_file_2d: ONNX path for 2D model component.
export.onnx_file_3d: ONNX path for 3D model component.
export.max_voxels: Max voxels for engine input. Default 700000.
inference.mode: Options: semantic, instance, panoptic.

model.sem_seg_head.num_classes: 语义类别数量。默认值13。
model.mode: 预测模式。选项：panoptic、instance、semantic。默认值panoptic。
model.backbone_type: 骨干网络。默认值vggt（模式中唯一选项）。
model.mask_former.num_object_queries: 对象查询数。默认值100。
model.mask_former.dec_layers: 解码器层数。默认值10。
model.frustum3d.truncation: 3D视锥体截断值。默认值3。
model.frustum3d.panoptic_weight: 全景损失权重。默认值25。
model.frustum3d.completion_weights: 补全损失权重。默认值[50, 25, 10]。
dataset.name: 数据集名称。选项：front3d、matterport、synthetic_hospital、synthetic_warehouse。
dataset.downsample_factor: 图像下采样因子。默认值1（Front3D）、2（Matterport）。
dataset.target_size: 目标图像尺寸。默认值[320, 240]。
dataset.depth_min: 最小深度。默认值0.4米。
dataset.depth_max: 最大深度。默认值6.0米。
train.lr: 学习率。默认值2e-4。backbone_multiplier=0.1。
train.lr_scheduler: 选项：MultiStep、Warmuppoly。里程碑[88, 96]。
train.precision: 选项：fp16、fp32。默认值fp16。
train.distributed_strategy: 选项：ddp、fsdp。默认启用activation_checkpoint=True。
train.clip_grad_norm: 梯度裁剪范数。默认值0.1。
export.onnx_file_2d: 2D模型组件的ONNX路径。
export.onnx_file_3d: 3D模型组件的ONNX路径。
export.max_voxels: 引擎输入的最大体素数。默认值700000。
inference.mode: 选项：semantic、instance、panoptic。

Multi-GPU / Multi-Node

多GPU/多节点

Launch method: Lightning-managed (single

python

process, Lightning spawns workers).

Spec Key	Description	Default
`train.num_gpus`	Number of GPUs	1
`train.gpu_ids`	GPU device indices	[0]
`train.num_nodes`	Number of nodes	1
`train.distributed_strategy`	`ddp` only	`ddp`

fsdp
is NOT supported for NVPanoptix3D (code only handles
```
ddp
```
)
```
ddp
```
with activation checkpointing (enabled by default):
```
find_unused_parameters=False
```
```
ddp
```
without:
```
find_unused_parameters=True
```
FAN backbones with 3D enabled auto-enable
```
sync_batchnorm
```

Multi-node env vars (set by orchestrator):

WORLD_SIZE

NODE_RANK

MASTER_ADDR

MASTER_PORT

NUM_GPU_PER_NODE

启动方式: Lightning管理（单个

python

进程，Lightning生成工作进程）。

规格键	描述	默认值
`train.num_gpus`	GPU数量	1
`train.gpu_ids`	GPU设备索引	[0]
`train.num_nodes`	节点数量	1
`train.distributed_strategy`	仅支持 `ddp`	`ddp`

NVPanoptix3D 不支持
fsdp
（代码仅处理
```
ddp
```
）
启用激活 checkpoint 的
```
ddp
```
（默认启用）：
```
find_unused_parameters=False
```
未启用激活 checkpoint 的
```
ddp
```
：
```
find_unused_parameters=True
```
启用3D的FAN骨干网络会自动启用
```
sync_batchnorm
```

多节点环境变量（由编排器设置）：

WORLD_SIZE

、

NODE_RANK

、

MASTER_ADDR

、

MASTER_PORT

、

NUM_GPU_PER_NODE

。

Export / TRT Defaults

导出/TRT默认值

Exports separate 2D and 3D ONNX models (onnx_file_2d, onnx_file_3d)
TRT data types: FP32, FP16 only
max_voxels: 700000 (engine input tensor limit)

导出独立的2D和3D ONNX模型（onnx_file_2d、onnx_file_3d）
TRT数据类型：仅支持FP32、FP16
max_voxels：700000（引擎输入张量限制）

Hardware

硬件要求

Minimum 2 GPU(s), recommended 4 GPU(s). 40GB+ (A100 recommended) VRAM per GPU. 3D reconstruction is very memory intensive. fp16 recommended. activation_checkpoint enabled by default. FSDP for multi-node. AutoML is enabled at the model layer; preserve this GPU/VRAM guidance when routing train through AutoML.

最少2块GPU，推荐4块GPU。每块GPU需40GB以上显存（推荐A100）。3D重建对内存要求极高。推荐使用fp16。默认启用activation_checkpoint。多节点使用FSDP。模型层已启用AutoML；通过AutoML路由训练任务时，请遵循此GPU/显存指导。

Error Patterns

错误模式

Missing frustum mask: Ensure meta/frustum_mask.npz is present in the dataset directory.

Downsample factor mismatch: Use downsample_factor=2 for Matterport3D, 1 for Front3D / synthetic datasets.

3D occupancy OOM: Reduce frustum_dims or grid_dimensions if running out of GPU memory during 3D reconstruction.

缺失视锥体掩码: 确保数据集目录中存在meta/frustum_mask.npz。

下采样因子不匹配: Matterport3D使用downsample_factor=2，Front3D/合成数据集使用downsample_factor=1。

3D占据内存不足: 若3D重建期间GPU内存不足，可减小frustum_dims或grid_dimensions。

Spec Param / Parent Model Inference

规格参数/父模型推理

Model-specific inference mappings belong in this MD file, not in

config.json

. Generated runners should read this section and apply the mappings with SDK helpers before

create_job()

. This mirrors the old microservices

infer_params.py

flow.

Inference mappings from TAO Core

nvpanoptix3d.config.json

Action	Spec Field	Inference Function	Meaning
evaluate	`encryption_key`	`key`	encryption key
evaluate	`evaluate.checkpoint`	`parent_model`	model file inferred from the parent job results folder
evaluate	`results_dir`	`output_dir`	current job results directory
export	`encryption_key`	`key`	encryption key
export	`export.checkpoint`	`parent_model`	model file inferred from the parent job results folder
export	`export.onnx_file_2d`	`create_onnx_file_2d`	create_onnx_file_2d
export	`export.onnx_file_3d`	`create_onnx_file_3d`	create_onnx_file_3d
export	`results_dir`	`output_dir`	current job results directory
inference	`encryption_key`	`key`	encryption key
inference	`inference.checkpoint`	`parent_model`	model file inferred from the parent job results folder
inference	`results_dir`	`output_dir`	current job results directory
train	`encryption_key`	`key`	encryption key
train	`results_dir`	`output_dir`	current job results directory
train	`train.checkpoint_2d`	`parent_model_or_ptm`	parent model if available, otherwise PTM
train	`train.checkpoint_3d`	`ptm`	pretrained model
train	`train.resume_training_checkpoint_path`	`resume_model`	model file inferred from the current job results folder

For

parent_model

parent_model_folder

, pass the upstream train/export/AutoML child job id as

parent_job_id

. The SDK lists the parent result folder, filters checkpoint artifacts, and returns the selected model file or folder. Do not add these mappings back to

config.json

and do not patch generated runner scripts to guess checkpoint paths.

模型特定的推理映射应放在此MD文件中，而非

config.json

。生成的运行器应读取此部分，并在

create_job()

前使用SDK助手应用这些映射。这与旧微服务的

infer_params.py

流程一致。

来自TAO Core

nvpanoptix3d.config.json

的推理映射：

操作	规格字段	推理函数	含义
evaluate	`encryption_key`	`key`	加密密钥
evaluate	`evaluate.checkpoint`	`parent_model`	从父任务结果文件夹推断出的模型文件
evaluate	`results_dir`	`output_dir`	当前任务结果目录
export	`encryption_key`	`key`	加密密钥
export	`export.checkpoint`	`parent_model`	从父任务结果文件夹推断出的模型文件
export	`export.onnx_file_2d`	`create_onnx_file_2d`	create_onnx_file_2d
export	`export.onnx_file_3d`	`create_onnx_file_3d`	create_onnx_file_3d
export	`results_dir`	`output_dir`	当前任务结果目录
inference	`encryption_key`	`key`	加密密钥
inference	`inference.checkpoint`	`parent_model`	从父任务结果文件夹推断出的模型文件
inference	`results_dir`	`output_dir`	当前任务结果目录
train	`encryption_key`	`key`	加密密钥
train	`results_dir`	`output_dir`	当前任务结果目录
train	`train.checkpoint_2d`	`parent_model_or_ptm`	若可用则为父模型，否则为PTM
train	`train.checkpoint_3d`	`ptm`	预训练模型
train	`train.resume_training_checkpoint_path`	`resume_model`	从当前任务结果文件夹推断出的模型文件

对于

parent_model

或

parent_model_folder

，传入上游训练/导出/AutoML子任务ID作为

parent_job_id

。SDK会列出父结果文件夹，过滤检查点工件，并返回选中的模型文件或文件夹。请勿将这些映射添加回

config.json

，也不要修改生成的运行器脚本以猜测检查点路径。