tao-generate-referring-expressions

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Image Referring Expression Pipeline

图像指代表达流水线

Generate referring-expression and grounding annotations from images with KITTI-format bounding box labels. A single VLM (Gemini or any OpenAI-compatible endpoint) runs four steps: per-object region descriptions, holistic image captions, grouped grounding expressions tied to bboxes, and an optional double-check verification pass.

从带有KITTI格式边界框标签的图像生成指代表达与grounding标注。单个VLM（Gemini或任何兼容OpenAI的端点）执行四个步骤：逐对象区域描述、整体图像标题、与边界框关联的分组grounding表达，以及可选的双重校验步骤。

Purpose

用途

Transform

(image, KITTI labels)

pairs into a unified

annotations.jsonl

containing rich, grounded referring expressions. The VLM acts as a "teacher" annotator: Steps 0-1 see the image; Step 2 groups Step 0 outputs into grouping phrases with bbox lists; Step 3 (optional) re-examines those bboxes against the image and corrects mismatches.

将

(图像, KITTI标签)

对转换为包含丰富grounded指代表达的统一

annotations.jsonl

文件。VLM充当「教师」标注器：步骤0-1读取图像；步骤2将步骤0的输出分组为带有边界框列表的分组短语；步骤3（可选）对照图像重新检查这些边界框并纠正不匹配项。

Pipeline Architecture

流水线架构

Step 0: Region expression  ──┐
                              ├──▶  Step 2: Grounding expression  ──▶  [Step 3: Double check]
Step 1: Image caption  ──────┘                                                   (optional)

Step 0 (region_expr) — VLM emits one short discriminative phrase per KITTI bbox (
```
bbox_2d
```
,
```
type
```
,
```
color
```
,
```
description
```
).
Step 1 (image_caption) — VLM emits a holistic, location-agnostic scene caption.
Step 2 (grounding_expr) — VLM groups Step 0 objects into grouping phrases and returns one bbox list per group, optionally using Step 1's caption as extra context.
Step 3 (double_check) — VLM re-checks each Step 2 bbox against the image; bad matches are removed, slightly-off boxes get tightened.

Steps 0 and 1 run in parallel within a single thread pool (they only depend on the seed records). Each step writes its own

step_<N>_*/annotations.jsonl

and skips already-processed images on re-run unless

workflow.force_reprocess: true

Step 0: Region expression  ──┐
                              ├──▶  Step 2: Grounding expression  ──▶  [Step 3: Double check]
Step 1: Image caption  ──────┘                                                   (optional)

Step 0 (region_expr) — VLM为每个KITTI边界框生成一个简短的区分性短语（包含
```
bbox_2d
```
、
```
type
```
、
```
color
```
、
```
description
```
）。
Step 1 (image_caption) — VLM生成一个整体的、与位置无关的场景标题。
Step 2 (grounding_expr) — VLM将步骤0中的对象分组为分组短语，并为每个组返回一个边界框列表，可选择使用步骤1的标题作为额外上下文。
Step 3 (double_check) — VLM对照图像重新检查步骤2中的每个边界框；移除匹配错误的框，微调位置略有偏差的框。

步骤0和1在单个线程池中并行运行（它们仅依赖于初始记录）。每个步骤都会生成自己的

step_<N>_*/annotations.jsonl

文件，重新运行时会跳过已处理的图像，除非设置

workflow.force_reprocess: true

。

Instructions

使用说明

Initial setup

初始设置

When a user wants to run this pipeline, walk through these steps:

Images: Ask for
```
data.image_dir
```
, the directory containing
```
.jpg
```
,
```
.jpeg
```
, or
```
.png
```
images.
KITTI labels: Ask for
```
data.kitti_label_dir
```
, the directory containing one
```
.txt
```
label file per image. Each label line must use KITTI format:
```
<type> <truncated> <occluded> <alpha> <bbox_left> <bbox_top> <bbox_right> <bbox_bottom> ...
```
. Lines with fewer than 8 fields are silently skipped. Set this even for Step 1-only runs because Steps 0 and 2 require it.
Resume from existing annotations: If the user already has a unified
```
annotations.jsonl
```
from a previous run, set
```
data.input_annotations_jsonl
```
to that file instead of seeding from
```
data.image_dir
```
and
```
data.kitti_label_dir
```
.
API access: Ask the user which VLM endpoint they want to use. Present these five options and act on the choice:
1. Gemini — set
```
vlm.backend: "gemini"
```
  ; require
```
GOOGLE_API_KEY
```
  (env var or
```
vlm.gemini.api_key
```
  ).
2. NIM (e.g.
```
https://inference-api.nvidia.com/v1
```
  ) — set
```
vlm.backend: "openai"
```
  ; collect
```
base_url
```
  ,
```
model_name
```
  , and
```
api_key
```
  .
3. TAO inference microservice (self-hosted, OpenAI-compatible). Confirm whether the server is already running:
  - Running — collect
```
base_url
```
    ,
```
model_name
```
    , and (optionally)
```
api_key
```
    ; set
```
vlm.backend: "openai"
```
    .
  - Not running — guide the user through the
```
skills/applications/tao-run-inference-service
```
    skill, which stands up a local TAO inference microservice with an OpenAI-compatible API. Before promising a specific model, check
```
skills/applications/tao-run-inference-service/references/service.yaml
```
    for
```
valid_network_arch_config_basenames
```
    . Once the server is up, collect
```
base_url
```
    ,
```
model_name
```
    , and (optionally)
```
api_key
```
    ; set
```
vlm.backend: "openai"
```
    .
4. vLLM (self-hosted, OpenAI-compatible). Confirm whether the server is already running:
  - Running — collect
```
base_url
```
    ,
```
model_name
```
    , and (optionally)
```
api_key
```
    ; set
```
vlm.backend: "openai"
```
    .
  - Not running — follow references/vllm_server.md to install and launch a vLLM server, then collect
```
base_url
```
    ,
```
model_name
```
    , and (optionally)
```
api_key
```
    ; set
```
vlm.backend: "openai"
```
    .
5. Custom (any other OpenAI-compatible endpoint) — set
```
vlm.backend: "openai"
```
  ; collect
```
base_url
```
  ,
```
model_name
```
  , and (optionally)
```
api_key
```
  .
If the user has no endpoint and does not want to set one up, stop and help resolve API access first.
Workflow steps: Choose one of:
- Full pipeline:
```
["0", "1", "2", "3"]
```
- No caption generation:
```
["0", "2", "3"]
```
  , where Step 2 falls back to image-only context
- No verification:
```
["0", "1", "2"]
```
- Custom subset: any supported subset of steps
Output format: Choose one of:
- ```
jsonl
```
  : unified schema only
- ```
legacy
```
  : byte-compatible
```
.txt.stepN
```
  files only
- ```
both
```
  : writes both formats and is the default for downstream tooling

当用户想要运行此流水线时，请按以下步骤操作：

图像：询问用户
```
data.image_dir
```
，即包含
```
.jpg
```
、
```
.jpeg
```
或
```
.png
```
格式图像的目录。
KITTI标签：询问用户
```
data.kitti_label_dir
```
，即包含每个图像对应
```
.txt
```
标签文件的目录。每个标签行必须采用KITTI格式：
```
<type> <truncated> <occluded> <alpha> <bbox_left> <bbox_top> <bbox_right> <bbox_bottom> ...
```
。字段少于8个的行将被自动跳过。即使仅运行步骤1也需设置此参数，因为步骤0和2需要它。
从现有标注恢复：如果用户已有之前运行生成的统一
```
annotations.jsonl
```
文件，请将
```
data.input_annotations_jsonl
```
设置为该文件，而非从
```
data.image_dir
```
和
```
data.kitti_label_dir
```
初始化。
API访问：询问用户想要使用哪个VLM端点。提供以下五个选项并根据选择执行操作：
1. Gemini — 设置
```
vlm.backend: "gemini"
```
  ；需要
```
GOOGLE_API_KEY
```
  （环境变量或
```
vlm.gemini.api_key
```
  ）。
2. NIM（例如
```
https://inference-api.nvidia.com/v1
```
  ） — 设置
```
vlm.backend: "openai"
```
  ；收集
```
base_url
```
  、
```
model_name
```
  和
```
api_key
```
  。
3. TAO推理微服务（自托管，兼容OpenAI）。确认服务器是否已运行：
  - 已运行 — 收集
```
base_url
```
    、
```
model_name
```
    和（可选）
```
api_key
```
    ；设置
```
vlm.backend: "openai"
```
    。
  - 未运行 — 引导用户使用
```
skills/applications/tao-run-inference-service
```
    技能，该技能会启动一个带有兼容OpenAI API的本地TAO推理微服务。在承诺特定模型之前，请查看
```
skills/applications/tao-run-inference-service/references/service.yaml
```
    中的
```
valid_network_arch_config_basenames
```
    。服务器启动后，收集
```
base_url
```
    、
```
model_name
```
    和（可选）
```
api_key
```
    ；设置
```
vlm.backend: "openai"
```
    。
4. vLLM（自托管，兼容OpenAI）。确认服务器是否已运行：
  - 已运行 — 收集
```
base_url
```
    、
```
model_name
```
    和（可选）
```
api_key
```
    ；设置
```
vlm.backend: "openai"
```
    。
  - 未运行 — 按照references/vllm_server.md中的说明安装并启动vLLM服务器，然后收集
```
base_url
```
    、
```
model_name
```
    和（可选）
```
api_key
```
    ；设置
```
vlm.backend: "openai"
```
    。
5. 自定义（任何其他兼容OpenAI的端点） — 设置
```
vlm.backend: "openai"
```
  ；收集
```
base_url
```
  、
```
model_name
```
  和（可选）
```
api_key
```
  。
如果用户没有端点且不想设置，请先帮助解决API访问问题。
工作流步骤：选择以下选项之一：
- 完整流水线：
```
["0", "1", "2", "3"]
```
- 不生成标题：
```
["0", "2", "3"]
```
  ，此时步骤2将回退到仅图像上下文
- 不进行校验：
```
["0", "1", "2"]
```
- 自定义子集：任何受支持的步骤子集
输出格式：选择以下选项之一：
- ```
jsonl
```
  ：仅统一格式
- ```
legacy
```
  ：仅字节兼容的
```
.txt.stepN
```
  文件
- ```
both
```
  ：同时写入两种格式，是下游工具的默认选项

Running the pipeline

运行流水线

The pipeline runs inside the TAO Toolkit container via the

auto_label

CLI:

bash

auto_label generate -e /path/to/spec.yaml \
    results_dir=/results \
    image_referring_expression.data.image_dir=/data/images \
    image_referring_expression.data.kitti_label_dir=/data/labels \
    image_referring_expression.vlm.gemini.api_key=$GOOGLE_API_KEY

Generate a default spec:

auto_label default_specs results_dir=/results module_name=auto_label

, then set

autolabel_type: "image_referring_expression"

. All fields support Hydra dot-notation overrides on the command line.

See references/configuration.md for the full YAML structure, all parameters, model/endpoint setup, and error patterns.

流水线在TAO Toolkit容器内通过

auto_label

CLI运行：

bash

auto_label generate -e /path/to/spec.yaml \
    results_dir=/results \
    image_referring_expression.data.image_dir=/data/images \
    image_referring_expression.data.kitti_label_dir=/data/labels \
    image_referring_expression.vlm.gemini.api_key=$GOOGLE_API_KEY

生成默认配置：

auto_label default_specs results_dir=/results module_name=auto_label

，然后设置

autolabel_type: "image_referring_expression"

。所有字段都支持在命令行使用Hydra点符号进行覆盖。

有关完整YAML结构、所有参数、模型/端点设置和错误模式，请参阅references/configuration.md。

Recommended pilot workflow

Configuration

配置

Key configuration fields (full reference in references/configuration.md):

Field	Default	Description
`workflow.steps`	`["0","1","2","3"]`	Which steps to execute ( `0` =region_expr, `1` =image_caption, `2` =grounding_expr, `3` =double_check)
`workflow.max_workers`	`4`	Parallel threads per step (watch API rate limits)
`workflow.force_reprocess`	`false`	Ignore cached per-step outputs and reprocess from scratch
`workflow.output_format`	`"jsonl"` (set to `"both"` in the default spec)	`"jsonl"` , `"legacy"` , or `"both"`
`vlm.backend`	`"gemini"`	`"gemini"` or `"openai"` (OpenAI-compatible endpoint)
`data.image_dir`	required	Directory of input images ( `.jpg` / `.jpeg` / `.png` )
`data.kitti_label_dir`	required (unless resuming)	Directory of KITTI-format `.txt` label files
`data.input_annotations_jsonl`	`""`	Optional pre-seeded `annotations.jsonl` (skips KITTI seeding)

关键配置字段（完整参考请见references/configuration.md）：

字段	默认值	描述
`workflow.steps`	`["0","1","2","3"]`	要执行的步骤（ `0` =region_expr, `1` =image_caption, `2` =grounding_expr, `3` =double_check）
`workflow.max_workers`	`4`	每个步骤的并行线程数（注意API速率限制）
`workflow.force_reprocess`	`false`	忽略缓存的步骤输出并从头开始重新处理
`workflow.output_format`	`"jsonl"` （默认配置中设置为 `"both"` ）	`"jsonl"` 、 `"legacy"` 或 `"both"`
`vlm.backend`	`"gemini"`	`"gemini"` 或 `"openai"` （兼容OpenAI的端点）
`data.image_dir`	必填	输入图像目录（ `.jpg` / `.jpeg` / `.png` ）
`data.kitti_label_dir`	必填（除非从已有标注恢复）	KITTI格式 `.txt` 标签文件目录
`data.input_annotations_jsonl`	`""`	可选的预初始化 `annotations.jsonl` （跳过KITTI初始化）

Inputs

输入

Two ways to seed the pipeline:

Image directory + KITTI labels (default). Set
```
data.image_dir
```
and
```
data.kitti_label_dir
```
. The orchestrator walks the image directory, reads the matching
```
<stem>.txt
```
KITTI file, parses bboxes (fields 0 + 4-7), reads each image's
```
width
```
/
```
height
```
via PIL, and writes a
```
seed_annotations.jsonl
```
to
```
results_dir/
```
.
Pre-seeded annotations JSONL (resume / pre-computed regions). Set
```
data.input_annotations_jsonl
```
to a file with one
```
{"image_id", "image_path", "width", "height", "kitti_bboxes": [...]}
```
object per line.

有两种方式初始化流水线：

图像目录 + KITTI标签（默认）。设置
```
data.image_dir
```
和
```
data.kitti_label_dir
```
。编排器遍历图像目录，读取匹配的
```
<stem>.txt
```
KITTI文件，解析边界框（字段0 + 4-7），通过PIL读取每个图像的
```
width
```
/
```
height
```
，并将
```
seed_annotations.jsonl
```
写入
```
results_dir/
```
。
预初始化标注JSONL（恢复/预计算区域）。将
```
data.input_annotations_jsonl
```
设置为每行包含一个
```
{"image_id", "image_path", "width", "height", "kitti_bboxes": [...]}
```
对象的文件。

Outputs

输出

All outputs go to

results_dir/

```
seed_annotations.jsonl
```
— initial per-image records (unless
```
input_annotations_jsonl
```
was supplied).

step_0_region_expr/annotations.jsonl

— adds

regions[]

(each with

bbox

bbox_2d

type

color

description

step_1_image_caption/annotations.jsonl

— adds

caption

(string).

step_2_grounding_expr/annotations.jsonl

— adds

expressions[]

(each

{text, instances: [{bbox: [x1,y1,x2,y2]}]}

```
step_3_double_check/annotations.jsonl
```
— same shape as Step 2, with bboxes removed/updated.
```
results_dir/annotations.jsonl
```
— copy of the last completed step's output.
When
```
workflow.output_format
```
is
```
"legacy"
```
or
```
"both"
```
, each step also writes byte-compatible
```
step_<N>_*/labels/<stem>.txt.stepN
```
files for the original 2d-data-engine tooling.

所有输出均保存到

results_dir/

：

```
seed_annotations.jsonl
```
— 初始的逐图像记录（除非提供了
```
input_annotations_jsonl
```
）。

step_0_region_expr/annotations.jsonl

— 添加

regions[]

（每个包含

bbox

bbox_2d

、

type

、

color

、

description

）。

step_1_image_caption/annotations.jsonl

— 添加

caption

（字符串）。

step_2_grounding_expr/annotations.jsonl

— 添加

expressions[]

（每个为

{text, instances: [{bbox: [x1,y1,x2,y2]}]}

）。

```
step_3_double_check/annotations.jsonl
```
— 与步骤2结构相同，但边界框已被移除/更新。
```
results_dir/annotations.jsonl
```
— 最后完成步骤输出的副本。
当
```
workflow.output_format
```
设置为
```
"legacy"
```
或
```
"both"
```
时，每个步骤还会为原始2d-data-engine工具写入字节兼容的
```
step_<N>_*/labels/<stem>.txt.stepN
```
文件。

Prerequisites

前置条件

Container:

nvcr.io/nvidia/tao/tao-toolkit:6.26.3-pyt

API access: At least one VLM endpoint (Gemini API key or OpenAI-compatible endpoint capable of image input)
PIL / Pillow: Required to read image dimensions during seeding (already present in the TAO container)

容器：

nvcr.io/nvidia/tao/tao-toolkit:6.26.3-pyt

API访问：至少一个VLM端点（Gemini API密钥或支持图像输入的兼容OpenAI端点）
PIL / Pillow：初始化期间读取图像尺寸所需（已包含在TAO容器中）

tao-generate-referring-expressions

Original

Translation

Image Referring Expression Pipeline

图像指代表达流水线

Purpose

用途

Pipeline Architecture

流水线架构

Instructions

使用说明

Initial setup

初始设置

Running the pipeline

运行流水线

Recommended pilot workflow

推荐试点工作流

Configuration

配置

Inputs

输入

Outputs

输出

Prerequisites

前置条件