rerun-mp4

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Rerun mp4 ingestion

Rerun MP4 数据导入

Mp4Reader

turns one

.mp4

file into a lazy chunk stream on one entity: compressed video samples, no decode to pixels, nothing re-encoded unless it has to be. Everything is configured on the constructor;

stream()

takes no arguments. Stream mechanics after

.stream()

(filter, map, merge, collect, write) are in

rerun-chunk-processing

Mp4Reader

可将单个

.mp4

文件转换为单个实体上的惰性块流：仅处理压缩视频样本，无需解码为像素，除非必要否则不会重新编码。所有配置均在构造函数中完成；

stream()

方法无需传入参数。调用

.stream()

后的流操作（过滤、映射、合并、收集、写入）由

rerun-chunk-processing

提供支持。

The API

API 介绍

python

from rerun.experimental import Mp4Reader, Mp4TranscodeOptions

reader = Mp4Reader(video_path, entity_path="/camera/front")  # mode="stream" by default
stream = reader.stream()  # lazy: nothing is decoded yet

Every parameter after

path

is keyword-only. One reader handles one file — for several cameras, build one reader per file and

LazyChunkStream.merge(...)

them (see below).

python

from rerun.experimental import Mp4Reader, Mp4TranscodeOptions

reader = Mp4Reader(video_path, entity_path="/camera/front")  # 默认 mode="stream"
stream = reader.stream()  # 惰性执行：此时尚未解码任何内容

path

之后的所有参数均为关键字参数。一个阅读器仅处理一个文件——若要处理多个摄像头的视频，需为每个文件创建一个阅读器，然后使用

LazyChunkStream.merge(...)

合并流（详见下文）。

Two modes

两种模式

`mode`	emits	when
`"stream"` (default)	a codec chunk, per-GOP sample chunks, a keyframe marker	almost always — every frame is time-indexed and queryable
`"asset"`	the whole file as one blob, plus a frame index	the codec cannot be a `VideoStream` , or the source starts mid-GOP

Reach for stream mode; the next section covers exactly what it emits. Asset mode copies the entire file into the recording as one blob, so nothing downstream can look at a single frame without the whole asset. It is the fallback for codecs a

VideoStream

cannot carry (

mp4v

, image-sequence mp4), and it is what

rerun video.mp4

and the built-in file importer use.

`mode`	输出内容	使用场景
`"stream"` （默认）	一个编解码器块、每个GOP对应的样本块、一个关键帧标记	绝大多数场景——每一帧都带有时间索引，可被查询
`"asset"`	整个文件作为单个 blob，外加一个帧索引	编解码器无法被 `VideoStream` 支持，或源视频从GOP中间开始

优先选择流模式；下一节将详细介绍其输出内容。资产模式会将整个文件作为单个 blob 复制到录制文件中，因此下游无法在不加载整个资产的情况下查看单个帧。它是

VideoStream

不支持的编解码器（如

mp4v

、图像序列MP4）的 fallback 方案，也是

rerun video.mp4

命令及内置文件导入器所使用的模式。

What stream mode emits

流模式的输出内容

For

tests/assets/video/Big_Buck_Bunny_1080_1s_h264_nobframes.mp4

(30 frames, one GOP) with

entity_path="/camera/front"

, the whole output is three chunks:

[0] entity=/camera/front static=True rows=1 timelines=[] cols=['VideoStream:codec']
[1] entity=/camera/front static=False rows=30 timelines=['video'] cols=['VideoStream:sample']
[2] entity=/camera/front static=False rows=1 timelines=['video'] cols=['VideoStream:is_keyframe']

One static chunk, one row, holding
```
VideoStream:codec
```
. It carries no timeline — code that walks the stream must handle that (
```
if chunk.is_static: continue
```
).
One temporal chunk per GOP: a keyframe plus every sample that depends on it, up to (not including) the next keyframe. Samples only — the keyframe flags are not a column here.
One trailing
is_keyframe
marker chunk, holding a sparse
```
True
```
row at each keyframe's time and nothing else. Keeping it out of the sample chunks is what lets a keyframe-only query skip the sample payload, and what
```
collect(optimize=…)
```
accepts as canonical. It is list-per-row, so a row reads back as
```
[True]
```
, not
```
True
```
.
Timeline
video
,
```
duration[ns]
```
, values are the mp4 PTS from the start of the video.
Every chunk on the same entity.
```
entity_path=None
```
(the default) derives it from the file's absolute path, so
```
foo/video.mp4
```
run from
```
/data
```
becomes
```
/data/foo/video.mp4
```
. Pass
```
entity_path
```
explicitly in any real pipeline.

A short clip is often a single GOP, so "three chunks total" is the normal shape — not a sign that something was dropped.

chunk_by_gop=False

emits one Rerun chunk per sample instead (the marker chunk is unaffected). That is a debugging shape — one chunk per frame is a poor storage layout, and it also makes a following optimize pass 6× more expensive (measured below) — so leave the default alone unless you are inspecting individual samples.

Distinguish the three by their columns, not by position or

is_static

: the codec chunk is the static one, sample chunks carry

VideoStream:sample

, and the marker chunk carries only

VideoStream:is_keyframe

Supported stream-mode codecs are exactly the five

VideoCodec

values: H264, H265, AV1, VP8, VP9.

以

tests/assets/video/Big_Buck_Bunny_1080_1s_h264_nobframes.mp4

（30帧，单个GOP）为例，设置

entity_path="/camera/front"

后，输出包含三个块：

[0] entity=/camera/front static=True rows=1 timelines=[] cols=['VideoStream:codec']
[1] entity=/camera/front static=False rows=30 timelines=['video'] cols=['VideoStream:sample']
[2] entity=/camera/front static=False rows=1 timelines=['video'] cols=['VideoStream:is_keyframe']

一个静态块，包含一行数据，存储
```
VideoStream:codec
```
。该块不包含时间线——遍历流的代码需处理这种情况（
```
if chunk.is_static: continue
```
）。
每个GOP对应一个时间块：包含一个关键帧及其依赖的所有样本，直到（不包含）下一个关键帧。仅包含样本——关键帧标记不在此列中。
一个末尾的
is_keyframe
标记块，仅在每个关键帧的时间点存储稀疏的
```
True
```
行。将其与样本块分离，使得仅查询关键帧时可跳过样本负载，同时也是
```
collect(optimize=…)
```
接受的标准格式。它采用每行列表的形式，因此读取时返回
```
[True]
```
而非
```
True
```
。
时间线
video
，单位为
```
duration[ns]
```
，值为视频开始后的 mp4 PTS。
所有块都属于同一个实体。默认
```
entity_path=None
```
会从文件的绝对路径生成实体路径，例如从
```
/data
```
目录运行时，
```
foo/video.mp4
```
会变为
```
/data/foo/video.mp4
```
。在实际流水线中需显式传入
```
entity_path
```
。

短片段通常是单个GOP，因此“总共三个块”是正常结构——并非表示有内容丢失。

chunk_by_gop=False

会为每个样本生成一个 Rerun 块（标记块不受影响）。这是一种调试模式——每个帧一个块的存储布局效率低下，还会导致后续的优化过程耗时增加6倍（如下文测试数据所示）——因此除非需要检查单个样本，否则请保留默认设置。

通过列而非位置或

is_static

属性区分这三种块：编解码器块是静态块，样本块包含

VideoStream:sample

，标记块仅包含

VideoStream:is_keyframe

。

流模式支持的编解码器恰好是五种

VideoCodec

类型：H264、H265、AV1、VP8、VP9。

Transcoding through FFmpeg

通过 FFmpeg 转码

VideoStream

cannot yet model DTS != PTS, so an H.264/H.265 source with container-level B-frame reordering cannot be emitted directly. The reader handles that itself: it re-encodes through FFmpeg with

-bf 0

, streams the result back as a fragmented mp4, and turns each fragment into one GOP chunk, so only one GOP is resident at a time. This is automatic and invisible in the API — the output has the same shape as a clean source. It does need an

ffmpeg

executable. Asset mode is unaffected by B-frames.

Mp4TranscodeOptions

(stream mode only) additionally requests a transcode:

field	effect
`output_codec`	re-encode to another `VideoCodec` ; the emitted `VideoStream:codec` follows
`gop_size`	force a keyframe every N frames — the knob for seek cost in the viewer
`try_gpu`	best-effort hardware encode, NVENC / VideoToolbox only
`ffmpeg_override`	use this `ffmpeg` instead of the one on `PATH`

Requesting the

output_codec

the source already uses is a no-op: it stays on the direct, no-FFmpeg path. With

chunk_by_gop=True

gop_size=N

makes every GOP chunk but the last hold exactly N samples — a 30-frame clip at

gop_size=10

gives 10, 10, 10.

try_gpu

realistically covers H264/H265 plus AV1 on newer NVIDIA; VP8/VP9 always fall back to software, and it does nothing unless a transcode is already happening.

python

Mp4Reader(
    video_path,
    entity_path="/camera/front",
    # ~1s GOPs on a 60fps source, so seeking in the viewer stays snappy.
    transcode=Mp4TranscodeOptions(gop_size=64),
).stream()

VideoStream

目前无法处理 DTS != PTS 的情况，因此带有容器级B帧重排序的 H.264/H.265 源视频无法直接输出。阅读器会自动处理此问题：通过 FFmpeg 重新编码并设置

-bf 0

，将结果作为碎片化 mp4 流返回，然后将每个片段转换为一个GOP块，因此同一时间仅需加载一个GOP。此过程在API中是自动且透明的——输出结构与无B帧的源视频一致。需要系统中存在

ffmpeg

可执行文件。资产模式不受B帧影响。

Mp4TranscodeOptions

（仅适用于流模式）可额外触发转码：

字段	效果
`output_codec`	重新编码为另一种 `VideoCodec` ；输出的 `VideoStream:codec` 会同步更新
`gop_size`	强制每N帧生成一个关键帧——用于控制查看器中的搜索成本
`try_gpu`	尝试硬件编码，仅支持 NVENC / VideoToolbox
`ffmpeg_override`	使用指定的 `ffmpeg` 可执行文件，而非系统PATH中的版本

若请求的

output_codec

与源视频的编解码器相同，则不会执行任何操作：将直接采用无FFmpeg的路径。当

chunk_by_gop=True

时，

gop_size=N

会让除最后一个外的所有GOP块恰好包含N个样本——例如30帧的片段设置

gop_size=10

后，会分为10、10、10帧三个块。

try_gpu

实际支持H264/H265以及新版本NVIDIA的AV1；VP8/VP9始终回退到软件编码，且仅在需要转码时生效。

python

Mp4Reader(
    video_path,
    entity_path="/camera/front",
    # 60fps源视频设置约1秒的GOP，确保查看器中的搜索操作流畅。
    transcode=Mp4TranscodeOptions(gop_size=64),
).stream()

Timelines: mp4 PTS is not your recording's clock

时间线：mp4 PTS 并非录制时钟

The emitted times are always PTS — elapsed nanoseconds from the start of that video. In a multi-sensor recording, that is almost never the timeline you want to align on.

```
timeline_name="real_time"
```
renames the timeline.
```
timeline_type="timestamp"
```
only retypes the same PTS values as nanoseconds since the Unix epoch. The reader does not shift them, so on its own it renders the video near 1970. It is meaningful only paired with a retag step.
Retag with
stream.map(...)
. Samples arrive in presentation order (B-frames are already stripped), so a running cursor maps sample
```
i
```
to
```
capture_times_ns[i]
```
— but the cursor must skip the keyframe marker chunk, which holds no samples. Map that one through the PTS the samples were already assigned:

python

SAMPLE_COL = "VideoStream:sample"


def _reindex_to_capture_times(stream, capture_times_ns, timeline_name):
    cursor = 0
    pts_to_time = {}

    def _retag(chunk):
        nonlocal cursor
        if chunk.is_static:  # the codec chunk carries no timeline
            return chunk
        batch = chunk.to_record_batch()
        col_index = batch.schema.get_field_index(timeline_name)
        old_field = batch.schema.field(col_index)
        old_pts = np.asarray(batch.column(col_index).cast(pa.int64()))

        if SAMPLE_COL in batch.schema.names:
            # Clamp in case the decoder yields a slightly different frame count.
            indices = np.clip(np.arange(cursor, cursor + chunk.num_rows), 0, len(capture_times_ns) - 1)
            cursor += chunk.num_rows
            new_times_ns = capture_times_ns[indices]
            pts_to_time.update(zip(old_pts.tolist(), new_times_ns.tolist()))
        else:
            # The sparse keyframe marker, emitted after every sample chunk.
            new_times_ns = np.array([pts_to_time[pts] for pts in old_pts.tolist()], dtype=np.int64)

        times = pa.array(new_times_ns.astype("datetime64[ns]"))
        # `metadata=` is load-bearing — see gotcha 3.
        new_field = pa.field(old_field.name, times.type, nullable=old_field.nullable, metadata=old_field.metadata)
        return Chunk.from_record_batch(batch.set_column(col_index, new_field, times))[0]

    return stream.map(_retag)

The cursor makes this order-dependent, so keep the retag on the single reader's stream, before any merge. This is the DROID loader's pattern — see the references.

Any

map

flat_map

that assumes "every non-static chunk is samples, in order" has this same bug: it advances over the marker chunk and stamps it with whatever time comes next, silently moving the keyframe markers off their samples.

输出的时间始终为PTS——即视频开始后的纳秒数。在多传感器录制场景中，这几乎不可能是你需要对齐的时间线。

```
timeline_name="real_time"
```
可重命名时间线。
```
timeline_type="timestamp"
```
仅将相同的PTS值重新标记为 Unix 纪元以来的纳秒数。阅读器不会对其进行偏移，因此单独使用时视频会显示在1970年左右。仅当与重新标记步骤结合时才有意义。
使用
stream.map(...)
重新标记。样本按展示顺序到达（B帧已被移除），因此可使用游标将第i个样本映射到
```
capture_times_ns[i]
```
——但游标必须跳过不包含样本的关键帧标记块。需根据样本已分配的PTS来映射该块：

python

SAMPLE_COL = "VideoStream:sample"


def _reindex_to_capture_times(stream, capture_times_ns, timeline_name):
    cursor = 0
    pts_to_time = {}

    def _retag(chunk):
        nonlocal cursor
        if chunk.is_static:  # 编解码器块不包含时间线
            return chunk
        batch = chunk.to_record_batch()
        col_index = batch.schema.get_field_index(timeline_name)
        old_field = batch.schema.field(col_index)
        old_pts = np.asarray(batch.column(col_index).cast(pa.int64()))

        if SAMPLE_COL in batch.schema.names:
            # 限制索引范围，避免解码器返回的帧数量略有差异。
            indices = np.clip(np.arange(cursor, cursor + chunk.num_rows), 0, len(capture_times_ns) - 1)
            cursor += chunk.num_rows
            new_times_ns = capture_times_ns[indices]
            pts_to_time.update(zip(old_pts.tolist(), new_times_ns.tolist()))
        else:
            # 稀疏的关键帧标记块，在所有样本块之后输出。
            new_times_ns = np.array([pts_to_time[pts] for pts in old_pts.tolist()], dtype=np.int64)

        times = pa.array(new_times_ns.astype("datetime64[ns]"))
        # `metadata=` 是必需的——参见注意事项3。
        new_field = pa.field(old_field.name, times.type, nullable=old_field.nullable, metadata=old_field.metadata)
        return Chunk.from_record_batch(batch.set_column(col_index, new_field, times))[0]

    return stream.map(_retag)

游标依赖于顺序，因此请在合并流之前对单个阅读器的流进行重新标记。这是DROID加载器采用的模式——参见参考资料。

任何假设“所有非静态块都是按顺序排列的样本”的

map

flat_map

操作都会出现相同的问题：它会推进游标并为标记块打上后续的时间戳，导致关键帧标记与对应样本错位。

Several cameras into one recording

将多个摄像头的视频合并到一个录制文件中

One reader per file, distinct entity paths, then merge:

python

streams = [
    Mp4Reader(path, entity_path=f"/camera/{name}", timeline_name="real_time", timeline_type="timestamp").stream()
    for name, path in cameras.items()
]
(
    LazyChunkStream
    .merge(*streams)
    .collect(optimize=OptimizationProfile.OBJECT_STORE)
    .write_rrd(out_path, application_id=app_id, recording_id=segment_id)
)

OBJECT_STORE

GOP-rebatches the video and preserves the reader's keyframe marker as-is; no

fix_keyframe=True

is needed. If you do see

skipping GoP rebatching … is_keyframe data is incorrect

, something upstream rewrote the marker (see gotcha 4) —

fix_keyframe=True

re-derives it from the encoded samples as an escape hatch.

为每个文件创建一个阅读器，设置不同的实体路径，然后合并流：

python

streams = [
    Mp4Reader(path, entity_path=f"/camera/{name}", timeline_name="real_time", timeline_type="timestamp").stream()
    for name, path in cameras.items()
]
(
    LazyChunkStream
    .merge(*streams)
    .collect(optimize=OptimizationProfile.OBJECT_STORE)
    .write_rrd(out_path, application_id=app_id, recording_id=segment_id)
)

OBJECT_STORE

会对视频进行GOP重新分块，并保留阅读器的关键帧标记；无需设置

fix_keyframe=True

。如果出现

skipping GoP rebatching … is_keyframe data is incorrect

提示，说明上游重写了标记块（参见注意事项4）——

fix_keyframe=True

可作为应急方案，从编码样本中重新生成标记。

How the reader's chunks relate to optimize's

阅读器块与优化块的关系

The reader and

collect(optimize=…)

agree on where GOPs start but not on how many GOPs share a chunk, and that is by design:

The reader emits the finest GOP-aligned partition: exactly one chunk per GOP. It cannot do better, because it does not know which profile the data is headed for.
Optimize then applies the size policy, merging consecutive GOPs up to the profile's
```
max_bytes
```
/
```
max_rows
```
. It never splits a GOP across chunks, so every boundary it keeps is one the reader already produced.

So optimize's partition is a pure coarsening of the reader's: same samples in the same order, every optimized chunk a run of whole reader GOPs, keyframe marker untouched. Where each GOP already sits near the profile's budget the two come out identical (a 30-frame clip at

gop_size=10

is 3 chunks either way). Where GOPs are small they diverge sharply — a 12-GOP clip becomes 1 chunk under

OBJECT_STORE

and 4 under

LIVE

The practical consequence: do not pre-merge or re-chunk the reader's output to "help" optimize. Hand it the per-GOP stream and let the profile decide. And if you skip optimize entirely (writing straight through

send_chunks

, as the DROID loader does), you are storing the finest partition — correct, but more chunks than object storage wants.

阅读器与

collect(optimize=…)

在GOP起始位置上达成一致，但在单个块包含的GOP数量上有所不同，这是有意设计的：

阅读器输出最细粒度的GOP对齐分区：每个GOP恰好对应一个块。它无法做得更优，因为它不知道数据最终会采用哪种配置文件。
优化过程会应用大小策略，合并连续的GOP直到达到配置文件的
```
max_bytes
```
/
```
max_rows
```
限制。它绝不会将一个GOP拆分到多个块中，因此保留的所有边界都是阅读器已生成的边界。

因此优化后的分区是阅读器分区的纯粗化：样本顺序和内容完全相同，每个优化块由多个连续的阅读器GOP块组成，关键帧标记保持不变。当每个GOP的大小接近配置文件的限制时，两者的输出结构完全相同（例如30帧的片段设置

gop_size=10

后，两种方式都会生成3个块）。当GOP较小时，两者差异显著——一个包含12个GOP的片段在

OBJECT_STORE

配置下会变为1个块，而在

LIVE

配置下会变为4个块。

实际结论：不要预先合并或重新分块阅读器的输出来“帮助”优化。直接传入按GOP划分的流，让配置文件决定最终结构。如果完全跳过优化（直接通过

send_chunks

写入，如DROID加载器所示），则会存储最细粒度的分区——虽然正确，但块数量会超出对象存储的预期。

The optimize pass is cheap on GOP-chunked input

对按GOP分块的输入执行优化过程成本很低

Running optimize over the reader's output is not redundant work worth avoiding. On an 89 MB / 1800-frame / 60-GOP H.264 file:

	`collect()`	`collect(optimize=OBJECT_STORE)`	delta
`chunk_by_gop=True` (default)	11.1 ms	16.3 ms	+5.2 ms (1.5×)
`chunk_by_gop=False`	14.2 ms	48.6 ms	+34.4 ms (3.4×)

Neither half of the pass is expensive on GOP-aligned input:

Detecting GOP starts is header-only.
```
build_sample_index
```
runs
```
detect_gop_start
```
on every sample, but that reads a few bytes of each sample's header — it never decodes. The whole optimize delta above (5.2 ms) is smaller than a Python loop calling the same detector over the same 1800 samples (6.0 ms), which is mostly FFI overhead.
Rebuilding the chunks copies nothing.
```
chunk_from_gop
```
calls
```
taken(0..n)
```
, and
```
re_arrow_util::take_array
```
returns the array uncopied when the indices are consecutive from zero over the whole array. One chunk per GOP means that is always the case, so the multi-MB sample buffers are never duplicated — hence 89 MB "rebuilt" in 5 ms.

That fast path is exactly what

chunk_by_gop=False

gives up: each GOP then spans 30 source chunks, so

chunk_from_gop

has to

concat_and_sort

them and the sample bytes really are copied.

LIVE

also costs more than

OBJECT_STORE

(+13.2 ms vs +5.2 ms on the same file), because its much smaller chunk budget makes it attempt compaction it cannot complete — a GOP is never split.

Retag each camera before the merge — the retag above walks chunks with a cursor, so it must see one video's chunks in order.

对阅读器的输出执行优化并非冗余操作，无需刻意避免。在一个89 MB / 1800帧 / 60个GOP的H.264文件上测试：

	`collect()`	`collect(optimize=OBJECT_STORE)`	差值
`chunk_by_gop=True` （默认）	11.1 ms	16.3 ms	+5.2 ms（1.5倍）
`chunk_by_gop=False`	14.2 ms	48.6 ms	+34.4 ms（3.4倍）

对按GOP对齐的输入执行优化的两个步骤成本都很低：

检测GOP起始位置仅需读取头部。
```
build_sample_index
```
会对每个样本调用
```
detect_gop_start
```
，但仅读取每个样本头部的几个字节——无需解码。上述优化的差值（5.2 ms）甚至小于Python循环对1800个样本调用相同检测器的耗时（6.0 ms），后者主要是FFI开销。
重建块无需复制数据。
```
chunk_from_gop
```
调用
```
taken(0..n)
```
，当索引是从0开始的连续序列时，
```
re_arrow_util::take_array
```
会直接返回原数组而不复制。每个GOP对应一个块意味着始终满足此条件，因此多MB的样本缓冲区永远不会被复制——因此89 MB的数据“重建”仅需5 ms。

chunk_by_gop=False

恰好放弃了这种快速路径：此时每个GOP跨越30个源块，因此

chunk_from_gop

必须对这些块进行

concat_and_sort

，样本字节会被实际复制。

LIVE

配置的成本也高于

OBJECT_STORE

（同一文件下差值为+13.2 ms vs +5.2 ms），因为其块大小限制更小，会尝试无法完成的压缩——而GOP绝不会被拆分。

在合并之前对每个摄像头的流进行重新标记——上述重新标记函数通过游标遍历块，因此必须按顺序处理单个视频的块。

Gotchas

注意事项

Errors surface on the first pull, not at construction. The constructor only checks that the file exists and validates its arguments;
```
stream()
```
builds a lazy pipeline and also succeeds. Codec support, keyframe layout, and FFmpeg availability are checked when the stream runs, so wrap the consumption (
```
to_chunks()
```
,
```
send_chunks
```
, iteration), not the
```
Mp4Reader(…)
```
call.
A codec outside the five
VideoCodec
values cannot be a
VideoStream
.
```
mp4v
```
(MPEG-4 Part 2) is the one you will actually meet in robot datasets; it raises
```
RuntimeError: MP4 error: MP4 demux: Video track uses unsupported codec "mp4v"
```
. Asset mode does accept the file, but only partly: the blob chunk is emitted and the
```
VideoFrameReference
```
index chunk is skipped with a warning, because the frame timestamps cannot be read either — so you get the bytes and no timeline. Skipping that camera and recording the fact as a recording property (what the DROID loader does) is usually better than a timeline-less blob, and either beats failing a whole episode over one camera.
A
map
/
flat_map
that rewrites the time column must carry the field metadata over.
```
pa.field(...)
```
without
```
metadata=old_field.metadata
```
drops
```
rerun:kind: 'index'
```
, and the rebuilt chunk silently becomes static — no error, no timeline, and the samples land outside time entirely.
The trailing marker chunk is temporal but holds no samples, so any per-chunk logic keyed on
```
not chunk.is_static
```
will process it as if it were samples. Key on the
```
VideoStream:sample
```
column instead. See the retag above.
Asset mode is capped at ~2 GiB by Arrow's i32 offsets, and duplicates the file's bytes into the RRD.
```
mode="asset"
```
rejects both
```
chunk_by_gop=False
```
and
```
transcode=
```
with
```
ValueError
```
— those are stream-mode-only knobs.
```
timeline_type="timestamp"
```
on its own shifts nothing. Without a retag, the video sits at the epoch.
The reader emits compressed samples, never pixels. Thumbnails, CLIP embeddings, or anything else needing RGB have to decode the file separately (OpenCV, PyAV); the
```
VideoStream
```
chunks cannot supply them.
Every
```
stream()
```
call re-decodes the file from scratch, and every terminal call re-runs the pipeline.
```
collect()
```
once when you need more than one pass.
Handle the static codec chunk explicitly in any
```
map
```
/
```
flat_map
```
— it has no timeline and no samples, and blind indexing into a time column will fail on it.
Samples before the first keyframe are rejected in stream mode (a decoder cannot start mid-GOP); use asset mode for such a file.

错误在首次拉取时才会暴露，而非构造阶段。构造函数仅检查文件是否存在并验证参数；
```
stream()
```
仅构建惰性流水线且始终成功。编解码器支持、关键帧布局和FFmpeg可用性会在流运行时检查，因此请包装消费操作（
```
to_chunks()
```
、
```
send_chunks
```
、迭代），而非
```
Mp4Reader(…)
```
调用。
五种
VideoCodec
之外的编解码器无法用于
VideoStream
。
```
mp4v
```
（MPEG-4 Part 2）是机器人数据集中常见的不支持编解码器；此时会抛出
```
RuntimeError: MP4 error: MP4 demux: Video track uses unsupported codec "mp4v"
```
。资产模式可接受该文件，但存在限制：会输出blob块，但会跳过
VideoFrameReference
索引块并发出警告，因为无法读取帧时间戳——因此仅能获取文件字节，无时间线。通常跳过该摄像头并将此情况记录为录制属性（如DROID加载器的做法），比存储无时间线的blob更好，也比因一个摄像头导致整个录制失败更优。
重写时间列的
map
/
flat_map
操作必须保留字段元数据。不带
```
metadata=old_field.metadata
```
的
```
pa.field(...)
```
会丢失
```
rerun:kind: 'index'
```
，重建后的块会被静默标记为静态块——无错误提示、无时间线，样本会被存储在时间线之外。
末尾的标记块是时间块但不包含样本，因此任何基于
```
not chunk.is_static
```
的逐块逻辑都会将其当作样本块处理。请改为基于
```
VideoStream:sample
```
列进行判断。参见上述重新标记函数。
资产模式的大小上限约为2 GiB，由Arrow的i32偏移量限制，且会将文件字节复制到RRD中。
```
mode="asset"
```
会拒绝
```
chunk_by_gop=False
```
和
```
transcode=
```
参数，抛出
```
ValueError
```
——这些仅适用于流模式。
单独使用
```
timeline_type="timestamp"
```
不会偏移时间。若不进行重新标记，视频会显示在纪元时间点。
阅读器输出的是压缩样本，而非像素。缩略图、CLIP嵌入或任何需要RGB数据的操作都需单独解码文件（如使用OpenCV、PyAV）；
```
VideoStream
```
块无法提供这些数据。
每次调用
```
stream()
```
都会从头解码文件，每次终端调用都会重新运行流水线。当需要多次处理时，请调用一次
```
collect()
```
。
在任何
```
map
```
/
```
flat_map
```
操作中需显式处理静态编解码器块——它无时间线也无样本，盲目索引时间列会失败。
第一个关键帧之前的样本会在流模式中被拒绝（解码器无法从GOP中间开始）；此类文件请使用资产模式。

References

参考资料

Canonical worked examples:
```
rerun_py/tests/integration/test_mp4_reader.py
```
(both modes,
```
chunk_by_gop
```
, entity paths,
```
timeline_type
```
, transcode transforms, and the error cases).
Rust core:
```
crates/store/re_mp4_reader/
```
(
```
stream.rs
```
for the GOP/transcode path,
```
asset.rs
```
for the blob+index path), with
```
crates/store/re_mp4_reader/tests/stream.rs
```
covering codec pairs and GOP spacing.
```
rerun-chunk-processing
```
(stream/lens mechanics),
```
rerun-data-model
```
(where video, calibration, and thumbnails belong in the recording).

标准示例：
```
rerun_py/tests/integration/test_mp4_reader.py
```
（涵盖两种模式、
```
chunk_by_gop
```
、实体路径、
```
timeline_type
```
、转码转换及错误场景）。
Rust核心实现：
```
crates/store/re_mp4_reader/
```
（
```
stream.rs
```
处理GOP/转码路径，
```
asset.rs
```
处理blob+索引路径），
```
crates/store/re_mp4_reader/tests/stream.rs
```
涵盖编解码器组合及GOP间隔。
```
rerun-chunk-processing
```
（流/透镜机制）、
```
rerun-data-model
```
（视频、校准数据和缩略图在录制文件中的存储位置）。