ui-recording-timeline

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

UI recording timeline

UI 录制时间线

Produces a self-contained folder with

index.html

and the video. The video fills the window. Below it are a one-line caption and a thin timeline (phase colors, paint ticks, hover thumbnails, scrubbing). Change regions flash on the video at each paint, and a paint list drawer opens with

Everything needed is in this skill directory:

```
scripts/timeline.py
```
: one CLI with all the steps. Dependencies are declared inline, and
```
uv run
```
installs them.
```
assets/template.html
```
: the page.
```
build
```
injects the data into it.

Requirements:

ffmpeg

ffprobe

and

uv

on PATH. Chrome or Chromium is needed only for

preview

Below,

means

uv run <this skill dir>/scripts/timeline.py

. Define it as a shell function, not a variable: zsh doesn't word-split variables. Use the absolute path of the skill directory:

bash

T(){ uv run "/abs/path/to/ui-recording-timeline/scripts/timeline.py" "$@"; }

WD

is the output folder. If the user's project has conventions for where research or collateral goes, follow them; otherwise put

WD

next to the video as

<video-stem>-timeline/

生成一个自包含文件夹，其中包含

index.html

和视频。视频铺满窗口。下方是一行简短说明和一条细时间线（阶段颜色、paint 刻度、悬停缩略图、拖动定位）。每次 paint 时，变化区域都会在视频上闪烁；按

可打开 paint 列表抽屉。

此 skill 目录中包含所有必需内容：

```
scripts/timeline.py
```
：一个完成全部步骤的 CLI。依赖项采用内联声明，并由
```
uv run
```
安装。
```
assets/template.html
```
：页面。
```
build
```
会将数据注入其中。

要求：PATH 中有

ffmpeg

ffprobe

和

uv

。仅

preview

需要 Chrome 或 Chromium。

下文中的

表示

uv run <this skill dir>/scripts/timeline.py

。请将其定义为 shell 函数，而不是变量，因为 zsh 不会对变量执行单词拆分。请使用 skill 目录的绝对路径：

bash

T(){ uv run "/abs/path/to/ui-recording-timeline/scripts/timeline.py" "$@"; }

WD

表示输出文件夹。如果用户的项目对调研或辅助材料的位置有约定，请遵循这些约定；否则，将

WD

放在视频旁边，命名为

<video-stem>-timeline/

。

Workflow

工作流

1. Extract and find candidate paints

1. 提取并查找候选 paint

bash

T extract /path/to/recording.mp4 WD     # copies video, writes meta.json (frame timestamps; VFR is fine)
T candidates WD                         # table of frames that differ from the previous frame

Each row has the frame index, time, % changed, kind (

change

full

cursor?

), and the normalized bbox. Small cursor-like changes are hidden (use

--all

to show them).

bash

T extract /path/to/recording.mp4 WD     # copies video, writes meta.json (frame timestamps; VFR is fine)
T candidates WD                         # table of frames that differ from the previous frame

每一行都包含帧索引、时间、变化百分比、类型（

change

full

cursor?

）和归一化边界框。较小的类光标变化会被隐藏（使用

--all

可显示）。

2. Look at the frames, then decide what counts as a paint

2. 查看各帧，然后判定哪些内容算作一次 paint

Never label from the table alone. Look at the frames:

bash

T sheet WD --frames 0,4,14,26,61 --name overview          # labelled contact sheets -> WD/sheets/*.jpg
T sheet WD --frames 18,19 --crop 0,0,0.25,0.15 --name zoom # zoom into a small change

Read the sheet images with the image viewer. Make sure to include the frame before each candidate, so you can see what changed.

Rules for choosing paints:

Is a paint: a visible content or layout change. That covers skeletons appearing, real content replacing skeletons, navigation to a blank page, a new page's first paint, a label or button-state change, and browser chrome that signals load state (the reload icon replacing the stop X).
Is not a paint: cursor movement, hover highlights, spinner rotation, caret blink, or a gradual shimmer on skeletons that are already there. Look at the bbox: tiny regions that wander across consecutive frames are the cursor.
Animations: a multi-frame animation (a check-mark animating, a panel sliding in) is one paint at its final frame. Give it
```
"base": <frame before the animation started>
```
so the diff compares the whole animation.
Minor paints: set
```
minor: true
```
on small in-place updates (a label, a button state, browser chrome). They get shorter ticks and muted text.
Phases: group paints into phases by screen or URL (for example: app shell, SSO gate, IdP, blank redirect, signed-in app, idle). The address bar in the frames usually tells you. Phases must cover 0 to the recording's duration with no gaps.
Idle phases: set
```
"idle": true
```
on stretches where nothing on screen changes. That always includes the tail after the last paint, and can include long waits on one unchanged screen. Idle phases are drawn hatched grey, not in a palette color: a saturated color would suggest activity, and dark grey makes the bar look like it ends early. Don't trim the idle tail; it's real time, and the video spans it.
Metrics: the first two appear in the caption, so put the most important first (for example "main content" and "visually complete"). Others, like time on the IdP or total blank time, go in the drawer.
Caveat: t=0 is the first frame of the recording, not navigation start. If the first frame already shows UI, say so in
```
notes
```
.

bash

T sheet WD --frames 0,4,14,26,61 --name overview          # labelled contact sheets -> WD/sheets/*.jpg
T sheet WD --frames 18,19 --crop 0,0,0.25,0.15 --name zoom # zoom into a small change

绝不要仅根据表格来添加标签。请查看这些帧：

使用图片查看器读取拼图。务必包含每个候选帧之前的帧，以便看清具体变化。

判定 paint 的规则：

是 paint： 可见内容或布局发生变化。这包括骨架屏出现、真实内容替换骨架屏、导航至空白页面、新页面的首次绘制、标签或按钮状态变化，以及能够表示加载状态的浏览器 chrome 变化（例如重新加载图标替换停止按钮 X）。
不是 paint： 光标移动、悬停高亮、spinner 旋转、插入符闪烁，或已存在的骨架屏逐渐闪烁。查看边界框：在连续帧中于小区域游走的变化就是光标。
动画： 多帧动画（例如勾选标记动画、面板滑入）在其最终帧算作一次 paint。为其设置
```
base: <动画开始前的帧>
```
，使差异检测比较整个动画。
次要 paint： 对小范围原位更新（标签、按钮状态、浏览器 chrome）设置
```
minor: true
```
。它们会显示为更短的刻度和更淡的文字。
阶段： 按屏幕或 URL 对 paint 分组（例如：应用 shell、SSO 门禁、IdP、空白重定向、已登录应用、闲置）。通常可以通过帧中的地址栏判断。阶段必须覆盖从 0 到录制时长的时间范围，且不能有空隙。
闲置阶段： 对屏幕上没有任何变化的时段设置
```
idle: true
```
。这始终包括最后一次 paint 之后的尾段，也可以包括同一不变画面上的长时间等待。闲置阶段以灰色斜线绘制，而不是使用调色板颜色：饱和色会暗示此处有活动，而深灰色会让时间线看起来像提前结束了。不要裁掉闲置尾段；它是真实时间，而且视频也覆盖该时段。
指标： 前两个指标会显示在说明行中，因此请把最重要的指标放在前面（例如“主要内容”和“视觉完成”）。其他指标，如处于 IdP 页面的时间或空白总时长，则放在抽屉中。
注意： t=0 是录制的第一帧，而不是导航开始时间。如果第一帧已经显示 UI，请在
```
notes
```
中说明。

3. Write

WD/paints.json

3. 编写

WD/paints.json

json

{
  "title": "vercel.com/vercel-labs with an SSO redirect",
  "notes": "Recording starts mid-navigation: the dashboard shell is already on screen at t=0.",
  "metrics": [
    {"value": "6.50 s", "label": "main content"},
    {"value": "8.37 s", "label": "visually complete"},
    {"value": "2.53 s", "label": "on Okta"}
  ],
  "phases": [
    {"start": 0.0,   "end": 0.802,  "name": "Dashboard (unauthenticated)"},
    {"start": 0.802, "end": 2.002,  "name": "Vercel SSO gate"},
    {"start": 2.002, "end": 8.468,  "name": "Okta and back"},
    {"start": 8.468, "end": 10.935, "name": "Idle", "idle": true}
  ],
  "paints": [
    {"frame": 0,  "label": "Dashboard shell: sidebar nav and header skeleton (recording start)"},
    {"frame": 4,  "label": "Projects and usage skeleton cards"},
    {"frame": 19, "label": "Team switcher changes to \"Select Team\"", "minor": true},
    {"frame": 87, "label": "Okta Verify check badge (animates 2.668 to 3.068 s)", "minor": true, "base": 74}
  ]
}

```
frame
```
is the decoded frame index from
```
candidates
```
. The time comes from the frame's real timestamp, so don't hand-compute it. Phase boundaries are in seconds: use the
```
t
```
of the paint that starts each phase, and the duration from
```
extract
```
for the last one.
Labels are short and concrete. Say what appeared, changed or disappeared, and quote on-screen text.
Optional per phase:
```
"color": "#hex"
```
. By default colors come from a palette.

json

{
  "title": "vercel.com/vercel-labs with an SSO redirect",
  "notes": "Recording starts mid-navigation: the dashboard shell is already on screen at t=0.",
  "metrics": [
    {"value": "6.50 s", "label": "main content"},
    {"value": "8.37 s", "label": "visually complete"},
    {"value": "2.53 s", "label": "on Okta"}
  ],
  "phases": [
    {"start": 0.0,   "end": 0.802,  "name": "Dashboard (unauthenticated)"},
    {"start": 0.802, "end": 2.002,  "name": "Vercel SSO gate"},
    {"start": 2.002, "end": 8.468,  "name": "Okta and back"},
    {"start": 8.468, "end": 10.935, "name": "Idle", "idle": true}
  ],
  "paints": [
    {"frame": 0,  "label": "Dashboard shell: sidebar nav and header skeleton (recording start)"},
    {"frame": 4,  "label": "Projects and usage skeleton cards"},
    {"frame": 19, "label": "Team switcher changes to \"Select Team\"", "minor": true},
    {"frame": 87, "label": "Okta Verify check badge (animates 2.668 to 3.068 s)", "minor": true, "base": 74}
  ]
}

```
frame
```
是
```
candidates
```
中解码后的帧索引。时间来自该帧的真实时间戳，因此不要手动计算。阶段边界以秒为单位：每个阶段使用开始该阶段的 paint 的
```
t
```
，最后一个阶段则使用
```
extract
```
输出的录制时长作为结束时间。
标签应简短且具体。说明出现、变化或消失的内容，并引用屏幕上的文字。
每个阶段可选设置
```
color: "#hex"
```
。默认颜色取自调色板。

4. Compute and check the change regions

4. 计算并检查变化区域

bash

T regions WD     # move-aware diff for every paint -> regions.json
T check WD       # boxes drawn on every paint frame -> WD/sheets/check-*.jpg

Look at every check sheet. Pink means painted, blue means moved, and thin grey shows where the content moved from. How the diff works (details in

compute_regions

Changed layout shapes are snapped to whole elements: words, icons, inputs, cards. Faint shimmer ends are included.
A shape found nearby in the previous frame, whose old spot was vacated, counts as moved. For example, when a sidebar item is inserted, the rows below it slide down. Same-offset moves are grouped.
Ambiguous repeated shapes (identical chevrons, rows) count as moved when they line up with an offset already found in the same frame.
The cursor is learned, not guessed from size. Cursor sprites (arrow, hand, text cursor) come from the
```
candidates
```
frames where only something small moved on its own. Only boxes containing a known sprite, in either frame, are dropped. Small real paints like a logo, an icon or a toolbar button are kept. So run
```
candidates
```
before
```
regions
```
.
Everything else is painted. That includes flat fills like a new backdrop.

If a box looks wrong:

First check the paint choice. A wrong
```
base
```
or a hover state inside the paint frame is the usual cause.
Only then consider tuning the constants at the top of
```
compute_regions
```
. They're tuned at 1710 px wide and scaled automatically:
```
EDGE
```
/
```
CHG
```
thresholds,
```
SEARCH
```
distance, the cursor size, and the grow tolerance.

bash

T regions WD     # move-aware diff for every paint -> regions.json
T check WD       # boxes drawn on every paint frame -> WD/sheets/check-*.jpg

查看每一张检查拼图。粉色表示已绘制，蓝色表示发生移动，灰色细线标出内容移动前的位置。差异检测的工作方式如下（详见

compute_regions

）：

发生变化的布局形状会吸附到完整元素上：单词、图标、输入框、卡片。微光的结束尾迹也会被纳入。
如果在前一帧附近找到一个形状，且其旧位置已经空出，则计为移动。例如，在侧边栏中插入一个项目时，下面的各行会向下移动。偏移量相同的移动会被归为一组。
对于有歧义的重复形状（相同的 chevron、列表行），如果它们与同一帧中已找到的某个偏移对齐，也计为移动。
光标是通过学习识别的，而不是根据大小猜测。箭头、手形、文本光标等 cursor sprite 来自
```
candidates
```
中仅有一个小对象自行移动的帧。如果任一帧中的框包含已知 sprite，就将其丢弃。logo、图标或工具栏按钮等小型真实绘制会被保留。因此必须先运行
```
candidates
```
，再运行
```
regions
```
。
其他所有内容都视为已绘制，包括新背景这样的纯色填充。

如果某个框看起来不对：

首先检查 paint 的选择。错误的
```
base
```
，或 paint 帧中包含 hover 状态，是最常见的原因。
只有在完成上述检查后，才考虑调整
```
compute_regions
```
顶部的常量。这些常量以宽度 1710 px 为基准进行调优，并会自动缩放：
```
EDGE
```
/
```
CHG
```
阈值、
```
SEARCH
```
距离、光标大小和扩展容差。

5. Build, preview, and open

5. 构建、预览并打开

bash

T build WD                    # paint thumbnails + WD/index.html
T preview WD --t 6.502        # headless screenshot with boxes frozen -> WD/preview.png
open WD/index.html

Read

preview.png

to verify the layout and that the overlay lines up with the video. Re-run

build

after any change to

paints.json

regions.json

. Run

regions

again only when the paint frames or bases change.

bash

T build WD                    # paint thumbnails + WD/index.html
T preview WD --t 6.502        # headless screenshot with boxes frozen -> WD/preview.png
open WD/index.html

读取

preview.png

，验证布局以及覆盖框是否与视频对齐。每次更改

paints.json

或

regions.json

后，都要重新运行

build

。仅当 paint 帧或其

base

发生变化时，才再次运行

regions

。

Page reference

页面参考

Keys:
```
←
```
/
```
→
```
step between paints,
```
space
```
plays,
```
d
```
toggles the change flashes,
```
l
```
opens the paint list drawer,
```
f
```
goes fullscreen. Clicking the video plays or pauses. Clicking or dragging the timeline scrubs, and hovering shows a thumbnail.
URL options:
```
#t=6.502
```
deep-links to a time, and
```
?hold
```
freezes the boxes, which helps with screenshots.
"Shown" is the time until the next paint.
The folder is portable:
```
index.html
```
,
```
recording.*
```
,
```
paints/
```
.
```
meta.json
```
,
```
paints.json
```
,
```
regions.json
```
and
```
sheets/
```
are inputs and debug output, safe to keep.

按键：
```
←
```
/
```
→
```
在各次 paint 之间切换，
```
space
```
播放，
```
d
```
切换变化闪烁，
```
l
```
打开 paint 列表抽屉，
```
f
```
进入全屏。点击视频可播放或暂停。点击或拖动时间线可进行定位，悬停时会显示缩略图。
URL 选项：
```
#t=6.502
```
可深链到指定时间；
```
?hold
```
会冻结覆盖框，便于截图。
“Shown”表示距离下一次 paint 所经过的时间。
该文件夹可移植，包含：
```
index.html
```
、
```
recording.*
```
、
```
paints/
```
。
```
meta.json
```
、
```
paints.json
```
、
```
regions.json
```
和
```
sheets/
```
是输入与调试输出，可以安全保留。

Reporting back

结果汇报

Summarize the phases, the key metrics, and anything notable (blank screens, layout shifts that the diff shows as moves, long waits). Give the path to

index.html

. Mention the t=0 caveat, and that times are accurate to ±1 frame.

总结各阶段、关键指标以及任何值得注意的情况（空白屏幕、差异检测显示为移动的布局变化、长时间等待）。提供

index.html

的路径。提及 t=0 的注意事项，并说明时间精度为 ±1 帧。

ui-recording-timeline

Original

Translation

UI recording timeline

UI 录制时间线

Workflow

工作流

1. Extract and find candidate paints

1. 提取并查找候选 paint

2. Look at the frames, then decide what counts as a paint

2. 查看各帧，然后判定哪些内容算作一次 paint

3. Write
`WD/paints.json`

3. 编写
`WD/paints.json`

4. Compute and check the change regions

4. 计算并检查变化区域

5. Build, preview, and open

5. 构建、预览并打开

Page reference

页面参考

Reporting back

结果汇报