edit-footage-with-code

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Edit footage with code

用代码剪辑素材

An agent can cut footage well when it stops trying to do everything at once. The people getting usable edits work in the same stages, whatever engine or language they use: read the take as text, agree a plan, cut the speech, lock the audio, then add picture on top, sound last. One prompt on long raw footage is the most repeated failure.

Which engine matters less than this order. Cut with ffmpeg and word timings. Add overlays in whichever engine the project already uses.

Situation	Use
A plain cut with few graphics	ffmpeg only: stop after step 5
Graphics on a cut, HTML-first, agent does the checks	HyperFrames: references/hyperframes.md
React codebase, brand system, or data-driven batches	Remotion: references/remotion.md
A person will finish in Resolve or Premiere	Any of the above, plus the FCPXML from step 5

当 Agent 不再试图一次完成所有事情时，就能把素材剪得很好。无论使用什么引擎或语言，那些能产出可用成片的人都会按相同的阶段推进：先把拍摄内容读取为文本、商定方案、剪掉语音、锁定音频，然后再叠加画面，最后处理声音。对长篇原始素材只使用一句提示词，是最常见的失败方式。

使用哪个引擎没有这个顺序重要。用 ffmpeg 配合单词时间戳完成剪切。项目原本使用哪个引擎，就用哪个引擎添加叠加层。

情况	使用方案
只需粗剪，图形较少	仅使用 ffmpeg：在第 5 步后停止
在粗剪上添加图形，采用 HTML 优先方案，并由 Agent 执行检查	HyperFrames：references/hyperframes.md
React 代码库、品牌系统或数据驱动的批处理任务	Remotion：references/remotion.md
后续由人员在 Resolve 或 Premiere 中完成	以上任一方案，再加第 5 步生成的 FCPXML

Rules that do not bend

不可妥协的规则

Never overwrite the source. Write to new files.
Cut only in the silence between words, at the quietest 10 ms slice. Never inside a word.
Word timestamps say which words, not where to cut. ASR stretches words across pauses. On our test clip, gaps between words found 1 of 13 real pauses.
No plan, no cut. Show the plan and wait for a yes, unless the person asked for no stops.
Spoken cut first, nothing else until it is locked: no captions, no graphics, no music.
Every graphic, caption and sound keys to a word, looked up in a file, never estimated.
Green checks are not approval. Look at frames. A person listens.
Show numbers, names and prices exactly as spoken. Never invent a stat, a testimonial or a piece of UI.

**绝不覆盖源文件。**始终写入新文件。
只能在词语之间的静音处剪切，并选择音量最低的 10 ms 区间。绝不在单词内部剪切。
**单词时间戳只能说明有哪些词，不能说明在哪里剪切。**ASR 会让单词的时长跨过停顿。在我们的测试片段中，通过单词间隔只找到了 13 个真实停顿中的 1 个。
**没有计划就不剪切。**先展示计划并等待确认，除非对方明确要求全程无需停下确认。
**先剪语音，在锁定之前不做其他任何事情：**不加字幕、图形或音乐。
每个图形、字幕和声音都要关联到具体单词，从文件中查找，绝不靠估算。
**绿色检查通过并不等于获得认可。**要查看画面帧，最后还要由人来听。
**画面中的数字、名称和价格必须与口播完全一致。**绝不编造统计数据、客户评价或 UI 元素。

The loop

工作流程

1. Intake

1. 素材导入

bash

ffprobe -v error -show_entries format=duration:stream=codec_type,codec_name,width,height,r_frame_rate,avg_frame_rate,color_transfer -of json take.mp4

The two rates differ at all (phones run slow: 29.99 drifts up to 0.16 s over 12 minutes), or the rate is unlike the project's: conform first with
```
ffmpeg -i take.mov -vf fps=30,format=yuv420p -fps_mode cfr -c:a copy take.mp4
```
. Never
```
-r 30
```
.
```
color_transfer
```
of HLG or PQ: phone HDR. Convert to SDR before anything, and do not grade your way out.
Read
```
rules.md
```
and
```
style/<format>.md
```
if they exist. No style file on a new format: write one from 3-5 references (references/notes-and-rules.md).
A short video from a long ramble is the hardest case. If the person can still shoot, a short script beats carving one out.

bash

ffprobe -v error -show_entries format=duration:stream=codec_type,codec_name,width,height,r_frame_rate,avg_frame_rate,color_transfer -of json take.mp4

只要两个帧率有任何差异（手机拍摄可能会变慢：29.99 在 12 分钟内最多漂移 0.16 秒），或素材帧率与项目不同，就先执行以下命令统一格式：
```
ffmpeg -i take.mov -vf fps=30,format=yuv420p -fps_mode cfr -c:a copy take.mp4
```
。绝不要使用
```
-r 30
```
。
```
color_transfer
```
为 HLG 或 PQ：这是手机拍摄的 HDR 素材。在处理其他内容之前，先转换为 SDR，不要试图通过调色掩盖问题。
如果存在
```
rules.md
```
和
```
style/<format>.md
```
，则读取它们。新格式没有样式文件时，根据 3-5 个参考样例编写一个：references/notes-and-rules.md。
从一段长时间漫谈中剪出短视频，是最难的情况。如果对方还能补拍，短脚本胜过长篇漫谈中硬剪出一个短视频。

2. Read the take

2. 读取这次拍摄

bash

npx hyperframes transcribe take.mp4 -d edit --model small.en --json     # or any word-level JSON: [{text,start,end}]
python3 <skill>/scripts/analyze_take.py take.mp4 --words edit/transcript.json --out edit

bash

npx hyperframes transcribe take.mp4 -d edit --model small.en --json     # or any word-level JSON: [{text,start,end}]
python3 <skill>/scripts/analyze_take.py take.mp4 --words edit/transcript.json --out edit

with a local Whisper-family model, always make this second pass: it is what exposes retakes

python3 <skill>/scripts/transcribe_islands.py take.mp4 --take-json edit/take.json --out edit/islands python3 <skill>/scripts/analyze_take.py take.mp4 --words edit/islands/words.json --punct-from edit/transcript.json --out edit


Read `edit/review.md`, then read the whole transcript yourself. The script measures this file's room and voice, finds real pauses from the audio, keeps cuts clear of slates, fades and camera changes in the picture, and lists retake and filler candidates. It writes a draft `edit/edl.json`. **It is a draft.** It only cuts a retake when the repeat covers most of the earlier attempt, because parallel wording ("trying to go to the moon... trying to go to the space station") looks like a retake and is not. It never cuts fillers unless told to. A false cut is worse than a missed one.

Why two passes: Whisper tidies repeats into one clean sentence and stretches words across pauses. Given one stretch of speech at a time, it cannot. On our test clip the island pass raised pauses visible in word timings from 1 to 12 of 13, cut badly placed words from 23 to 2, and exposed a false start and a real stutter the first pass had hidden. It took about a minute and a half for an 85 s clip, so skip it on footage you know is clean or when using an engine that keeps fillers.

Before cutting a retake or a stutter, transcribe a short stretch around it alone and check its energy. Even the island pass can place words badly inside a long stretch, and ASR writes "you know" over an "um". The script's retake boundaries are a starting point.

Keep these rules in mind while reading:

- Prefer a word-level engine that keeps fillers when an API is allowed (ElevenLabs Scribe). Whisper-family models often drop um and uh, so listen before believing the transcript is clean. A fresh transcript of the cut can also reveal stutters the first one hid.
- Take text from the script if there is one. Keep only the times from ASR.
- Do not assume the last take is best. In one worked example it won 1 of 4 scenes. Choose on completeness and cleanliness, and disqualify a take with a dead pause mid-sentence.
- Look at frames only at decision points. Reading a 12 KB transcript beats burning tokens on every frame.
- Several speakers: separate them before choosing content.


阅读 `edit/review.md`，然后亲自通读整份 transcript。脚本会测量这段素材的房间声学和人声，从音频中找出真实停顿，让剪切点避开画面中的打板、淡变和机位切换，并列出重录和口头填充词候选项。它会生成一份草稿 `edit/edl.json`。**它只是一份草稿。**只有在后一次重复内容覆盖前一次尝试的大部分时，它才会剪掉该重录，因为措辞平行但语义不同的内容（“trying to go to the moon... trying to go to the space station”）看起来像重录，实际却不是。除非得到明确指示，否则它绝不会剪掉口头填充词。错误剪切比漏剪更糟糕。

之所以需要两轮处理：Whisper 会把重复内容整理成一句干净的话，并让单词时间戳跨过停顿。如果每次只提供一段连续语音，它就无法做到这一点。在我们的测试片段中，island pass 让单词时间戳中可见的停顿从 13 个中的 1 个增加到 12 个，把位置切错的单词从 23 个减少到 2 个，并发现了第一轮转写隐藏的一次错误起拍和一处真实口吃。处理一个 85 秒的片段大约用了 1 分半钟，因此对于已知干净的素材，或使用能够保留填充词的引擎时，可以跳过这一轮。

在剪掉重录或口吃之前，单独转写其前后的一小段，并检查音频能量。即便是 island pass，也可能在较长的连续语音中错误放置单词，而 ASR 可能用“you know”替代“um”。脚本给出的重录边界只是起点。

读取素材时需要遵守以下规则：

- 如果 API 允许，优先使用能够保留填充词的词级引擎（ElevenLabs Scribe）。Whisper 系列模型经常丢弃 um 和 uh，因此不要仅凭 transcript 就认为内容干净，应先听一遍。对粗剪重新转写，也能发现第一次转写中隐藏的口吃。
- 如果有口播稿，文本以口播稿为准，只采用 ASR 提供的时间戳。
- 不要假定最后一条素材最好。在一个实例中，最后一条素材只在 4 个场景中胜出 1 个。应根据完整性和干净程度进行选择，并排除在句子中间存在长时间死寂的素材。
- 只在需要做决定时查看画面帧。阅读一份 12 KB 的 transcript，比逐帧查看并消耗更多 token 更划算。
- 有多位说话人时，先区分说话人，再选择内容。

3. Propose, then wait

3. 提出方案，然后等待确认

Write 4-8 plain sentences: shape, takes chosen, what gets cut, target length, what stays for later. List every removed pause that sits right before a number or a punchline, since no detector can tell a deliberate beat from a stumble. Stop until the person says yes.

用 4-8 句直白的话写明：成片结构、选用了哪些素材、剪掉了什么、目标时长，以及哪些内容留到后续处理。逐一列出所有紧挨数字或包袱之前、且已被删除的停顿，因为任何检测器都无法区分刻意留白与磕巴重说。在对方明确同意之前停止操作。

4. Write the edit as a file

4. 将剪辑写成文件

Change

in

and

out

edit/edl.json

by hand. Write the reason for each big decision next to it. Choose ranges by word, assemble by story beat rather than file order, and keep the strongest line for a cold open if the format wants one. For multi-take jobs, use one sub-agent per scene plus a verifier.

Defaults for every number, with what changes them: references/numbers.md.

手动修改

edit/edl.json

中的

in

和

out

。在每项重大决策旁写明原因。根据词语确定片段范围，按叙事节拍而不是文件顺序进行拼接；如果格式需要冷开场，则把最有力的一句放进去。对于多条素材的任务，每个场景使用一个 sub-agent，并配备一个 verifier。

所有数值的默认值，以及会影响这些数值的因素：references/numbers.md。

5. Rough cut, then lock the sound

5. 先做粗剪，再锁定声音

bash

python3 <skill>/scripts/build_cut.py edit/edl.json --take take.mp4    # writes cut.sh, clips.html, Cut.tsx, cut.fcpxml
sh edit/cut.sh                                                        # edit/acut.mp4, about a minute
npx hyperframes transcribe edit/acut.mp4 -d edit/heard --model small.en --json
python3 <skill>/scripts/qa_cut.py edit/acut.mp4 --edl edit/edl.json --heard edit/heard/transcript.json --intended edit/intended.json --out qa --no-loudness

Read
```
word_diff
```
and
```
doubled_words
```
in
```
qa/qa.json
```
. ASR invents some stutters and hides others, so settle each one by transcribing that stretch alone and reading its energy.
Trust
hot_joins
, not the transcript, for clipped words. On our test clip, cutting 140 ms of speech at every join still matched the clean cut's transcript at 99.1%, because ASR fills in word ends. The audio check at the join caught 12 of 13 cuts, against 0 of 13 on the clean version.
Look at
```
qa/joins.jpg
```
. It catches what audio checks cannot: on our test clip, a slate flashed for one frame at a join with every audio check green.
A flash near a join: move that segment's
```
in
```
or
```
out
```
by a few frames, then rerun
```
build_cut.py
```
, which recomputes everything else.
Use the one filter graph in
```
cut.sh
```
. On our test clip, per-segment AAC extracts joined with
```
-c copy
```
drifted +300 ms over 14 cuts, about 21 ms per cut (@hassancs91 measured 15-20 ms). The filter graph matched the edit list to the frame.
A join with no silence to cut in (two words run together) stays flagged. Check its levels and transcript, then pass
```
--accept-join <seconds>
```
, and list it for the person to hear first. Give that join its own fade with
```
fade_in_ms
```
or
```
fade_out_ms
```
on the segment.
Give the person a way to review: the cut, the list of what was removed and why, the joins sheet. They listen to the joins. Move on only when the audio flows.
If joins keep sounding wrong, stop re-prompting. Hand over
```
cut.fcpxml
```
for a hand trim in Resolve or Final Cut. @ladprofit, who publishes the best numbers we found, still trims jump cuts by hand.

bash

python3 <skill>/scripts/build_cut.py edit/edl.json --take take.mp4    # writes cut.sh, clips.html, Cut.tsx, cut.fcpxml
sh edit/cut.sh                                                        # edit/acut.mp4, about a minute
npx hyperframes transcribe edit/acut.mp4 -d edit/heard --model small.en --json
python3 <skill>/scripts/qa_cut.py edit/acut.mp4 --edl edit/edl.json --heard edit/heard/transcript.json --intended edit/intended.json --out qa --no-loudness

阅读
```
qa/qa.json
```
中的
```
word_diff
```
和
```
doubled_words
```
。ASR 有时会凭空生成口吃，也会隐藏口吃，因此应单独转写对应片段并查看音频能量，逐个确认。
**判断连接点是否截断单词时，要信任
```
hot_joins
```
，而不是 transcript。**在我们的测试片段中，即使每个连接点都切掉了 140 ms 的语音，生成的 transcript 仍有 99.1% 与干净成片一致，因为 ASR 会补全词尾。连接处的音频检查捕获了 13 个剪切点中的 12 个，而干净版本中 13 个均未捕获。
查看
```
qa/joins.jpg
```
。它能发现音频检查无法发现的问题：在我们的测试片段中，某个连接点有一帧闪过打板画面，而所有音频检查都显示正常。
如果连接点附近有闪帧，将该片段的
```
in
```
或
```
out
```
移动几帧，然后重新运行
```
build_cut.py
```
，它会重新计算其他所有内容。
使用
```
cut.sh
```
中的统一滤镜图。在我们的测试片段中，分段提取 AAC 后再使用
```
-c copy
```
拼接，在 14 个剪切点上累计漂移了 +300 ms，平均每个剪切点约 21 ms（@hassancs91 测得 15-20 ms）。滤镜图则能让剪辑清单逐帧对齐。
如果某个连接点没有可供剪切的静音，也就是两个词连在一起，它会继续被标记。检查该处的电平和 transcript，然后传入
```
--accept-join <seconds>
```
，并将其列为需要人工优先试听的项目。通过片段上的
```
fade_in_ms
```
或
```
fade_out_ms
```
，为该连接点添加独立的淡化效果。
为人工提供审核方式：粗剪、已删除内容及其原因的清单、连接点检查表。人工试听连接点，确认音频衔接顺畅后再继续。
如果连接点始终听起来不对，不要反复重新提示 Agent。将
```
cut.fcpxml
```
交付给人员，在 Resolve 或 Final Cut 中手动微调。发布最佳数据的 @ladprofit 也仍会手动修剪跳切。

6. Stills before motion

6. 先做静帧，再做动态效果

One still per scene on a contact sheet, or 4-6 key frames. Give a target image when you have one: it reduces rounds more than prompt wording. Fixing a still takes seconds, fixing an animation takes a render. For a new format, render a 15-25 s sample before the rest.

每个场景放一张静帧组成 contact sheet，或者选取 4-6 个关键帧。如果有目标参考图，应直接提供：与调整提示词措辞相比，这更能减少迭代轮次。修复静帧只需几秒，而修复动画则需要重新渲染。对于新格式，先渲染一段 15-25 秒的样片，再处理其余内容。

7. Picture on top

7. 在粗剪上叠加画面

Time everything from the cut's own transcript,
```
edit/heard/transcript.json
```
. Do not push single-pass word times through the edit list: on our test clip that lost 23 of 222 words. With an engine you trust (Scribe, or the island pass), computing output times from the list is exact and needs no second transcript; re-transcribe as the check.
Punch-ins need spare pixels: a 1.5x crop on 1080p goes soft. Render the cut larger than the composition if you plan to zoom much.
Keep one timing file and a cue sheet. No frame number written as a literal in a component. Round every time to whole frames from the frame number: a value rounded to 4 decimals can land a frame late.
A transcript's word times stay loose around pauses, even on the cut: one tester saw a word start about 1 s early. Compare every cue with the speech list from a fresh analysis of the cut, and for a cue that must land exactly, transcribe its window alone.
Build motion plates separately, then composite. Each element enters on its word. Do not reveal unrelated things at once, or cover the face.
Hold back. "Heavy motion, lots of text" is itself a named failure. Keywords on screen, sentences in captions. B-roll needs to add new information, or it goes.
Real UI only.

For source files:

video-asset-sources

. For named effects:

varg-video-edit-library

caption-styles

title-design

transition-craft

, the HyperFrames registry. Other people's kits, job by job: references/other-skills.md.

所有时间都应以粗剪自身的 transcript
```
edit/heard/transcript.json
```
为准。不要将单次转写得到的单词时间戳直接映射到剪辑清单：在我们的测试片段中，这样做导致 222 个单词里的 23 个丢失。如果使用值得信赖的引擎（Scribe 或 island pass），根据剪辑清单计算输出时间就是精确的，无需第二次转写；重新转写仅用于检查。
Punch-ins 需要预留像素：在 1080p 素材上做 1.5x crop 会让画面变糊。如果计划大幅缩放，应以高于最终构图尺寸的分辨率渲染粗剪。
始终使用一份时间轴文件和一个 cue sheet。组件中不要硬编码帧号。所有时间都应根据帧号取整到整帧；四舍五入到 4 位小数的时间可能晚落一帧。
即使在粗剪上，transcript 中的单词时间戳在停顿附近也仍可能不准确：一位测试者曾看到一个单词的起始时间早了约 1 秒。将每个 cue 与重新分析粗剪后得到的语音列表进行比较；对于必须精确落点的 cue，应单独转写对应时间窗口。
分别构建 motion plates，然后再进行合成。每个元素都在对应词语出现时入场。不要同时展示互不相关的内容，也不要遮挡人物面部。
保持克制。“大量动态、大量文字”本身就是一种明确的失败模式。屏幕上使用关键词，字幕中呈现完整句子。B-roll 必须提供新的信息，否则就应删除。
只展示真实 UI。

素材文件：

video-asset-sources

。指定特效：

varg-video-edit-library

、

caption-styles

、

title-design

、

transition-craft

、HyperFrames 注册表。其他人提供的工具包，可按任务选择：references/other-skills.md。

8. Sound last

8. 最后处理声音

Voice chain, then music under speech if the format wants it, then a few effects tied to visible events, then loudness measured on the final mix. Start with no whooshes: the agent's default is to add them, and people remove them. You cannot hear this. Report the numbers, and flag the music for a person. For anything beyond a plain voice-and-bed mix, use the

video-sound

skill, which builds and grades the mix with scripts.

先处理人声链；如果格式需要，再在语音下方添加音乐；然后添加少量与可见事件对应的音效；最后在最终混音上测量响度。默认先不要添加 whoosh 音效：Agent 默认会添加它们，而人工通常会将其删除。**你无法亲自听出这一点。**应报告数值，并将音乐标记出来交由人工试听。对于超出单纯人声加底声混音范围的内容，使用

video-sound

skill，它可以通过脚本构建并调整混音。

9. Check your own work, at most 3 passes

9. 检查自己的工作，最多进行 3 轮

bash

python3 <skill>/scripts/qa_cut.py out.mp4 --heard heard.json --intended edit/intended.json --out qa

It fails on: length off the edit list, joins that cut through speech, pauses over 0.6 s, an opening on silence, loudness off -14 LUFS, true peak over -1.5 dBTP, black frames, flash frames, under 95% of intended words heard, and a phrase said twice in a row. Change the limits with its flags for your format. It writes

contact.jpg

and

joins.jpg

, which you must look at. Also check every cut boundary, the first and last 2 s, names and prices on screen, and that nothing private shows. Then list what you could not verify.

bash

python3 <skill>/scripts/qa_cut.py out.mp4 --heard heard.json --intended edit/intended.json --out qa

以下情况会被判定为失败：时长与剪辑清单不符、连接点切穿语音、停顿超过 0.6 秒、开头是静音、响度偏离 -14 LUFS、真峰值超过 -1.5 dBTP、出现黑帧或闪帧、预期词语的听到比例低于 95%，以及同一句话连续说两遍。可以根据格式使用脚本参数调整限制。它会生成

contact.jpg

和

joins.jpg

，必须查看这两张图。此外，还要检查每个剪切边界、开头和结尾的 2 秒、屏幕上的名称和价格，并确认没有暴露任何私密内容。然后列出所有无法验证的内容。

10. The person watches. Then notes, rules, and stopping

10. 由人工观看，然后记录意见、制定规则并停止

Ask them to watch all of it with sound before anything changes. Then one batch of notes, each with a timestamp. Freeze what is approved. Turn each note into a rule. Stop when a full watch produces no note that would block publishing.

The note formats, the rule file, the style file and the stop rule: references/notes-and-rules.md.

在做任何修改之前，先请对方完整观看一遍并听取声音。随后集中收集一批意见，每条意见都附时间戳。锁定已经批准的部分，并将每条意见转化为一条规则。当完整观看一次后不再产生任何会阻碍发布的意见时，即可停止。

意见格式、规则文件、样式文件和停止规则：references/notes-and-rules.md。

11. Deliver

11. 交付

One full render, then check the exported file, not the preview. Hand over the MP4, the SRT, the edit file and the timeline. Never publish without the person's yes.

完整渲染一次，然后检查导出的文件，而不是预览文件。交付 MP4、SRT、剪辑文件和 time line。没有得到对方的明确同意，绝不发布。

Comparing cuts

比较不同剪辑方案

To compare versions or to tell whether a change made edits better, use a blind judge: references/judge.md and references/judge-prompt.md. On one pair it can say "clear", "leaning" or "too close". Its real strength is across many footages. Scripts own time and motion, and a person owns sound.

Know the bar: in the largest test we found, the best agent passed 15 of 56 real editing tasks, and human editors preferred the reference edit in 83.5% of judgments. Most failures were craft, not defects: agents check for faults, and editors judge polish, shot selection and rhythm. Passing every check here does not make a cut good.

要比较不同版本，或判断某项修改是否让剪辑变得更好，请使用盲评评委：references/judge.md 和 references/judge-prompt.md。对于一对版本，它可以给出“clear”“leaning”或“too close”的判断。它真正擅长的是跨大量素材进行比较。脚本负责时间与动态，人负责声音。

要知道合格的标准：在我们找到的最大规模测试中，表现最好的 Agent 仅通过了 56 项真实剪辑任务中的 15 项；在 83.5% 的评判中，人工剪辑师更偏好参考剪辑。大多数失败来自工艺打磨，而非功能缺陷：Agent 检查的是故障，剪辑师判断的是精致度、镜头选择和节奏。通过本文列出的每项检查，并不意味着剪辑就是好的。

Prompts

提示词

Ready prompts for the person to paste, and for you to hand to sub-agents: references/prompts.md. A

/goal

prompt that runs this whole loop: references/goal-prompt.md.

供人工直接粘贴或交给 sub-agent 的现成提示词：references/prompts.md。可运行完整流程的

/goal

提示词：references/goal-prompt.md。

When it goes wrong

出现问题时

Symptoms you can detect, with fixes: references/failures.md. Who said what, what we tested and what nobody has measured: references/sources.md.

可检测的症状及修复方法：references/failures.md。各方观点、测试内容以及尚未有人测量的部分：references/sources.md。