hyperframes-media

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

HyperFrames Media

HyperFrames 媒体

Create the audio and media assets a composition needs — voiceover (TTS), background music + sound effects, transcription, captions, background removal — then consume and animate that data in HTML. For placing assets into compositions, see

hyperframes-core

创建合成内容所需的音频和媒体资源——配音（TTS）、背景音乐和音效、转录、字幕、背景移除——然后在 HTML 中使用这些数据并为其制作动画。如需将资源放入合成内容，请参阅

hyperframes-core

。

The audio engine — one source for TTS · BGM · SFX

音频引擎——TTS · BGM · SFX 的唯一来源

Workflows do NOT hand-roll audio or vendor a copy. There is one engine — scripts/audio.mjs
— that takes a neutral

audio_request.json

and writes

audio_meta.json

(plus assets under

assets/voice|bgm|sfx

bash

undefined

工作流绝不应自行实现音频逻辑或引入副本。这里只有一个引擎——scripts/audio.mjs
——它接收中立的

audio_request.json

并写入

audio_meta.json

（以及

assets/voice|bgm|sfx

下的资源）：

bash

undefined

<MEDIA_DIR> = this skill's directory

node <MEDIA_DIR>/scripts/audio.mjs --request ./audio_request.json --hyperframes . --out ./audio_meta.json


All three capabilities degrade on **ONE switch** — whether a HeyGen credential is present (resolved from `$HEYGEN_API_KEY` / `$HYPERFRAMES_API_KEY` / `~/.heygen`, **not** the CLI):

| Capability | HeyGen credential present                          | absent                                               |
| ---------- | -------------------------------------------------- | ---------------------------------------------------- |
| TTS        | HeyGen Starfish REST (native word timestamps)      | → ElevenLabs → Kokoro (chain `transcribe` for words) |
| BGM        | HeyGen music **retrieval**                         | Lyria → MusicGen local **generation** (detached)     |
| SFX        | HeyGen sound-effects **retrieval** (min_score 0.4) | bundled 21-file library (`assets/sfx/`)              |

- **Request** (`audio_request.json`): `{ provider?, lang?, speed?, lines: [{ id, text, sfx?: [names] }], bgm: { mode?, query?, prompt? } }`. `id` joins each line back to the caller's model (a frame number, a scene id, …). `bgm.mode` = `retrieve | generate | none`; omit for auto (retrieve when credentialed, else generate). An **explicit** `retrieve` is strict — it skips rather than starting a detached generate (for callers with no `wait-bgm` step).
- **Output** (`audio_meta.json`, id-keyed): `{ tts_provider, voice_id, bgm, bgm_pending, …, voices: [{ id, path, duration_s, words }], sfx: [{ id, name, file, source, offset_s, duration_s, volume }], total_duration_s }`.
- `--only tts,bgm,sfx` runs a subset and **merges** into an existing `--out` (e.g. TTS+BGM early, SFX once cues exist).
- BGM generate is spawned **detached** (`bgm_pending: true`) — run `scripts/wait-bgm.mjs` before assembling.
- `scripts/heygen-tts.mjs` is a single-shot CLI over the same code (one text → wav + words) for when you just need HeyGen TTS without a request file.

Full flag list + the `audio_meta.json` schema live in the header of `scripts/audio.mjs`. The references below cover the provider details and edge cases behind each capability.

node <MEDIA_DIR>/scripts/audio.mjs --request ./audio_request.json --hyperframes . --out ./audio_meta.json


三项能力都通过**同一个开关**进行回退——取决于是否存在 HeyGen 凭据（从 `$HEYGEN_API_KEY` / `$HYPERFRAMES_API_KEY` / `~/.heygen` 解析，**而非** CLI）：

| 能力 | 存在 HeyGen 凭据 | 不存在 |
| ---------- | -------------------------------------------------- | ---------------------------------------------------- |
| TTS | HeyGen Starfish REST（原生词级时间戳） | → ElevenLabs → Kokoro（链式调用 `transcribe` 获取词级数据） |
| BGM | HeyGen 音乐**检索** | Lyria → MusicGen 本地**生成**（分离进程运行） |
| SFX | HeyGen 音效**检索**（min_score 0.4） | 内置的 21 文件库（`assets/sfx/`） |

- **请求**（`audio_request.json`）：`{ provider?, lang?, speed?, lines: [{ id, text, sfx?: [names] }], bgm: { mode?, query?, prompt? } }`。`id` 用于将每一行关联回调用方的模型（帧号、场景 id 等）。`bgm.mode` = `retrieve | generate | none`；省略则自动选择（有凭据时检索，否则生成）。显式指定 `retrieve` 时严格执行——直接跳过，不会启动分离的生成任务（适用于没有 `wait-bgm` 步骤的调用方）。
- **输出**（`audio_meta.json`，以 id 为键）：`{ tts_provider, voice_id, bgm, bgm_pending, …, voices: [{ id, path, duration_s, words }], sfx: [{ id, name, file, source, offset_s, duration_s, volume }], total_duration_s }`。
- `--only tts,bgm,sfx` 仅运行指定子集，并将结果**合并**到现有的 `--out` 中（例如先生成 TTS+BGM，有了音效提示后再生成 SFX）。
- BGM 生成任务会以分离进程启动（`bgm_pending: true`）——组装前运行 `scripts/wait-bgm.mjs`。
- `scripts/heygen-tts.mjs` 是基于同一代码的单次执行 CLI（一条文本 → wav + 词级数据），适用于只需通过 HeyGen 生成 TTS 而不使用请求文件的场景。

完整参数列表和 `audio_meta.json` 架构位于 `scripts/audio.mjs` 的文件头注释中。下面的参考资料介绍了各项能力的提供商细节和边界情况。

Preflight — show sign-in status before any audio

预检——生成任何音频前显示登录状态

Always run this before generating voice or BGM — inside a full workflow or a one-off "generate me a BGM/voiceover" request. No HeyGen credential is not a reason to silently fall back to local engines: first recommend signing in and let the user decide. Run the shared preflight and relay its output verbatim — don't improvise your own "missing key" prompt, and don't offer to write keys into a per-repo

.env

bash

npx hyperframes auth status

Signed in → it prints the account; proceed.
Not signed in (
```
exit 1
```
is expected here — "not signed in" is a normal state, not a failure) → it prints registration-first guidance. Recommend signing in:
```
npx hyperframes auth login
```
is browser OAuth — it signs in and creates an account (always available through this repo's CLI). To use an existing HeyGen API key (from app.heygen.com/settings/api), run
```
npx hyperframes auth login --api-key
```
— it saves to the shared
```
~/.heygen
```
(no per-repo
```
.env
```
). The output also lists the local engines voice/BGM will fall back to and a
```
pip
```
hint when deps are missing. Relay this output as-is — don't paraphrase it into your own wording. Then STOP and wait for the user to choose — sign in, or say "go" / "local" to continue offline — before generating anything. This is a real decision point, not a passing note: don't fold it into another question, and don't proceed past it on your own. (Exception: in autonomous / non-interactive mode, note the status and continue offline.)

npx hyperframes auth status --json

returns

{ configured, recommended_action, offline_engines }

for deterministic branching.

If the CLI can't run (not on PATH and
```
npx
```
can't fetch it) → still recommend signing in (
```
npx hyperframes auth login
```
) and STOP for the user's choice — don't treat "no credential" as a silent green light for local generation.

Credential resolution, full key priority, and the local-dependency list are in

references/requirements.md

生成语音或 BGM 前始终运行此预检——无论是在完整工作流还是一次性“为我生成 BGM/配音”请求中。 没有 HeyGen 凭据并不是静默回退到本地引擎的理由：首先建议登录，让用户自行决定。运行共享预检并逐字转述其输出——不要自行编写“缺少密钥”提示，也不要主动提出把密钥写入仓库专用

.env

：

bash

npx hyperframes auth status

已登录 → 输出账户信息；继续执行。
未登录（预期
```
exit 1
```
——这里出现
```
exit 1
```
是正常状态，不是故障）→ 输出优先注册的引导信息。建议登录：
```
npx hyperframes auth login
```
使用浏览器 OAuth——会完成登录并创建账户（此仓库的 CLI 始终提供此方式）。若要使用现有的 HeyGen API key（来自 app.heygen.com/settings/api），运行
```
npx hyperframes auth login --api-key
```
——它会将密钥保存到共享的
```
~/.heygen
```
（不会写入仓库专用
```
.env
```
）。输出还会列出语音/BGM 将回退到的本地引擎，以及依赖缺失时的
```
pip
```
提示。请原样转述此输出——不要改写成你自己的措辞。 然后停止并等待用户选择——登录，或说“go”/“local”以离线继续——然后才能生成任何内容。这是一个真正的决策点，不是附带说明：不要把它并入另一个问题，也不要自行继续。（例外：在自主 / 非交互模式下，注明状态并离线继续。）

npx hyperframes auth status --json

{ configured, recommended_action, offline_engines }

，用于确定性分支处理。

如果 CLI 无法运行（不在 PATH 中且
```
npx
```
也无法获取它）→ 仍应建议登录（
```
npx hyperframes auth login
```
）并停止等待用户选择——不要把“没有凭据”视为静默允许本地生成的信号。

凭据解析方式、完整的密钥优先级以及本地依赖列表请参阅

references/requirements.md

。

Provider chains (the detail behind the engine)

提供商链（引擎背后的详细机制）

TTS — first available provider wins (the engine, or

npx hyperframes tts "..."

Order	Provider	Detected when	Word timestamps
1	HeyGen (Starfish)	`$HEYGEN_API_KEY` / `hyperframes auth login`	Yes, native — pass `--words narration.words.json` to capture
2	ElevenLabs	`$ELEVENLABS_API_KEY` set	No — chain `transcribe` after
3	Kokoro-82M (local, 54 voices)	always (no key required)	No — chain `transcribe` after

The published
hyperframes tts
CLI is often the local-only build (its
--help
says "Kokoro-82M", no
--provider
/
--words
) and silently falls back to Kokoro even with
$HEYGEN_API_KEY
set. That is why the engine's HeyGen path is the self-contained
scripts/heygen-tts.mjs
(REST), NOT the CLI; the CLI is used only for the Kokoro path. See
references/tts.md
.

BGM & SFX — by default retrieved from the HeyGen audio library (

/v3/audio/sounds

), same credential as HeyGen TTS, with the no-credential fallback from the switch above:

Asset	HeyGen `type`	Lands in	Fallback (no credential)
BGM	`music`	`assets/bgm/track.mp3` (retrieve) · `track.wav` (generate)	Lyria / MusicGen generation
SFX	`sound_effects` (min_score 0.4)	`assets/sfx/<slug>.mp3`	bundled 21-file library ( `assets/sfx/*` + `manifest.json` )

See

references/bgm.md

and

references/sfx.md

TTS——采用第一个可用的提供商（引擎或

npx hyperframes tts "..."

）：

顺序	提供商	检测条件	词级时间戳
1	HeyGen（Starfish）	设置了 `$HEYGEN_API_KEY` / `hyperframes auth login`	有，原生支持——传入 `--words narration.words.json` 以捕获词级数据
2	ElevenLabs	设置了 `$ELEVENLABS_API_KEY`	无——之后链式调用 `transcribe`
3	Kokoro-82M（本地，54 种语音）	始终可用（无需密钥）	无——之后链式调用 `transcribe`

已发布的
hyperframes tts
CLI 经常是仅支持本地引擎的构建版本（其
--help
显示“Kokoro-82M”，且没有
--provider
/
--words
），即使设置了
$HEYGEN_API_KEY
也会静默回退到 Kokoro。因此，HeyGen 路径使用自包含的
scripts/heygen-tts.mjs
（REST），而不是 CLI；CLI 仅用于 Kokoro 路径。请参阅
references/tts.md
。

BGM 与 SFX——默认从 HeyGen 音频库检索（

/v3/audio/sounds

），使用与 HeyGen TTS 相同的凭据，并采用上述无凭据回退方案：

资源	HeyGen `type`	保存位置	无凭据时的回退方案
BGM	`music`	`assets/bgm/track.mp3` （检索）· `track.wav` （生成）	Lyria / MusicGen 生成
SFX	`sound_effects` （min_score 0.4）	`assets/sfx/<slug>.mp3`	内置的 21 文件库（ `assets/sfx/*` + `manifest.json` ）

请参阅

references/bgm.md

和

references/sfx.md

。

Routing

路由

Task	Read
The audio engine — request/meta schema, `--only` , the switch	`scripts/audio.mjs` (header comment)
`npx hyperframes tts` / `heygen-tts.mjs` — providers, voices, words	`references/tts.md`
BGM — HeyGen retrieval + local Lyria / MusicGen generation	`references/bgm.md`
SFX — HeyGen retrieval (min_score 0.4) + bundled local library	`references/sfx.md`
`npx hyperframes transcribe` — Whisper, model rules, output shape	`references/transcribe.md`
`npx hyperframes remove-background` — transparent cutouts	`references/remove-background.md`
TTS → transcription → captions (no recorded voiceover)	`references/tts-to-captions.md`
Caption authoring — style detection, layout, word grouping, exit	`references/captions/authoring.md`
Transcript handling — input formats, quality gates, cleanup, APIs	`references/captions/transcript-handling.md`
Caption motion — karaoke, marker effects, audio-reactive	`references/captions/motion.md`
Model caches, system dependencies, troubleshooting	`references/requirements.md`

任务	查阅
音频引擎——请求/meta 架构、 `--only` 、切换开关	`scripts/audio.mjs` （文件头注释）
`npx hyperframes tts` / `heygen-tts.mjs` ——提供商、语音、词级数据	`references/tts.md`
BGM——HeyGen 检索 + 本地 Lyria / MusicGen 生成	`references/bgm.md`
SFX——HeyGen 检索（min_score 0.4）+ 内置本地库	`references/sfx.md`
`npx hyperframes transcribe` ——Whisper、模型规则、输出结构	`references/transcribe.md`
`npx hyperframes remove-background` ——透明抠图	`references/remove-background.md`
TTS → 转录 → 字幕（没有录音版配音）	`references/tts-to-captions.md`
字幕编写——样式检测、布局、词分组、退出	`references/captions/authoring.md`
转录文本处理——输入格式、质量门槛、清理、API	`references/captions/transcript-handling.md`
字幕动画——卡拉 OK、标记效果、音频响应	`references/captions/motion.md`
模型缓存、系统依赖、故障排查	`references/requirements.md`

Non-negotiable rules

不可协商的规则

One engine, no vendored copies. Produce audio via
```
scripts/audio.mjs
```
(or
```
heygen-tts.mjs
```
for one-shot HeyGen TTS). Don't re-implement TTS/BGM/SFX inside a workflow — write an
```
audio_request.json
```
adapter and call the engine.
"HeyGen available" = a resolvable credential, not the CLI. The whole switch keys off
```
heygenCredential()
```
; the published
```
hyperframes tts
```
may be Kokoro-only, and there is no
```
hyperframes bgm
```
/
```
hyperframes sfx
```
command at all.
Voice IDs are provider-specific.
```
am_michael
```
is Kokoro-only; HeyGen UUIDs don't work on Kokoro. If you pass
```
--voice
```
, also pin
```
--provider
```
to avoid silent provider drift when the user's env changes.
Always pass
--model
to
transcribe
. The CLI default
```
small.en
```
silently translates non-English audio. See
```
references/transcribe.md
```
→ "Language Rule".
HeyGen returns word timestamps; ElevenLabs / Kokoro do not. The engine chains
```
transcribe
```
automatically for the latter two; standalone, pass
```
--words
```
to HeyGen or run
```
transcribe
```
against the audio file.
Captions consume the flat word-array format with
```
{ id, text, start, end }
```
. See
```
references/transcribe.md
```
→ "Output Shape".
remove-background --background-output
is hole-cut, not inpainted. For "scene without the person", a different tool is needed. See
```
references/remove-background.md
```
→ "When NOT the right tool".
BGM/SFX default to HeyGen retrieval; the no-credential fallback is generation (BGM) or the bundled library (SFX).
```
/audio/sounds
```
ranks by a text query — name effects concretely (
```
glass shatter
```
, not
```
dramatic sound
```
); a no-match skips, never blocks the render. SFX sit at volume ~0.35 under voice + BGM. See
```
references/sfx.md
```
/
```
references/bgm.md
```
.
Treat workflow caption HTML as generated output. For preset-backed videos, the reusable skin source lives at
```
.hyperframes/caption-skin.html
```
and the workflow script writes
```
compositions/captions.html
```
; do not edit generated
```
compositions/captions.html
```
to fix the skin. Rebuild via the workflow's
```
captions.mjs
```
, or use that workflow's explicit overrides mechanism when present.

单一引擎，不复制代码。 通过
```
scripts/audio.mjs
```
生成音频（若只需一次性 HeyGen TTS，则使用
```
heygen-tts.mjs
```
）。不要在工作流中重新实现 TTS/BGM/SFX——应编写
```
audio_request.json
```
适配器并调用引擎。
“HeyGen 可用”= 可解析的凭据，而不是 CLI。 整个切换开关都基于
```
heygenCredential()
```
；已发布的
```
hyperframes tts
```
可能仅支持 Kokoro，而且根本不存在
```
hyperframes bgm
```
/
```
hyperframes sfx
```
命令。
语音 ID 取决于提供商。
```
am_michael
```
仅适用于 Kokoro；HeyGen UUID 无法用于 Kokoro。如果传入
```
--voice
```
，还应固定
```
--provider
```
，以免用户环境变化时静默切换到其他提供商。
调用
transcribe
时始终传入
--model
。 CLI 默认值
```
small.en
```
会将非英语音频静默翻译。请参阅
```
references/transcribe.md
```
→“Language Rule”。
HeyGen 返回词级时间戳；ElevenLabs / Kokoro 不返回。 引擎会自动为后两者链式调用
```
transcribe
```
；单独使用时，请向 HeyGen 传入
```
--words
```
，或针对音频文件运行
```
transcribe
```
。
字幕使用扁平词数组格式，即
```
{ id, text, start, end }
```
。请参阅
```
references/transcribe.md
```
→“Output Shape”。
remove-background --background-output
执行的是孔洞抠除，而不是图像修复。对于“不含人物的场景”，需要使用其他工具。请参阅
```
references/remove-background.md
```
→“When NOT the right tool”。
BGM/SFX 默认从 HeyGen 检索；无凭据时的回退方案是生成 BGM 或使用内置 SFX 库。
```
/audio/sounds
```
根据文本查询进行排序——请具体命名效果（
```
glass shatter
```
，而不是
```
dramatic sound
```
）；无匹配时会直接跳过，绝不会阻止渲染。SFX 在语音和 BGM 之下以约 0.35 的音量播放。请参阅
```
references/sfx.md
```
/
```
references/bgm.md
```
。
将工作流字幕 HTML 视为生成产物。 对于使用预设的视频，可复用的皮肤源文件位于
```
.hyperframes/caption-skin.html
```
，工作流脚本会写入
```
compositions/captions.html
```
；不要通过编辑生成的
```
compositions/captions.html
```
来修复皮肤。请通过工作流的
```
captions.mjs
```
重新构建，或在工作流提供显式覆盖机制时使用该机制。