Back to Details

video-translate

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Video Translate

视频翻译

One take in, the same take in another language out, in the creator's own voice (ElevenLabs clones it from the take itself). The image stays the original: no lip-sync. Then 
/video-edit
edits the translated take like any other, with captions in the new language.
Talk to them in their language. 
<slug>
: the take's slug; 
<lang>
: the target language code (
en
, 
es
, 
fr
, 
de
, 
pt
, 
it
, 
ja
...), 
<from>
: the source one.
一次输入一条 take，即可输出同一 take 的另一语言版本，并使用创作者本人的声音（ElevenLabs 会从 take 本身克隆该声音）。画面保持原样，不做 lip-sync。随后，
/video-edit
会像编辑其他视频一样处理已配音的 take，并添加新语言字幕。
用目标语言与观众交流。
<slug>
：take 的 slug；
<lang>
：目标语言代码（
en
、
es
、
fr
、
de
、
pt
、
it
、
ja
...）；
<from>
：源语言。

1. The ElevenLabs key (once)

1. ElevenLabs key（仅需设置一次）

The key lives in 
video-agent/.env
, never in a chat message, a script or a commit:
bash
grep -c ELEVENLABS_API_KEY video-agent/.env 2>/dev/null
0
or no file:
  1. They create a key at elevenlabs.io/app/settings/api-keys (an ElevenLabs account; dubbing uses the credits of their plan, the free plan adds a watermark).
  2. They paste it into 
    video-agent/.env
    themselves, as one line: 
    ELEVENLABS_API_KEY=sk_...
    . Offer to create the empty file and open it. Never ask them to paste the key in the chat; if they do anyway, write it to the file and tell them they can regenerate it.
  3. video-agent/.env
    must be in the project's 
    .gitignore
    (add the line 
    video-agent/.env
    if it isn't).
Every command below loads it with 
set -a; . video-agent/.env; set +a
and never prints it.
key 保存在 
video-agent/.env
中，绝不能出现在聊天消息、脚本或 commit 中：
bash
grep -c ELEVENLABS_API_KEY video-agent/.env 2>/dev/null
0
或文件不存在：
  1. 用户在 elevenlabs.io/app/settings/api-keys 创建 key（需要 ElevenLabs 账号；dubbing 会使用其套餐的 credits，免费套餐会添加水印）。
  2. 用户自行将 key 以单行形式粘贴到 
    video-agent/.env
    ：
    ELEVENLABS_API_KEY=sk_...
    。可以主动创建空文件并打开它。绝不能让用户在聊天中粘贴 key；如果用户仍然粘贴了，则将其写入文件，并告知用户可以重新生成 key。
  3. video-agent/.env
    必须包含在项目的 
    .gitignore
    中；如果尚未包含，则添加 
    video-agent/.env
    。
下面的每条命令都使用 
set -a; . video-agent/.env; set +a
加载 key，并且绝不打印它。

2. What to translate

2. 要翻译的内容

Translate the take, not a finished video: a finished video has captions burned in the old language. Then video-edit cuts, captions and times everything on the new voice.
  • A take already edited (
    video-agent/edits/<slug>/edit.json
    exists): send only the kept parts, so no silence, stutter or retake is paid for. Make the plain cut with video-edit's ranges (from 
    video-agent/studio/
    ):
    bash
    SEL=$(node -e 'const e=require("./public/<slug>.edit.json");console.log(e.ranges.map(r=>`between(t,${r.fromMs/1000},${r.toMs/1000})`).join("+"))')
ffmpeg -y -i public/<slug>.mp4 -vf "select='$SEL',setpts=N/FRAME_RATE/TB" -af "aselect='$SEL',asetpts=N/SR/TB" -c:v libx264 -crf 18 -c:a aac -b:a 192k ../.tmp/<slug>-cut.mp4
    The source is 
    video-agent/.tmp/<slug>-cut.mp4
    .
  • A raw take: offer to run 
    /video-edit
    up to its cut list first (steps 1 to 4) for the same reason. In a hurry: send the raw take as it is.
应翻译 take，而不是已完成的视频：已完成的视频已经烧录了旧语言字幕。随后由 video-edit 按照新配音对内容进行剪切、添加字幕并处理时间轴。
  • 已编辑的 take（存在 
    video-agent/edits/<slug>/edit.json
    ）：只提交保留部分，以免为静音、卡顿或重拍付费。使用 video-edit 的 ranges（位于 
    video-agent/studio/
    ）进行普通裁剪：
    bash
    SEL=$(node -e 'const e=require("./public/<slug>.edit.json");console.log(e.ranges.map(r=>`between(t,${r.fromMs/1000},${r.toMs/1000})`).join("+"))')
ffmpeg -y -i public/<slug>.mp4 -vf "select='$SEL',setpts=N/FRAME_RATE/TB" -af "aselect='$SEL',asetpts=N/SR/TB" -c:v libx264 -crf 18 -c:a aac -b:a 192k ../.tmp/<slug>-cut.mp4
    源文件为 
    video-agent/.tmp/<slug>-cut.mp4
    。
  • 原始 take：建议先运行 
    /video-edit
    ，执行其步骤 1 至 4 并生成 cut list，原因相同。赶时间的话：按原样提交原始 take。

3. Estimate, then send

3. 先估算，再提交

bash
ffprobe -v error -show_entries format=duration -of csv=p=0 "<source>"
Show one line and wait for a yes: 
Dub <slug> fr → en · 0:52 of video · billed on your ElevenLabs credits for about 1 minute (check your plan's dubbing price on elevenlabs.io/pricing) · no lip-sync
.
Then send it (about 10 s per minute of upload):
bash
mkdir -p video-agent/.tmp
set -a; . video-agent/.env; set +a
curl -sS --fail-with-body -X POST https://api.elevenlabs.io/v1/dubbing \
  -H "xi-api-key: $ELEVENLABS_API_KEY" \
  -F "file=@<source>" -F "target_lang=<lang>" -F "source_lang=<from or auto>" \
  -F "num_speakers=1" -F "watermark=false" -F "name=<slug>-<lang>" \
  > video-agent/.tmp/<slug>-<lang>.job.json
cat video-agent/.tmp/<slug>-<lang>.job.json
It returns 
{"dubbing_id": "...", "expected_duration_sec": ...}
. Keep the id: if anything stops, polling again with it costs nothing. A 401 is a wrong key; a 402 or "quota" means no credits left on their plan; 
watermark
refused: send it again without that field.
bash
ffprobe -v error -show_entries format=duration -of csv=p=0 "<source>"
显示一行并等待用户回复 yes：
配音 <slug> fr → en · 视频时长 0:52 · 将从 ElevenLabs credits 中扣除约 1 分钟的费用（请在 elevenlabs.io/pricing 查看套餐的 dubbing 价格）· 不做 lip-sync
。
然后提交（上传处理时间约为每分钟视频 10 秒）：
bash
mkdir -p video-agent/.tmp
set -a; . video-agent/.env; set +a
curl -sS --fail-with-body -X POST https://api.elevenlabs.io/v1/dubbing \
  -H "xi-api-key: $ELEVENLABS_API_KEY" \
  -F "file=@<source>" -F "target_lang=<lang>" -F "source_lang=<from or auto>" \
  -F "num_speakers=1" -F "watermark=false" -F "name=<slug>-<lang>" \
  > video-agent/.tmp/<slug>-<lang>.job.json
cat video-agent/.tmp/<slug>-<lang>.job.json
返回 
{"dubbing_id": "...", "expected_duration_sec": ...}
。保留 id：如果流程中断，使用该 id 再次轮询不会产生额外费用。401 表示 key 错误；402 或 "quota" 表示套餐中没有剩余 credits；如果 
watermark
被拒绝，则去掉该字段后重新提交。

4. Wait, then download

4. 等待，然后下载

Poll every 15 s until 
status
is 
dubbed
(
failed
: show the error and stop). Usually 1 to 5 minutes for a short; run it in the background:
bash
set -a; . video-agent/.env; set +a
curl -sS --fail-with-body -H "xi-api-key: $ELEVENLABS_API_KEY" https://api.elevenlabs.io/v1/dubbing/<dubbing_id>
Then the dubbed file:
bash
set -a; . video-agent/.env; set +a
curl -sS --fail-with-body -H "xi-api-key: $ELEVENLABS_API_KEY" -o video-agent/.tmp/<slug>-<lang>.dub https://api.elevenlabs.io/v1/dubbing/<dubbing_id>/audio/<lang>
ffprobe -v error -show_entries stream=codec_type -of csv=p=0 video-agent/.tmp/<slug>-<lang>.dub
  • It has a 
    video
    stream: it is already the dubbed video, move it to 
    video-agent/takes/<slug>-<lang>.mp4
    .
  • Audio only: lay it back on the original image:
    bash
    ffmpeg -y -i "<source>" -i video-agent/.tmp/<slug>-<lang>.dub -map 0:v -map 1:a -c:v copy -c:a aac -b:a 192k -shortest video-agent/takes/<slug>-<lang>.mp4
Check it: same length as the source (within a second). Then ask them to listen to the first 10 seconds before editing (voice, accent, a wrong word).
每 15 秒轮询一次，直到 
status
为 
dubbed
（若为 
failed
：显示错误并停止）。短视频通常需要 1 到 5 分钟；在后台运行：
bash
set -a; . video-agent/.env; set +a
curl -sS --fail-with-body -H "xi-api-key: $ELEVENLABS_API_KEY" https://api.elevenlabs.io/v1/dubbing/<dubbing_id>
然后下载配音文件：
bash
set -a; . video-agent/.env; set +a
curl -sS --fail-with-body -H "xi-api-key: $ELEVENLABS_API_KEY" -o video-agent/.tmp/<slug>-<lang>.dub https://api.elevenlabs.io/v1/dubbing/<dubbing_id>/audio/<lang>
ffprobe -v error -show_entries stream=codec_type -of csv=p=0 video-agent/.tmp/<slug>-<lang>.dub
  • 如果包含 
    video
    stream，则它已经是配音后的视频，将其移动到 
    video-agent/takes/<slug>-<lang>.mp4
    。
  • 如果只有音频：将其重新合成到原始画面中：
    bash
    ffmpeg -y -i "<source>" -i video-agent/.tmp/<slug>-<lang>.dub -map 0:v -map 1:a -c:v copy -c:a aac -b:a 192k -shortest video-agent/takes/<slug>-<lang>.mp4
检查文件：长度应与源文件相同（误差在 1 秒以内）。然后请用户在编辑前试听前 10 秒（声音、口音、错误词语）。

5. Edit the translated take

5. 编辑翻译后的 take

/video-edit <slug>-<lang>
, and tell it:
  • The language changed: in 
    video-agent/studio/whisper-config.mjs
    , set 
    WHISPER_LANG
    to 
    <lang>
    for this transcription (English: 
    WHISPER_MODEL = "medium.en"
    works best; any other language: 
    "medium"
    ), and put it back after. The hook card, callouts and 
    post.md
    are written in 
    <lang>
    .
  • Same edit as the original when there is one: same hook card and callouts (translated), placed on the matching words of the new transcript.
  • The ask (CTA) in 
    <lang>
    : ask them once for its translation, or translate the one in voice.md and show it.
Name everything 
<slug>-<lang>
so both versions sit side by side. Then 
/video-publish <slug>-<lang>
on the account for that language.
Delete 
video-agent/.tmp/<slug>-*
at the end, except the job file if they may want to download the dub again.
运行 
/video-edit <slug>-<lang>
，并告知它：
  • 语言已更改：在 
    video-agent/studio/whisper-config.mjs
    中，将 
    WHISPER_LANG
    设置为 
    <lang>
    以用于本次转录（英语：
    WHISPER_MODEL = "medium.en"
    效果最好；其他语言使用 
    "medium"
    ），完成后恢复原设置。hook card、callouts 和 
    post.md
    均以 
    <lang>
    编写。
  • 如果原始版本已有编辑，则沿用相同编辑：使用相同的 hook card 和 callouts（翻译后），并放到新转录稿中对应的词语处。
  • <lang>
    中的 CTA（行动号召）：请用户提供一次翻译，或翻译 
    voice.md
    中现有的 CTA 并展示。
将所有内容命名为 
<slug>-<lang>
，让两个版本并排保存。然后在对应语言的账号上运行 
/video-publish <slug>-<lang>
。
最后删除 
video-agent/.tmp/<slug>-*
，但如果用户可能还想再次下载 dub 音频，则保留 job file。

Honest limits

限制说明

  • No lip-sync: the lips move in the original language. Fine with a lot of motion, a small face or quick cuts; noticeable on a long close-up.
  • The voice clone comes from the take: a noisy take or music under the voice gives a worse clone.
  • Machine translation: names, jargon and jokes can come out wrong. Ask them (or a native speaker) to check the transcript of the dubbed take before posting.
  • Dubbing costs ElevenLabs credits per minute, and the price depends on their plan. Trimming the take first is the cheapest saving.
  • 不做 lip-sync：嘴型仍按原始语言运动。画面动作较多、脸部较小或镜头切换较快时问题不大；长时间特写中会很明显。
  • 声音克隆来自 take 本身：如果 take 噪声较大，或人声下方有音乐，克隆效果会变差。
  • 机器翻译：姓名、术语和笑话可能译错。发布前，请让用户（或母语者）检查配音版 take 的转录稿。
  • Dubbing 按视频分钟数消耗 ElevenLabs credits，价格取决于其套餐。先裁剪 take 是最省钱的做法。