Back to Details

short-translate

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Short Translate

Short Translate

One take in, the same take in another language out, in the creator's own voice (ElevenLabs clones it from the take itself). The image stays the original: no lip-sync. Then 
/short-editing
edits the translated take like any other, with captions in the new language.
Talk to them in their language. 
<slug>
: the take's slug; 
<lang>
: the target language code (
en
, 
es
, 
fr
, 
de
, 
pt
, 
it
, 
ja
...), 
<from>
: the source one.
一次输入一条 take，即可输出同一 take 的另一语言版本，并采用创作者本人的声音（ElevenLabs 会直接从该 take 克隆声音）。画面保留原样：不做 lip-sync。随后，
/short-editing
会像处理其他 take 一样编辑译后的版本，并添加新语言字幕。
用用户能理解的语言与其沟通。
<slug>
：take 的 slug；
<lang>
：目标语言代码（
en
、
es
、
fr
、
de
、
pt
、
it
、
ja
……），
<from>
：源语言。

1. The ElevenLabs key (once)

1. ElevenLabs API key（只需配置一次）

The key lives in 
video-agent/.env
, never in a chat message, a script or a commit:
bash
grep -c ELEVENLABS_API_KEY video-agent/.env 2>/dev/null
0
or no file:
  1. They create a key at elevenlabs.io/app/settings/api-keys (an ElevenLabs account; dubbing uses the credits of their plan, the free plan adds a watermark).
  2. They paste it into 
    video-agent/.env
    themselves, as one line: 
    ELEVENLABS_API_KEY=sk_...
    . Offer to create the empty file and open it. Never ask them to paste the key in the chat; if they do anyway, write it to the file and tell them they can regenerate it.
  3. video-agent/.env
    must be in the project's 
    .gitignore
    (add the line 
    video-agent/.env
    if it isn't).
Every command below loads it with 
set -a; . video-agent/.env; set +a
and never prints it.
API key 存放在 
video-agent/.env
中，绝不能出现在聊天消息、脚本或 commit 中：
bash
grep -c ELEVENLABS_API_KEY video-agent/.env 2>/dev/null
输出为 
0
或文件不存在：
  1. 用户需要在 elevenlabs.io/app/settings/api-keys 创建 API key（需要 ElevenLabs 账号；dubbing 会消耗其套餐 credits，免费套餐会添加水印）。
  2. 用户自行将 API key 粘贴到 
    video-agent/.env
    ，格式为一行：
    ELEVENLABS_API_KEY=sk_...
    。可以主动提出创建空文件并打开它。绝不能要求用户把 API key 粘贴到聊天中；如果用户仍然这样做，应将其写入文件，并告知用户可以重新生成。
  3. video-agent/.env
    必须位于项目的 
    .gitignore
    中；如果尚未加入，则添加 
    video-agent/.env
    这一行。
以下每条命令都通过 
set -a; . video-agent/.env; set +a
加载 API key，且绝不会将其打印出来。

2. What to translate

2. 要翻译的内容

Translate the take, not a finished video: a finished video has captions burned in the old language. Then short-editing cuts, captions and times everything on the new voice.
  • A take already edited (
    video-agent/edits/<slug>/edit.json
    exists): send only the kept parts, so no silence, stutter or retake is paid for. Make the plain cut with short-editing's ranges (from 
    video-agent/studio/
    ):
    bash
    SEL=$(node -e 'const e=require("./public/<slug>.edit.json");console.log(e.ranges.map(r=>`between(t,${r.fromMs/1000},${r.toMs/1000})`).join("+"))')
ffmpeg -y -i public/<slug>.mp4 -vf "select='$SEL',setpts=N/FRAME_RATE/TB" -af "aselect='$SEL',asetpts=N/SR/TB" -c:v libx264 -crf 18 -c:a aac -b:a 192k ../.tmp/<slug>-cut.mp4
    The source is 
    video-agent/.tmp/<slug>-cut.mp4
    .
  • A raw take: offer to run 
    /short-editing
    up to its cut list first (steps 1 to 4) for the same reason. In a hurry: send the raw take as it is.
应翻译 take，而不是已完成的视频：成片中的字幕已经以原语言烧录进画面。随后再由 short-editing 根据新声音完成剪辑、字幕和时间轴处理。
  • 已编辑的 take（存在 
    video-agent/edits/<slug>/edit.json
    ）：只发送保留片段，从而无需为静音、卡顿或重拍内容付费。使用 short-editing 的 ranges（来自 
    video-agent/studio/
    ）进行普通裁剪：
    bash
    SEL=$(node -e 'const e=require("./public/<slug>.edit.json");console.log(e.ranges.map(r=>`between(t,${r.fromMs/1000},${r.toMs/1000})`).join("+"))')
ffmpeg -y -i public/<slug>.mp4 -vf "select='$SEL',setpts=N/FRAME_RATE/TB" -af "aselect='$SEL',asetpts=N/SR/TB" -c:v libx264 -crf 18 -c:a aac -b:a 192k ../.tmp/<slug>-cut.mp4
    裁剪后的源文件位于 
    video-agent/.tmp/<slug>-cut.mp4
    。
  • 原始 take：主动提出先运行 
    /short-editing
    ，完成至其 cut list（步骤 1 到 4），原因相同。如果用户赶时间，则按原样发送原始 take。

3. Estimate, then send

3. 先估算，再发送

bash
ffprobe -v error -show_entries format=duration -of csv=p=0 "<source>"
Show one line and wait for a yes: 
Dub <slug> fr → en · 0:52 of video · billed on your ElevenLabs credits for about 1 minute (check your plan's dubbing price on elevenlabs.io/pricing) · no lip-sync
.
Then send it (about 10 s per minute of upload):
bash
mkdir -p video-agent/.tmp
set -a; . video-agent/.env; set +a
curl -sS --fail-with-body -X POST https://api.elevenlabs.io/v1/dubbing \
  -H "xi-api-key: $ELEVENLABS_API_KEY" \
  -F "file=@<source>" -F "target_lang=<lang>" -F "source_lang=<from or auto>" \
  -F "num_speakers=1" -F "watermark=false" -F "name=<slug>-<lang>" \
  > video-agent/.tmp/<slug>-<lang>.job.json
cat video-agent/.tmp/<slug>-<lang>.job.json
It returns 
{"dubbing_id": "...", "expected_duration_sec": ...}
. Keep the id: if anything stops, polling again with it costs nothing. A 401 is a wrong key; a 402 or "quota" means no credits left on their plan; 
watermark
refused: send it again without that field.
bash
ffprobe -v error -show_entries format=duration -of csv=p=0 "<source>"
显示一行并等待用户回复 yes：
Dub <slug> fr → en · 视频时长 0:52 · 将从 ElevenLabs credits 中扣除约 1 分钟的费用（请在 elevenlabs.io/pricing 查看套餐的 dubbing 价格）· 不做 lip-sync
然后发送（上传大约每分钟素材需 10 秒）：
bash
mkdir -p video-agent/.tmp
set -a; . video-agent/.env; set +a
curl -sS --fail-with-body -X POST https://api.elevenlabs.io/v1/dubbing \
  -H "xi-api-key: $ELEVENLABS_API_KEY" \
  -F "file=@<source>" -F "target_lang=<lang>" -F "source_lang=<from or auto>" \
  -F "num_speakers=1" -F "watermark=false" -F "name=<slug>-<lang>" \
  > video-agent/.tmp/<slug>-<lang>.job.json
cat video-agent/.tmp/<slug>-<lang>.job.json
响应为 
{"dubbing_id": "...", "expected_duration_sec": ...}
。保留该 id：如果流程中断，使用它再次轮询不会产生额外费用。401 表示 API key 错误；402 或“quota”表示套餐中没有剩余 credits；如果 
watermark
被拒绝，则不传该字段重新发送。

4. Wait, then download

4. 等待，然后下载

Poll every 15 s until 
status
is 
dubbed
(
failed
: show the error and stop). Usually 1 to 5 minutes for a short; run it in the background:
bash
set -a; . video-agent/.env; set +a
curl -sS --fail-with-body -H "xi-api-key: $ELEVENLABS_API_KEY" https://api.elevenlabs.io/v1/dubbing/<dubbing_id>
Then the dubbed file:
bash
set -a; . video-agent/.env; set +a
curl -sS --fail-with-body -H "xi-api-key: $ELEVENLABS_API_KEY" -o video-agent/.tmp/<slug>-<lang>.dub https://api.elevenlabs.io/v1/dubbing/<dubbing_id>/audio/<lang>
ffprobe -v error -show_entries stream=codec_type -of csv=p=0 video-agent/.tmp/<slug>-<lang>.dub
  • It has a 
    video
    stream: it is already the dubbed video, move it to 
    video-agent/takes/<slug>-<lang>.mp4
    .
  • Audio only: lay it back on the original image:
    bash
    ffmpeg -y -i "<source>" -i video-agent/.tmp/<slug>-<lang>.dub -map 0:v -map 1:a -c:v copy -c:a aac -b:a 192k -shortest video-agent/takes/<slug>-<lang>.mp4
Check it: same length as the source (within a second). Then ask them to listen to the first 10 seconds before editing (voice, accent, a wrong word).
每 15 秒轮询一次，直到 
status
为 
dubbed
（若为 
failed
，则显示错误并停止）。短视频通常需要 1 到 5 分钟；可在后台运行：
bash
set -a; . video-agent/.env; set +a
curl -sS --fail-with-body -H "xi-api-key: $ELEVENLABS_API_KEY" https://api.elevenlabs.io/v1/dubbing/<dubbing_id>
然后下载配音文件：
bash
set -a; . video-agent/.env; set +a
curl -sS --fail-with-body -H "xi-api-key: $ELEVENLABS_API_KEY" -o video-agent/.tmp/<slug>-<lang>.dub https://api.elevenlabs.io/v1/dubbing/<dubbing_id>/audio/<lang>
ffprobe -v error -show_entries stream=codec_type -of csv=p=0 video-agent/.tmp/<slug>-<lang>.dub
  • 如果包含 
    video
    stream，则它已经是配音后的视频，将其移动到 
    video-agent/takes/<slug>-<lang>.mp4
    。
  • 如果只有音频，则将其重新合成到原始画面上：
    bash
    ffmpeg -y -i "<source>" -i video-agent/.tmp/<slug>-<lang>.dub -map 0:v -map 1:a -c:v copy -c:a aac -b:a 192k -shortest video-agent/takes/<slug>-<lang>.mp4
检查文件：时长应与源文件相同（误差不超过 1 秒）。然后让用户在编辑前试听前 10 秒（声音、口音、错误单词）。

5. Edit the translated take

5. 编辑翻译后的 take

/short-editing <slug>-<lang>
, and tell it:
  • The language changed: in 
    video-agent/studio/whisper-config.mjs
    , set 
    WHISPER_LANG
    to 
    <lang>
    for this transcription (English: 
    WHISPER_MODEL = "medium.en"
    works best; any other language: 
    "medium"
    ), and put it back after. The hook card, callouts and 
    post.md
    are written in 
    <lang>
    .
  • Same edit as the original when there is one: same hook card and callouts (translated), placed on the matching words of the new transcript.
  • The ask (CTA) in 
    <lang>
    : ask them once for its translation, or translate the one in voice.md and show it.
Name everything 
<slug>-<lang>
so both versions sit side by side. Then 
/publish <slug>-<lang>
on the account for that language.
Delete 
video-agent/.tmp/<slug>-*
at the end, except the job file if they may want to download the dub again.
运行 
/short-editing <slug>-<lang>
，并向它说明：
  • 语言已更改：在 
    video-agent/studio/whisper-config.mjs
    中，将 
    WHISPER_LANG
    设为 
    <lang>
    以用于此次 transcription（英语：
    WHISPER_MODEL = "medium.en"
    效果最佳；其他语言使用 
    "medium"
    ），完成后恢复原值。hook card、callouts 和 
    post.md
    均使用 
    <lang>
    编写。
  • 若原版已有编辑方案，则沿用同一套方案：使用相同的 hook card 和 callouts（翻译后），并放置在新 transcript 中对应的词语上。
  • <lang>
    版本的 CTA：可向用户询问一次 CTA 的翻译，或直接翻译 
    voice.md
    中的 CTA 并展示。
将所有产物命名为 
<slug>-<lang>
，让两个版本并存。随后在该语言对应的账号上运行 
/publish <slug>-<lang>
。
最后删除 
video-agent/.tmp/<slug>-*
，但如果用户之后可能需要再次下载 dub，则保留 job file。

Honest limits

实际限制

  • No lip-sync: the lips move in the original language. Fine with a lot of motion, a small face or quick cuts; noticeable on a long close-up.
  • The voice clone comes from the take: a noisy take or music under the voice gives a worse clone.
  • Machine translation: names, jargon and jokes can come out wrong. Ask them (or a native speaker) to check the transcript of the dubbed take before posting.
  • Dubbing costs ElevenLabs credits per minute, and the price depends on their plan. Trimming the take first is the cheapest saving.
  • 不做 lip-sync：嘴唇仍按原语言活动。画面运动较多、脸部较小或切换较快时问题不明显；在长时间特写中会比较明显。
  • voice clone 来自 take 本身：take 中噪声较大或人声下方有音乐时，克隆效果会变差。
  • machine translation 可能出错：名称、术语和笑话都可能被译错。发布前应让用户（或母语者）检查配音后 take 的 transcript。
  • dubbing 按视频分钟数消耗 ElevenLabs credits，价格取决于用户的套餐。先裁剪 take 是最省 credits 的方法。