add-dictation

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Add Dictation

添加听写功能

Add Grok Speech to Text to an existing app: a mic button that dictates into the composer, live captions, or transcripts of recorded audio. Run on
/add-dictation
, typed Dictate, or clear “transcribe” intent. Cursor has no mic; wire the app, not the IDE.
为现有应用添加 Grok 语音转文字功能:可在编辑器中输入听写内容的麦克风按钮、实时字幕,或录制音频的转录文本。在执行
/add-dictation
、输入 Dictate 或明确的“转写”意图时触发。Cursor 没有麦克风;请对接应用,而非 IDE。

Docs

文档

Pick the path

选择实现路径

NeedPath
Tap, speak, tap, text appears. Uploaded files. URLs.Batch
POST https://api.x.ai/v1/stt
(default)
Text appears while speaking: captions, long dictation, push-to-talkStreaming
wss://api.x.ai/v1/stt
through a backend relay
Batch is the default for a composer mic button: one request, no socket, the key never leaves the server. Go streaming only when the UX needs interim text.
需求路径
点击、说话、再点击,文本即可出现。支持上传文件、URL。批量模式
POST https://api.x.ai/v1/stt
(默认)
说话时实时显示文本:字幕、长听写、按键说话流式模式
wss://api.x.ai/v1/stt
(通过后端中继)
批量模式是编辑器麦克风按钮的默认选项:仅需一次请求,无需 Socket,密钥不会离开服务器。仅当用户体验需要实时中间文本时才使用流式模式。

Auth

认证

  • Bearer
    XAI_API_KEY
    , server side only. The STT docs document no ephemeral-token flow, and browsers cannot set WebSocket headers, so browser streaming goes through your backend relay. Do not invent a token flow.
  • Never put the key in a client bundle. Do not paste keys in chat.
  • 采用 Bearer
    XAI_API_KEY
    认证,仅在服务端使用。STT 文档未提及临时令牌流程,且浏览器无法设置 WebSocket 请求头,因此浏览器端的流式传输需通过你的后端中继。请勿自行发明令牌流程。
  • 切勿将密钥放在客户端打包文件中。请勿在聊天中粘贴密钥。

Steps

实现步骤

  1. Map the app
    • Composer or input component, where the text should land (insert at cursor vs replace), server framework, package manager.
    • The microphone icon belongs to dictation. If
      /add-voice
      is installed, its waveform primary button stays as is; add the mic as a secondary ghost button beside it.
    • Existing mic capture? If
      /add-voice
      ran, its PCM capture can feed streaming STT; pass its rate as
      sample_rate
      . 16 kHz is the model’s native rate; other supported rates (8000, 16000, 22050, 24000, 44100, 48000) are resampled server side.
  2. Batch path (default)
    • Client:
      MediaRecorder
      →
      Blob
      →
      POST
      to your own route. The endpoint auto-detects containers (WAV, MP3, OGG, Opus, FLAC, AAC, MP4, M4A, MKV, WebM), so send whatever
      MediaRecorder
      produces.
    • Server: forward as
      multipart/form-data
      . Option fields first,
      file
      last
      ; fields after
      file
      may be ignored.
      file
      or
      url
      , max 500 MB.
ts
// server (any runtime with fetch + FormData)
export async function transcribe(blob: Blob, filename: string) {
  const form = new FormData();
  form.append("format", "true");     // written-form numbers/currency; requires language
  form.append("language", "en");
  // form.append("keyterm", "Acme"); // repeat per term, ≤100 terms × 50 chars
  form.append("file", blob, filename); // last
  const res = await fetch("https://api.x.ai/v1/stt", {
    method: "POST",
    headers: { Authorization: `Bearer ${process.env.XAI_API_KEY}` },
    body: form,
  });
  if (!res.ok) throw new Error(`STT ${res.status}`); // 400 bad input, 413 >500 MB, 429 back off, 502 url fetch failed, 503 retry
  return (await res.json()) as {
    text: string; language: string; duration: number;
    words?: { text: string; start: number; end: number; speaker?: number }[];
    channels?: { index: number; text: string; words: unknown[] }[];
  };
}
ts
// client
const mime = MediaRecorder.isTypeSupported("audio/webm;codecs=opus") ? "audio/webm;codecs=opus" : "audio/mp4";
const rec = new MediaRecorder(stream, { mimeType: mime });
const parts: BlobPart[] = [];
rec.ondataavailable = (e) => parts.push(e.data);
rec.onstop = async () => {
  const fd = new FormData();
  fd.append("file", new Blob(parts, { type: mime }), "dictation");
  const { text } = await (await fetch("/api/dictation", { method: "POST", body: fd })).json();
  insertAtCursor(text);
};
rec.start(); // second tap: rec.stop()
  1. Streaming path
    • Relay: server holds the key, upgrades the browser socket, forwards binary frames and client control messages up, JSON events down. Build the query string server side.
ts
import { WebSocketServer, WebSocket } from "ws";

new WebSocketServer({ port: 8788 }).on("connection", (client) => {
  const q = new URLSearchParams({ sample_rate: "16000", encoding: "pcm", interim_results: "true", language: "en" });
  const up = new WebSocket(`wss://api.x.ai/v1/stt?${q}`, { headers: { Authorization: `Bearer ${process.env.XAI_API_KEY}` } });
  up.on("message", (d) => client.send(d.toString()));                       // transcript.* and error events
  client.on("message", (d, isBinary) => up.readyState === WebSocket.OPEN && up.send(d, { binary: isBinary })); // audio + finalize/audio.done
  const end = () => { client.close(); up.close(); };
  up.on("close", end); up.on("error", end); client.on("close", end);
});
  • Browser capture: PCM16 little-endian, mono, 16 kHz, 100 ms frames = 3,200 bytes, raw binary, no base64. Wait for
    transcript.created
    before sending.
    MediaRecorder
    output is a container, not raw frames; do not stream it.
ts
const ws = new WebSocket(relayUrl); ws.binaryType = "arraybuffer";
const ctx = new AudioContext({ sampleRate: 16000 }); // if ctx.sampleRate !== 16000, downsample in the worklet
await ctx.audioWorklet.addModule("/pcm16-worklet.js"); // Float32 → Int16LE, posts one 3,200-byte frame per 100 ms
const node = new AudioWorkletNode(ctx, "pcm16");
ctx.createMediaStreamSource(stream).connect(node);
let ready = false;
node.port.onmessage = (e) => ready && ws.readyState === WebSocket.OPEN && ws.send(e.data);

let committed = "", locked = "", live = "";
ws.addEventListener("message", (e) => {
  const ev = JSON.parse(e.data);
  if (ev.type === "transcript.created") ready = true;
  else if (ev.type === "transcript.partial") {
    if (ev.speech_final) { committed += ev.text + " "; locked = ""; live = ""; } // complete stitched utterance
    else if (ev.is_final) { locked += ev.text + " "; live = ""; }                // chunk final: text will not change
    else live = ev.text;                                                          // interim: may change
    render(committed + locked + live);
  } else if (ev.type === "transcript.done") ws.close();                          // after audio.done
  else if (ev.type === "error") showError(ev.message);                           // most errors close the socket
});
// stop: ws.send(JSON.stringify({ type: "audio.done" }))
// push-to-talk release: ws.send(JSON.stringify({ type: "Finalize" })) then keep streaming (docs show both `finalize` and `Finalize`; the examples use `Finalize`)
  1. Options (query params for streaming, form fields for batch)
WantSet
Text while speaking
interim_results=true
“one hundred dollars” →
$100
streaming:
language=en
; batch:
format=true
+
language=en
Product names, jargon
keyterm=
repeated
Not cut off mid-sentence while dictating numbers
smart_turn=0.7&smart_turn_timeout=3000
Faster or slower end of utterance
endpointing=
ms, default 400
Who said what (meetings)
diarize=true
→
words[].speaker
Agent and customer on separate channels
multichannel=true&channels=2
(PCM only, not Opus)
Keep “um”, “uh”
filler_words=true
(removed by default)
Low bandwidth or mobile
encoding=opus
, exactly one raw Opus packet per frame, omit
sample_rate
Raw audio to batch`audio_format=pcm
Quiet or telephony audiolower
vad_threshold
(streaming default 0.08, batch 0.5)
  1. Python twin (only if the server is Python)
python
import os, requests
r = requests.post(
    "https://api.x.ai/v1/stt",
    headers={"Authorization": f"Bearer {os.environ['XAI_API_KEY']}"},
    data=[("format", "true"), ("language", "en")],
    files={"file": ("dictation.webm", blob, "audio/webm")},  # requests sends data fields before files
)
r.raise_for_status(); text = r.json()["text"]
  1. 梳理应用情况
    • 编辑器或输入组件,文本应插入的位置(光标处插入 vs 替换),服务端框架、包管理器。
    • 麦克风图标属于听写功能。如果已安装
      /add-voice
      ,其波形主按钮保持不变;将麦克风按钮作为次要幽灵按钮添加在旁边。
    • 是否已有麦克风采集功能?如果已运行
      /add-voice
      ,其 PCM 采集数据可用于流式 STT;将其采样率作为
      sample_rate
      参数传入。16 kHz 是模型的原生采样率;其他支持的采样率(8000、16000、22050、24000、44100、48000)会在服务端重采样。
  2. 批量模式(默认)
    • 客户端:
      MediaRecorder
      →
      Blob
      →
      POST
      到你自己的路由。接口会自动检测音频容器格式(WAV、MP3、OGG、Opus、FLAC、AAC、MP4、M4A、MKV、WebM),因此直接发送
      MediaRecorder
      产生的任何格式即可。
    • 服务端:以
      multipart/form-data
      格式转发。选项字段在前,
      file
      字段在最后
      ;
      file
      之后的字段可能会被忽略。支持
      file
      或
      url
      参数,最大 500 MB。
ts
// server (any runtime with fetch + FormData)
export async function transcribe(blob: Blob, filename: string) {
  const form = new FormData();
  form.append("format", "true");     // written-form numbers/currency; requires language
  form.append("language", "en");
  // form.append("keyterm", "Acme"); // repeat per term, ≤100 terms × 50 chars
  form.append("file", blob, filename); // last
  const res = await fetch("https://api.x.ai/v1/stt", {
    method: "POST",
    headers: { Authorization: `Bearer ${process.env.XAI_API_KEY}` },
    body: form,
  });
  if (!res.ok) throw new Error(`STT ${res.status}`); // 400 bad input, 413 >500 MB, 429 back off, 502 url fetch failed, 503 retry
  return (await res.json()) as {
    text: string; language: string; duration: number;
    words?: { text: string; start: number; end: number; speaker?: number }[];
    channels?: { index: number; text: string; words: unknown[] }[];
  };
}
ts
// client
const mime = MediaRecorder.isTypeSupported("audio/webm;codecs=opus") ? "audio/webm;codecs=opus" : "audio/mp4";
const rec = new MediaRecorder(stream, { mimeType: mime });
const parts: BlobPart[] = [];
rec.ondataavailable = (e) => parts.push(e.data);
rec.onstop = async () => {
  const fd = new FormData();
  fd.append("file", new Blob(parts, { type: mime }), "dictation");
  const { text } = await (await fetch("/api/dictation", { method: "POST", body: fd })).json();
  insertAtCursor(text);
};
rec.start(); // second tap: rec.stop()
  1. 流式模式
    • 中继:服务端保存密钥,升级浏览器 Socket 连接,向上转发二进制帧和客户端控制消息,向下转发 JSON 事件。在服务端构建查询字符串。
ts
import { WebSocketServer, WebSocket } from "ws";

new WebSocketServer({ port: 8788 }).on("connection", (client) => {
  const q = new URLSearchParams({ sample_rate: "16000", encoding: "pcm", interim_results: "true", language: "en" });
  const up = new WebSocket(`wss://api.x.ai/v1/stt?${q}`, { headers: { Authorization: `Bearer ${process.env.XAI_API_KEY}` } });
  up.on("message", (d) => client.send(d.toString()));                       // transcript.* and error events
  client.on("message", (d, isBinary) => up.readyState === WebSocket.OPEN && up.send(d, { binary: isBinary })); // audio + finalize/audio.done
  const end = () => { client.close(); up.close(); };
  up.on("close", end); up.on("error", end); client.on("close", end);
});
  • 浏览器采集:PCM16 小端格式、单声道、16 kHz,100 毫秒帧 = 3200 字节,原始二进制格式,不使用 base64。发送前请等待
    transcript.created
    事件。
    MediaRecorder
    的输出是容器格式,不是原始帧;请勿直接流式传输其输出。
ts
const ws = new WebSocket(relayUrl); ws.binaryType = "arraybuffer";
const ctx = new AudioContext({ sampleRate: 16000 }); // if ctx.sampleRate !== 16000, downsample in the worklet
await ctx.audioWorklet.addModule("/pcm16-worklet.js"); // Float32 → Int16LE, posts one 3,200-byte frame per 100 ms
const node = new AudioWorkletNode(ctx, "pcm16");
ctx.createMediaStreamSource(stream).connect(node);
let ready = false;
node.port.onmessage = (e) => ready && ws.readyState === WebSocket.OPEN && ws.send(e.data);

let committed = "", locked = "", live = "";
ws.addEventListener("message", (e) => {
  const ev = JSON.parse(e.data);
  if (ev.type === "transcript.created") ready = true;
  else if (ev.type === "transcript.partial") {
    if (ev.speech_final) { committed += ev.text + " "; locked = ""; live = ""; } // complete stitched utterance
    else if (ev.is_final) { locked += ev.text + " "; live = ""; }                // chunk final: text will not change
    else live = ev.text;                                                          // interim: may change
    render(committed + locked + live);
  } else if (ev.type === "transcript.done") ws.close();                          // after audio.done
  else if (ev.type === "error") showError(ev.message);                           // most errors close the socket
});
// stop: ws.send(JSON.stringify({ type: "audio.done" }))
// push-to-talk release: ws.send(JSON.stringify({ type: "Finalize" })) then keep streaming (docs show both `finalize` and `Finalize`; the examples use `Finalize`)
  1. 可选参数(流式模式为查询参数,批量模式为表单字段)
所需功能设置参数
说话时实时显示文本
interim_results=true
“one hundred dollars” 转换为
$100
(书面格式)
流式模式:
language=en
;批量模式:
format=true
+
language=en
产品名称、专业术语重复设置
keyterm=
参数
听写数字时不会在句子中间被截断
smart_turn=0.7&smart_turn_timeout=3000
调整话语结束检测的快慢
endpointing=
毫秒,默认 400
区分说话人(会议场景)
diarize=true
→
words[].speaker
Agent 和客户在独立声道
multichannel=true&channels=2
(仅支持 PCM,不支持 Opus)
保留“um”、“uh”等填充词
filler_words=true
(默认会移除)
低带宽或移动设备场景
encoding=opus
,每帧恰好一个原始 Opus 数据包,省略
sample_rate
参数
批量模式下使用原始音频`audio_format=pcm
安静环境或电话音频降低
vad_threshold
(流式模式默认 0.08,批量模式默认 0.5)
  1. Python 版本实现(仅当服务端为 Python 时使用)
python
import os, requests
r = requests.post(
    "https://api.x.ai/v1/stt",
    headers={"Authorization": f"Bearer {os.environ['XAI_API_KEY']}"},
    data=[("format", "true"), ("language", "en")],
    files={"file": ("dictation.webm", blob, "audio/webm")},  # requests sends data fields before files
)
r.raise_for_status(); text = r.json()["text"]

streaming: websockets.connect(url, additional_headers={"Authorization": f"Bearer {key}"}); await ws.send(pcm_bytes)

streaming: websockets.connect(url, additional_headers={"Authorization": f"Bearer {key}"}); await ws.send(pcm_bytes)


6. **Smoke**
   - Batch: `curl -X POST https://api.x.ai/v1/stt -H "Authorization: Bearer $XAI_API_KEY" -F language=en -F file=@short.wav` → 200 with `text`. Same call with `-F format=true` and no `language` → 400.
   - Streaming: dictate two sentences with a pause between them. Expect interim text, then a final; no duplicated or vanished words at the utterance boundary (if words vanish, the stitched `speech_final` text did not include the chunk finals: append instead of replacing `locked`). `audio.done` → `transcript.done`, socket closes.
   - Search the client bundle for `XAI_API_KEY`; it must not be there.
   - Debug from logs with `/debug-voice`; swap its hook points to `transcript.*` events.

6. **冒烟测试**
   - 批量模式:执行 `curl -X POST https://api.x.ai/v1/stt -H "Authorization: Bearer $XAI_API_KEY" -F language=en -F file=@short.wav` → 返回 200 且包含 `text` 字段。同样的调用加上 `-F format=true` 且不带 `language` 参数 → 返回 400。
   - 流式模式:听写两个句子,中间稍作停顿。预期会先出现中间文本,然后出现最终文本;话语边界处没有重复或丢失的单词(如果单词丢失,说明拼接的 `speech_final` 文本没有包含块级最终结果:应追加而不是替换 `locked` 变量)。发送 `audio.done` → 收到 `transcript.done`,Socket 关闭。
   - 在客户端打包文件中搜索 `XAI_API_KEY`;绝对不能存在。
   - 使用 `/debug-voice` 从日志调试;将其钩子点替换为 `transcript.*` 事件。

Out of scope

不在范围内

  • Speech that talks back (
    /add-voice
    ), speaking text (
    /add-read-aloud
    )
  • Inventing an STT token flow, endpoints, or event names not in the docs
  • 会回话的语音(
    /add-voice
    )、文本朗读(
    /add-read-aloud
    )
  • 发明文档中没有的 STT 令牌流程、接口或事件名称