add-dictation
Compare original and translation side by side
🇺🇸
Original
English🇨🇳
Translation
ChineseAdd Dictation
添加听写功能
Add Grok Speech to Text to an existing app: a mic button that dictates into the composer, live captions, or transcripts of recorded audio. Run on , typed Dictate, or clear “transcribe” intent. Cursor has no mic; wire the app, not the IDE.
/add-dictation为现有应用添加 Grok 语音转文字功能:可在编辑器中输入听写内容的麦克风按钮、实时字幕,或录制音频的转录文本。在执行 、输入 Dictate 或明确的“转写”意图时触发。Cursor 没有麦克风;请对接应用,而非 IDE。
/add-dictationDocs
文档
- https://docs.x.ai/developers/model-capabilities/audio/speech-to-text
- Pricing (cite docs only): https://docs.x.ai/developers/pricing
Pick the path
选择实现路径
| Need | Path |
|---|---|
| Tap, speak, tap, text appears. Uploaded files. URLs. | Batch |
| Text appears while speaking: captions, long dictation, push-to-talk | Streaming |
Batch is the default for a composer mic button: one request, no socket, the key never leaves the server. Go streaming only when the UX needs interim text.
| 需求 | 路径 |
|---|---|
| 点击、说话、再点击,文本即可出现。支持上传文件、URL。 | 批量模式 |
| 说话时实时显示文本:字幕、长听写、按键说话 | 流式模式 |
批量模式是编辑器麦克风按钮的默认选项:仅需一次请求,无需 Socket,密钥不会离开服务器。仅当用户体验需要实时中间文本时才使用流式模式。
Auth
认证
- Bearer , server side only. The STT docs document no ephemeral-token flow, and browsers cannot set WebSocket headers, so browser streaming goes through your backend relay. Do not invent a token flow.
XAI_API_KEY - Never put the key in a client bundle. Do not paste keys in chat.
- 采用 Bearer 认证,仅在服务端使用。STT 文档未提及临时令牌流程,且浏览器无法设置 WebSocket 请求头,因此浏览器端的流式传输需通过你的后端中继。请勿自行发明令牌流程。
XAI_API_KEY - 切勿将密钥放在客户端打包文件中。请勿在聊天中粘贴密钥。
Steps
实现步骤
-
Map the app
- Composer or input component, where the text should land (insert at cursor vs replace), server framework, package manager.
- The microphone icon belongs to dictation. If is installed, its waveform primary button stays as is; add the mic as a secondary ghost button beside it.
/add-voice - Existing mic capture? If ran, its PCM capture can feed streaming STT; pass its rate as
/add-voice. 16 kHz is the model’s native rate; other supported rates (8000, 16000, 22050, 24000, 44100, 48000) are resampled server side.sample_rate
-
Batch path (default)
- Client: →
MediaRecorder→Blobto your own route. The endpoint auto-detects containers (WAV, MP3, OGG, Opus, FLAC, AAC, MP4, M4A, MKV, WebM), so send whateverPOSTproduces.MediaRecorder - Server: forward as . Option fields first,
multipart/form-datalast; fields afterfilemay be ignored.fileorfile, max 500 MB.url
- Client:
ts
// server (any runtime with fetch + FormData)
export async function transcribe(blob: Blob, filename: string) {
const form = new FormData();
form.append("format", "true"); // written-form numbers/currency; requires language
form.append("language", "en");
// form.append("keyterm", "Acme"); // repeat per term, ≤100 terms × 50 chars
form.append("file", blob, filename); // last
const res = await fetch("https://api.x.ai/v1/stt", {
method: "POST",
headers: { Authorization: `Bearer ${process.env.XAI_API_KEY}` },
body: form,
});
if (!res.ok) throw new Error(`STT ${res.status}`); // 400 bad input, 413 >500 MB, 429 back off, 502 url fetch failed, 503 retry
return (await res.json()) as {
text: string; language: string; duration: number;
words?: { text: string; start: number; end: number; speaker?: number }[];
channels?: { index: number; text: string; words: unknown[] }[];
};
}ts
// client
const mime = MediaRecorder.isTypeSupported("audio/webm;codecs=opus") ? "audio/webm;codecs=opus" : "audio/mp4";
const rec = new MediaRecorder(stream, { mimeType: mime });
const parts: BlobPart[] = [];
rec.ondataavailable = (e) => parts.push(e.data);
rec.onstop = async () => {
const fd = new FormData();
fd.append("file", new Blob(parts, { type: mime }), "dictation");
const { text } = await (await fetch("/api/dictation", { method: "POST", body: fd })).json();
insertAtCursor(text);
};
rec.start(); // second tap: rec.stop()- Streaming path
- Relay: server holds the key, upgrades the browser socket, forwards binary frames and client control messages up, JSON events down. Build the query string server side.
ts
import { WebSocketServer, WebSocket } from "ws";
new WebSocketServer({ port: 8788 }).on("connection", (client) => {
const q = new URLSearchParams({ sample_rate: "16000", encoding: "pcm", interim_results: "true", language: "en" });
const up = new WebSocket(`wss://api.x.ai/v1/stt?${q}`, { headers: { Authorization: `Bearer ${process.env.XAI_API_KEY}` } });
up.on("message", (d) => client.send(d.toString())); // transcript.* and error events
client.on("message", (d, isBinary) => up.readyState === WebSocket.OPEN && up.send(d, { binary: isBinary })); // audio + finalize/audio.done
const end = () => { client.close(); up.close(); };
up.on("close", end); up.on("error", end); client.on("close", end);
});- Browser capture: PCM16 little-endian, mono, 16 kHz, 100 ms frames = 3,200 bytes, raw binary, no base64. Wait for before sending.
transcript.createdoutput is a container, not raw frames; do not stream it.MediaRecorder
ts
const ws = new WebSocket(relayUrl); ws.binaryType = "arraybuffer";
const ctx = new AudioContext({ sampleRate: 16000 }); // if ctx.sampleRate !== 16000, downsample in the worklet
await ctx.audioWorklet.addModule("/pcm16-worklet.js"); // Float32 → Int16LE, posts one 3,200-byte frame per 100 ms
const node = new AudioWorkletNode(ctx, "pcm16");
ctx.createMediaStreamSource(stream).connect(node);
let ready = false;
node.port.onmessage = (e) => ready && ws.readyState === WebSocket.OPEN && ws.send(e.data);
let committed = "", locked = "", live = "";
ws.addEventListener("message", (e) => {
const ev = JSON.parse(e.data);
if (ev.type === "transcript.created") ready = true;
else if (ev.type === "transcript.partial") {
if (ev.speech_final) { committed += ev.text + " "; locked = ""; live = ""; } // complete stitched utterance
else if (ev.is_final) { locked += ev.text + " "; live = ""; } // chunk final: text will not change
else live = ev.text; // interim: may change
render(committed + locked + live);
} else if (ev.type === "transcript.done") ws.close(); // after audio.done
else if (ev.type === "error") showError(ev.message); // most errors close the socket
});
// stop: ws.send(JSON.stringify({ type: "audio.done" }))
// push-to-talk release: ws.send(JSON.stringify({ type: "Finalize" })) then keep streaming (docs show both `finalize` and `Finalize`; the examples use `Finalize`)- Options (query params for streaming, form fields for batch)
| Want | Set |
|---|---|
| Text while speaking | |
“one hundred dollars” → | streaming: |
| Product names, jargon | |
| Not cut off mid-sentence while dictating numbers | |
| Faster or slower end of utterance | |
| Who said what (meetings) | |
| Agent and customer on separate channels | |
| Keep “um”, “uh” | |
| Low bandwidth or mobile | |
| Raw audio to batch | `audio_format=pcm |
| Quiet or telephony audio | lower |
- Python twin (only if the server is Python)
python
import os, requests
r = requests.post(
"https://api.x.ai/v1/stt",
headers={"Authorization": f"Bearer {os.environ['XAI_API_KEY']}"},
data=[("format", "true"), ("language", "en")],
files={"file": ("dictation.webm", blob, "audio/webm")}, # requests sends data fields before files
)
r.raise_for_status(); text = r.json()["text"]-
梳理应用情况
- 编辑器或输入组件,文本应插入的位置(光标处插入 vs 替换),服务端框架、包管理器。
- 麦克风图标属于听写功能。如果已安装 ,其波形主按钮保持不变;将麦克风按钮作为次要幽灵按钮添加在旁边。
/add-voice - 是否已有麦克风采集功能?如果已运行 ,其 PCM 采集数据可用于流式 STT;将其采样率作为
/add-voice参数传入。16 kHz 是模型的原生采样率;其他支持的采样率(8000、16000、22050、24000、44100、48000)会在服务端重采样。sample_rate
-
批量模式(默认)
- 客户端:→
MediaRecorder→Blob到你自己的路由。接口会自动检测音频容器格式(WAV、MP3、OGG、Opus、FLAC、AAC、MP4、M4A、MKV、WebM),因此直接发送POST产生的任何格式即可。MediaRecorder - 服务端:以 格式转发。选项字段在前,
multipart/form-data字段在最后;file之后的字段可能会被忽略。支持file或file参数,最大 500 MB。url
- 客户端:
ts
// server (any runtime with fetch + FormData)
export async function transcribe(blob: Blob, filename: string) {
const form = new FormData();
form.append("format", "true"); // written-form numbers/currency; requires language
form.append("language", "en");
// form.append("keyterm", "Acme"); // repeat per term, ≤100 terms × 50 chars
form.append("file", blob, filename); // last
const res = await fetch("https://api.x.ai/v1/stt", {
method: "POST",
headers: { Authorization: `Bearer ${process.env.XAI_API_KEY}` },
body: form,
});
if (!res.ok) throw new Error(`STT ${res.status}`); // 400 bad input, 413 >500 MB, 429 back off, 502 url fetch failed, 503 retry
return (await res.json()) as {
text: string; language: string; duration: number;
words?: { text: string; start: number; end: number; speaker?: number }[];
channels?: { index: number; text: string; words: unknown[] }[];
};
}ts
// client
const mime = MediaRecorder.isTypeSupported("audio/webm;codecs=opus") ? "audio/webm;codecs=opus" : "audio/mp4";
const rec = new MediaRecorder(stream, { mimeType: mime });
const parts: BlobPart[] = [];
rec.ondataavailable = (e) => parts.push(e.data);
rec.onstop = async () => {
const fd = new FormData();
fd.append("file", new Blob(parts, { type: mime }), "dictation");
const { text } = await (await fetch("/api/dictation", { method: "POST", body: fd })).json();
insertAtCursor(text);
};
rec.start(); // second tap: rec.stop()- 流式模式
- 中继:服务端保存密钥,升级浏览器 Socket 连接,向上转发二进制帧和客户端控制消息,向下转发 JSON 事件。在服务端构建查询字符串。
ts
import { WebSocketServer, WebSocket } from "ws";
new WebSocketServer({ port: 8788 }).on("connection", (client) => {
const q = new URLSearchParams({ sample_rate: "16000", encoding: "pcm", interim_results: "true", language: "en" });
const up = new WebSocket(`wss://api.x.ai/v1/stt?${q}`, { headers: { Authorization: `Bearer ${process.env.XAI_API_KEY}` } });
up.on("message", (d) => client.send(d.toString())); // transcript.* and error events
client.on("message", (d, isBinary) => up.readyState === WebSocket.OPEN && up.send(d, { binary: isBinary })); // audio + finalize/audio.done
const end = () => { client.close(); up.close(); };
up.on("close", end); up.on("error", end); client.on("close", end);
});- 浏览器采集:PCM16 小端格式、单声道、16 kHz,100 毫秒帧 = 3200 字节,原始二进制格式,不使用 base64。发送前请等待 事件。
transcript.created的输出是容器格式,不是原始帧;请勿直接流式传输其输出。MediaRecorder
ts
const ws = new WebSocket(relayUrl); ws.binaryType = "arraybuffer";
const ctx = new AudioContext({ sampleRate: 16000 }); // if ctx.sampleRate !== 16000, downsample in the worklet
await ctx.audioWorklet.addModule("/pcm16-worklet.js"); // Float32 → Int16LE, posts one 3,200-byte frame per 100 ms
const node = new AudioWorkletNode(ctx, "pcm16");
ctx.createMediaStreamSource(stream).connect(node);
let ready = false;
node.port.onmessage = (e) => ready && ws.readyState === WebSocket.OPEN && ws.send(e.data);
let committed = "", locked = "", live = "";
ws.addEventListener("message", (e) => {
const ev = JSON.parse(e.data);
if (ev.type === "transcript.created") ready = true;
else if (ev.type === "transcript.partial") {
if (ev.speech_final) { committed += ev.text + " "; locked = ""; live = ""; } // complete stitched utterance
else if (ev.is_final) { locked += ev.text + " "; live = ""; } // chunk final: text will not change
else live = ev.text; // interim: may change
render(committed + locked + live);
} else if (ev.type === "transcript.done") ws.close(); // after audio.done
else if (ev.type === "error") showError(ev.message); // most errors close the socket
});
// stop: ws.send(JSON.stringify({ type: "audio.done" }))
// push-to-talk release: ws.send(JSON.stringify({ type: "Finalize" })) then keep streaming (docs show both `finalize` and `Finalize`; the examples use `Finalize`)- 可选参数(流式模式为查询参数,批量模式为表单字段)
| 所需功能 | 设置参数 |
|---|---|
| 说话时实时显示文本 | |
“one hundred dollars” 转换为 | 流式模式: |
| 产品名称、专业术语 | 重复设置 |
| 听写数字时不会在句子中间被截断 | |
| 调整话语结束检测的快慢 | |
| 区分说话人(会议场景) | |
| Agent 和客户在独立声道 | |
| 保留“um”、“uh”等填充词 | |
| 低带宽或移动设备场景 | |
| 批量模式下使用原始音频 | `audio_format=pcm |
| 安静环境或电话音频 | 降低 |
- Python 版本实现(仅当服务端为 Python 时使用)
python
import os, requests
r = requests.post(
"https://api.x.ai/v1/stt",
headers={"Authorization": f"Bearer {os.environ['XAI_API_KEY']}"},
data=[("format", "true"), ("language", "en")],
files={"file": ("dictation.webm", blob, "audio/webm")}, # requests sends data fields before files
)
r.raise_for_status(); text = r.json()["text"]streaming: websockets.connect(url, additional_headers={"Authorization": f"Bearer {key}"}); await ws.send(pcm_bytes)
streaming: websockets.connect(url, additional_headers={"Authorization": f"Bearer {key}"}); await ws.send(pcm_bytes)
6. **Smoke**
- Batch: `curl -X POST https://api.x.ai/v1/stt -H "Authorization: Bearer $XAI_API_KEY" -F language=en -F file=@short.wav` → 200 with `text`. Same call with `-F format=true` and no `language` → 400.
- Streaming: dictate two sentences with a pause between them. Expect interim text, then a final; no duplicated or vanished words at the utterance boundary (if words vanish, the stitched `speech_final` text did not include the chunk finals: append instead of replacing `locked`). `audio.done` → `transcript.done`, socket closes.
- Search the client bundle for `XAI_API_KEY`; it must not be there.
- Debug from logs with `/debug-voice`; swap its hook points to `transcript.*` events.
6. **冒烟测试**
- 批量模式:执行 `curl -X POST https://api.x.ai/v1/stt -H "Authorization: Bearer $XAI_API_KEY" -F language=en -F file=@short.wav` → 返回 200 且包含 `text` 字段。同样的调用加上 `-F format=true` 且不带 `language` 参数 → 返回 400。
- 流式模式:听写两个句子,中间稍作停顿。预期会先出现中间文本,然后出现最终文本;话语边界处没有重复或丢失的单词(如果单词丢失,说明拼接的 `speech_final` 文本没有包含块级最终结果:应追加而不是替换 `locked` 变量)。发送 `audio.done` → 收到 `transcript.done`,Socket 关闭。
- 在客户端打包文件中搜索 `XAI_API_KEY`;绝对不能存在。
- 使用 `/debug-voice` 从日志调试;将其钩子点替换为 `transcript.*` 事件。Out of scope
不在范围内
- Speech that talks back (), speaking text (
/add-voice)/add-read-aloud - Inventing an STT token flow, endpoints, or event names not in the docs
- 会回话的语音()、文本朗读(
/add-voice)/add-read-aloud - 发明文档中没有的 STT 令牌流程、接口或事件名称