add-voice

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Add Voice

添加语音功能

Add Grok Speech to Speech to an existing app. Run on

/add-voice

, typed Voice Mode, or clear “add Grok voice” intent.

为现有应用添加 Grok 语音转语音功能。在用户执行

/add-voice

、输入 Voice Mode 或明确表达“添加 Grok 语音”的意图时使用。

Goal

目标

Working duplex path: user-app mic in, audio out,

wss://api.x.ai/v1/realtime?model=grok-voice-latest

, safe auth. Cursor has no native mic; wire the app (or a sample client), not the IDE.

实现可用的双向通路：用户端应用麦克风输入、音频输出，连接

wss://api.x.ai/v1/realtime?model=grok-voice-latest

，具备安全认证。Cursor 没有原生麦克风支持；需对接应用（或示例客户端），而非 IDE。

Protocol first

优先基于协议实现

Language-agnostic event loop. TypeScript samples default. Short Python twins only where the client API differs (e.g.

ws

websockets

与语言无关的事件循环。默认提供 TypeScript 示例。 仅在客户端 API 存在差异时（例如

ws

与

websockets

）提供精简的 Python 对应示例。

Docs

文档

https://docs.x.ai/developers/model-capabilities/audio/speech-to-speech
https://docs.x.ai/developers/model-capabilities/audio/ephemeral-tokens
Pricing (cite docs only): https://docs.x.ai/developers/pricing (~$0.08/min STS + $0.004/text item; max session 120 min)

https://docs.x.ai/developers/model-capabilities/audio/speech-to-speech
https://docs.x.ai/developers/model-capabilities/audio/ephemeral-tokens
定价（仅引用文档）：https://docs.x.ai/developers/pricing （约 0.08 美元/分钟 STS + 0.004 美元/文本项；单会话最长 120 分钟）

Steps

步骤

Map the app
- Stack: none, OpenAI Realtime, STT→LLM→TTS cascade, TTS/STT only.
- Client: web / Node / iOS / Android / server.
- If a cascade or OpenAI Realtime exists: replace it with the single duplex loop below (URL, model, voice, event diffs); keep standalone
```
/v1/stt
```
  or
```
/v1/tts
```
  only if the product still needs one-shot listen or speak outside the agent.
Auth
- Server: Bearer
```
XAI_API_KEY
```
  .
- Browser/mobile: backend
```
POST https://api.x.ai/v1/realtime/client_secrets
```
  , client uses ephemeral token (Bearer or browser
```
sec-websocket-protocol
```
  :
```
xai-client-secret.<token>
```
  ).
- Never put a long-lived key in client bundles. Do not paste keys in chat.

Connect + session

URL:

wss://api.x.ai/v1/realtime?model=grok-voice-latest

On open:
```
session.update
```
with
```
voice
```
(default
```
eve
```
),
```
instructions
```
,
```
turn_detection: { type: "server_vad" }
```
(or
```
null
```
for push-to-talk), PCM 24 kHz unless the app already standardizes elsewhere.

Set

audio.input.transcription.model: "grok-transcribe"

or no user transcript arrives (

conversation.item.input_audio_transcription.updated

is cumulative, not a delta).

Tools if needed:

web_search

x_search

file_search

mcp

, custom

function

Audio I/O (app-side)
- One
```
AudioContext
```
  per session for capture and playback, created inside the user gesture (autoplay policy). Ask for 24 kHz; if the browser gives another rate, resample before sending.
- Mic → AudioWorklet in ~100 ms chunks →
```
input_audio_buffer.append
```
  (or binary transport). Start WS and mic in parallel; buffer early audio, flush on open.
- Play
```
response.output_audio.delta
```
  immediately; schedule with a ~150 ms lead so chunks butt together. On
```
input_audio_buffer.speech_started
```
  , stop everything queued (barge-in).
- Transcript rows: create the user row on
```
input_audio_buffer.committed
```
  (
```
item_id
```
  ), fill it on
```
…transcription.updated
```
  ; assistant text from
```
response.output_audio_transcript.delta
```
  /
```
.done
```
  , close the turn on
```
response.done
```
  .
- On function tools:
```
function_call_output
```
  , finish playback, then
```
response.create
```
  .
Composer UI convention
- One primary button, right side of the composer. Empty composer → waveform icon (stroked, e.g. Phosphor
```
WaveformIcon weight="bold"
```
  ; never the
```
fill
```
  weight, which renders as a blob at 16 px), starts voice mode. Any text present → classic send arrow; in voice mode that text goes into the live session (
```
conversation.item.create
```
  +
```
response.create
```
  ). Text reply streaming → stop square.
- While voice is live the same button shows an animated waveform (4 bars, ~3 px wide, 2 px gap, ~16 px tall, min scale 0.4 so they stay legible in a 28 px button) and ends the session on click. Phase drives the animation: listening slow, speaking fast, connecting/thinking slower and slightly dimmed (opacity ≥ 0.75). Honor
```
prefers-reduced-motion
```
  . No X button, no pulsing ring.
- Status lives in the composer, not around it: the placeholder reads
```
Connecting…
```
  /
```
Listening…
```
  /
```
Thinking…
```
  /
```
Speaking…
```
  , plus an
```
sr-only
```
```
role="status"
```
  . No separate status row.
- The microphone icon is reserved for dictation (
```
/add-dictation
```
  ). Never use it for voice mode.
TS skeleton (default)

const url = "wss://api.x.ai/v1/realtime?model=grok-voice-latest";
// Node: pass Authorization header. Browser: use xai-client-secret.<token> protocol.
const ws = new WebSocket(url /* , { headers: { Authorization: `Bearer ${token}` } } */);

ws.addEventListener("open", () => {
  ws.send(JSON.stringify({
    type: "session.update",
    session: {
      voice: "eve",
      instructions: "You are a helpful voice agent.",
      turn_detection: { type: "server_vad" },
    },
  }));
});

ws.addEventListener("message", (ev) => {
  const event = JSON.parse(String(ev.data));
  if (event.type === "response.output_audio.delta") {
    // decode base64 PCM and play
  }
});

Python twin (only if the app is Python)

python

import json, os, websockets

url = "wss://api.x.ai/v1/realtime?model=grok-voice-latest"
headers = {"Authorization": f"Bearer {os.environ['XAI_API_KEY']}"}

async with websockets.connect(url, additional_headers=headers) as ws:
    await ws.send(json.dumps({
        "type": "session.update",
        "session": {
            "voice": "eve",
            "instructions": "You are a helpful voice agent.",
            "turn_detection": {"type": "server_vad"},
        },
    }))
    async for raw in ws:
        event = json.loads(raw)
        if event.get("type") == "response.output_audio.delta":
            pass  # decode and play

Instrument (before the first human test)
- Run
```
/debug-voice
```
  : it proposes a plan, then installs a dev-only log sink (
```
POST /api/voice/log
```
  →
```
.voice-logs/<sessionId>.ndjson
```
  , gitignored), a client logger with audio reduced to byte counts, and the session id in the UI, in the app's own language.
- The same skill carries the fix loop and the symptom → log signature → fix table.
Smoke
- Text turn via
```
conversation.item.create
```
  +
```
response.create
```
  ; confirm audio or transcript events.
- Confirm no long-lived key in client (grep the built client bundle for the env name and
```
client_secrets
```
  ).
- Hand the app to the user with headphones. On speakers the mic hears the reply and the model answers itself; that is echo, not a bug in the loop.
- Iterate with
```
/debug-voice
```
  .

梳理应用现状
- 技术栈：无、OpenAI Realtime、STT→LLM→TTS 级联、仅 TTS/STT。
- 客户端类型：web / Node / iOS / Android / 服务端。
- 如果已存在级联方案或 OpenAI Realtime：用下方的单双向循环替换（注意 URL、模型、语音、事件差异）；仅当产品仍需要 Agent 之外的单次监听或朗读功能时，保留独立的
```
/v1/stt
```
  或
```
/v1/tts
```
  。
认证
- 服务端：Bearer 方式使用
```
XAI_API_KEY
```
  。
- 浏览器/移动端：后端调用
```
POST https://api.x.ai/v1/realtime/client_secrets
```
  ，客户端使用临时令牌（Bearer 方式或浏览器
```
sec-websocket-protocol
```
  ：
```
xai-client-secret.<token>
```
  ）。
- 切勿将长期有效密钥放入客户端包中。不要在聊天中粘贴密钥。
连接与会话
- URL：
```
wss://api.x.ai/v1/realtime?model=grok-voice-latest
```
- 连接建立时：发送
```
session.update
```
  事件，配置
```
voice
```
  （默认
```
eve
```
  ）、
```
instructions
```
  、
```
turn_detection: { type: "server_vad" }
```
  （按键说话模式则设为
```
null
```
  ），音频格式默认 PCM 24 kHz，除非应用已有其他统一标准。
- 设置
```
audio.input.transcription.model: "grok-transcribe"
```
  ，否则不会返回用户转写结果（
```
conversation.item.input_audio_transcription.updated
```
  是累积式的，而非增量式）。
- 如需工具调用：支持
```
web_search
```
  、
```
x_search
```
  、
```
file_search
```
  、
```
mcp
```
  、自定义
```
function
```
  。
音频输入输出（应用端）
- 每个会话使用一个
```
AudioContext
```
  进行采集和播放，需在用户手势触发的回调中创建（符合自动播放策略）。请求 24 kHz 采样率；如果浏览器返回其他采样率，发送前先重采样。
- 麦克风 → AudioWorklet 以约 100 ms 的块为单位 →
```
input_audio_buffer.append
```
  （或二进制传输）。并行启动 WebSocket 和麦克风；缓冲早期音频，连接建立后立即发送。
- 立即播放
```
response.output_audio.delta
```
  ；以约 150 ms 的提前量调度播放，确保音频块无缝衔接。收到
```
input_audio_buffer.speech_started
```
  时，停止所有排队的播放（支持打断）。
- 转写行逻辑：在
```
input_audio_buffer.committed
```
  事件（携带
```
item_id
```
  ）时创建用户消息行，在
```
…transcription.updated
```
  事件时填充内容；助手文本来自
```
response.output_audio_transcript.delta
```
  /
```
.done
```
  ，在
```
response.done
```
  事件时结束本轮对话。
- 函数工具调用：返回
```
function_call_output
```
  ，等待播放完成后，再发送
```
response.create
```
  。
输入框 UI 规范
- 一个主按钮，位于输入框右侧。输入框为空时显示波形图标（描边样式，例如 Phosphor 的
```
WaveformIcon weight="bold"
```
  ；切勿使用
```
fill
```
  权重，16 px 下会显示成一团），点击启动语音模式。输入框有文本时显示经典的发送箭头；在语音模式下，该文本会被加入实时会话（通过
```
conversation.item.create
```
  +
```
response.create
```
  ）。文本回复流式输出时显示停止方形按钮。
- 语音激活时，同一按钮显示动画波形（4 条竖条，宽约 3 px，间距 2 px，高约 16 px，最小缩放比例 0.4，确保在 28 px 按钮内清晰可见），点击结束会话。动画由阶段驱动：监听时慢，说话时快，连接/思考时更慢且略微变暗（透明度 ≥ 0.75）。遵循
```
prefers-reduced-motion
```
  设置。不要使用 X 按钮，不要使用脉冲圆环效果。
- 状态显示在输入框内部，而非外部：占位符显示
```
连接中…
```
  /
```
聆听中…
```
  /
```
思考中…
```
  /
```
说话中…
```
  ，同时添加一个
```
sr-only
```
  的
```
role="status"
```
  元素。不要单独设置状态行。
- 麦克风图标预留给听写功能（
```
/add-dictation
```
  ），切勿用于语音模式。
TypeScript 骨架（默认）

const url = "wss://api.x.ai/v1/realtime?model=grok-voice-latest";
// Node: pass Authorization header. Browser: use xai-client-secret.<token> protocol.
const ws = new WebSocket(url /* , { headers: { Authorization: `Bearer ${token}` } } */);

ws.addEventListener("open", () => {
  ws.send(JSON.stringify({
    type: "session.update",
    session: {
      voice: "eve",
      instructions: "You are a helpful voice agent.",
      turn_detection: { type: "server_vad" },
    },
  }));
});

ws.addEventListener("message", (ev) => {
  const event = JSON.parse(String(ev.data));
  if (event.type === "response.output_audio.delta") {
    // decode base64 PCM and play
  }
});

Python 对应示例（仅当应用为 Python 技术栈时使用）

python

import json, os, websockets

url = "wss://api.x.ai/v1/realtime?model=grok-voice-latest"
headers = {"Authorization": f"Bearer {os.environ['XAI_API_KEY']}"}

async with websockets.connect(url, additional_headers=headers) as ws:
    await ws.send(json.dumps({
        "type": "session.update",
        "session": {
            "voice": "eve",
            "instructions": "You are a helpful voice agent.",
            "turn_detection": {"type": "server_vad"},
        },
    }))
    async for raw in ws:
        event = json.loads(raw)
        if event.get("type") == "response.output_audio.delta":
            pass  # decode and play

埋点（首次人工测试前）
- 运行
```
/debug-voice
```
  ：它会提出方案，然后安装仅用于开发的日志接收器（
```
POST /api/voice/log
```
  →
```
.voice-logs/<sessionId>.ndjson
```
  ，已加入 gitignore）、一个将音频简化为字节计数的客户端日志器，以及在 UI 中显示会话 ID，所有实现均使用应用自身的语言。
- 该技能同时包含修复循环和“症状 → 日志特征 → 修复方案”对照表。
冒烟测试
- 通过
```
conversation.item.create
```
  +
```
response.create
```
  发起文本轮次；确认收到音频或转写事件。
- 确认客户端中没有长期有效密钥（在构建后的客户端包中 grep 环境变量名和
```
client_secrets
```
  关键字）。
- 将应用交给用户时建议佩戴耳机。使用外放时麦克风会拾取回复，模型会自己应答自己；这是回声问题，不是循环逻辑的 bug。
- 使用
```
/debug-voice
```
  迭代优化。

Out of scope

不适用范围

Speech-to-text only (
```
/add-dictation
```
), speaking text (
```
/add-read-aloud
```
)
Image generation and text-only inference
Invented endpoints, events, or CLI flags

仅语音转文本（
```
/add-dictation
```
）、文本朗读（
```
/add-read-aloud
```
）
图像生成和纯文本推理
虚构的端点、事件或 CLI 标志