Back to Details

debug-voice

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Debug Voice

语音调试

Make a voice session readable after the fact, then fix from evidence. The agent cannot hear the app; the log is its ears, the user is its judge. No audio, no tokens, never in prod.
Works for any stack. The pipeline is a small contract (below); implement it in whatever the app already uses.
让语音会话可事后回溯，再基于证据修复问题。Agent无法感知应用的音频输出；日志就是它的耳朵，用户则是裁判。不记录音频、不记录token，绝不在生产环境启用。
适用于任意技术栈。该日志管道是一套轻量契约（见下文），使用应用已有的技术栈实现即可。

Workflow

工作流程

  1. Map the app (read only).
  2. Plan: write the change list, show it, stop. Nothing is edited until the user aligns.
  3. Install the agreed pieces in the app's language, framework, and conventions.
  4. Verify the sink, hand the app to the user.
  5. Fix loop from the log.
  1. 梳理应用结构（只读操作）。
  2. 制定方案：列出修改清单，展示给用户后暂停。在用户确认前，不修改任何文件。
  3. 安装部署：按照应用的语言、框架和规范，实现双方确认的功能模块。
  4. 验证日志接收端（sink），将应用交还给用户测试。
  5. 基于日志执行修复循环。

1. Map

1. 结构梳理

Find, and note the paths:
  • Voice client: where realtime events are received and sent, mic capture, audio playback, token fetch.
  • Server: framework, how routes are declared, where shared server code lives, how env is read, what "production" means here.
  • Conventions: language(s), module system, formatter, where scripts or tasks live (
    package.json
    , 
    Makefile
    , 
    pyproject
    , 
    justfile
    ), 
    .gitignore
    .
  • Client kind: browser, mobile, desktop, CLI. A non-browser client still POSTs the same JSON; a single-process app can skip HTTP and append to the file directly.
  • Where audio deltas are handled. They must be counted, never logged.
找到并记录以下路径/位置：
  • 语音客户端：实时事件的收发位置、mic采集、音频播放、token获取的相关代码。
  • 服务端：使用的框架、路由声明方式、服务端公共代码位置、环境变量读取方式、此处“生产环境”的判定逻辑。
  • 项目规范：使用的语言、模块系统、代码格式化工具、脚本/任务配置文件位置（
    package.json
    、
    Makefile
    、
    pyproject
    、
    justfile
    ）、
    .gitignore
    文件位置。
  • 客户端类型：浏览器、移动端、桌面端、CLI。非浏览器客户端仍可发送相同的JSON POST请求；单进程应用可跳过HTTP，直接将日志追加到文件中。
  • 音频delta的处理位置：这些数据仅做计数，绝对不能写入日志。

2. Plan, then stop

2. 制定方案并等待确认

Fill this in with real paths and the app's language, post it, and wait for a yes or a trimmed list. Do not edit files before that.
markdown
undefined
用实际路径和应用的语言填充以下模板，发送给用户后等待确认（用户可能同意，也可能删减清单）。在此之前不要修改任何文件。
markdown
undefined

Debug voice: plan

Debug voice: plan

Add
  • <path>: client logger (batch, redact, flush) in <language>
  • <path>: dev-only sink 
    POST /api/voice/log
    → 
    .voice-logs/<sessionId>.ndjson
  • <path> (optional): summary command 
    <cmd>
    ; otherwise read the NDJSON with jq
Modify
  • <voice client file>: hook points start, token, mic, env, ws.*, client/server events, audio.in (2 s windows), audio.out.first, audio.out, play.stop, stop
  • <token route>: append 
    server.token { ok, status, ms, upstream }
    (never the token)
  • <UI file>: session id in the voice status line and in voice error messages
  • .gitignore: 
    /.voice-logs
  • <scripts file>: a 
    voice:logs
    task (only if the summary command is wanted)
Logged: event names and non-audio fields, timings, byte counts, mic RMS. Never: tokens, API keys, raw audio, strings over 400 chars. Off in production unless 
VOICE_LOG=1
.
Reply "go", or strike lines you do not want.
undefined
Add
  • <path>: client logger (batch, redact, flush) in <language>
  • <path>: dev-only sink 
    POST /api/voice/log
    → 
    .voice-logs/<sessionId>.ndjson
  • <path> (optional): summary command 
    <cmd>
    ; otherwise read the NDJSON with jq
Modify
  • <voice client file>: hook points start, token, mic, env, ws.*, client/server events, audio.in (2 s windows), audio.out.first, audio.out, play.stop, stop
  • <token route>: append 
    server.token { ok, status, ms, upstream }
    (never the token)
  • <UI file>: session id in the voice status line and in voice error messages
  • .gitignore: 
    /.voice-logs
  • <scripts file>: a 
    voice:logs
    task (only if the summary command is wanted)
Logged: event names and non-audio fields, timings, byte counts, mic RMS. Never: tokens, API keys, raw audio, strings over 400 chars. Off in production unless 
VOICE_LOG=1
.
Reply "go", or strike lines you do not want.
undefined

3. Install: the contract

3. 安装部署：契约规范

Match the app. Same language as the surrounding code, same route style, same formatter. Write the pieces from the contract below; do not introduce a second language or toolchain for logging.
Session id: 8 lowercase hex chars from a UUID. The sink accepts 
^[a-z0-9]{4,64}$
; it becomes a file name.
Entry (one JSON object per line):
FieldClientServer
t
ms since the logger startedabsent; the reader aligns by 
ts
ts
epoch msepoch ms
kind
start
, 
server
, 
client
, 
error
, 
audio.in
, …
server.token
, …
src
absent
"server"
restthe hook's fields, redactedthe hook's fields
Redaction, applied client side before buffering: on audio event types (
response.output_audio.delta
, 
response.audio.delta
, 
input_audio_buffer.append
) replace 
delta
/ 
audio
with 
bytes
= decoded base64 length; strings over 400 chars cut to 400 + 
…[N chars]
; objects deeper than 4 → 
"[depth]"
; arrays over 50 items truncated.
Client logger: buffer entries; flush as 
POST <sink> {"sessionId","entries":[…]}
every 1 s or at 200 entries; on stop flush with keepalive (or the platform's "survive navigation" equivalent); swallow every transport error, logging must never throw into the voice path. Also mirror entries to the console in dev.
Sink: 
POST /api/voice/log
, JSON body. 
404
unless dev or 
VOICE_LOG=1
. 
400
if 
sessionId
fails the regex or 
entries
is not an array. Append at most 500 entries per request, drop any line over 16,000 chars, to 
.voice-logs/<sessionId>.ndjson
, creating the directory. Reply 
204
.
Pseudocode for any server:
handle POST /api/voice/log:
  if production and VOICE_LOG != "1": return 404
  body = parse json or return 400
  if not regex(body.sessionId) or not list(body.entries): return 400
  mkdir .voice-logs; append join(json(e) for e in body.entries[:500] if len < 16000) to .voice-logs/{sessionId}.ndjson
  return 204
Pseudocode for the client logger:
logger(sessionId, sink):
  buffer = []; started = now()
  log(kind, data): buffer.push({ ...redact(data), t: now() - started, ts: epoch_ms(), kind }); schedule flush (1 s timer, or immediately at 200 entries)
  server(event, extra): log("server", { ...redact_event(event), ...extra })   # never per audio delta
  client(event):        log("client", redact_event(event))                    # never per audio chunk
  error(where, err, extra): log("error", { where, name, message, ...extra })
  flush(final=false): POST sink {"sessionId","entries": buffer}; buffer = []; ignore all errors; keepalive when final
  close(): flush(final=true)
实现需与应用现有风格保持一致：使用与周边代码相同的语言、相同的路由风格、相同的格式化工具。按照下方的契约实现各模块，不要为了日志功能引入额外的语言或工具链。
会话ID（sessionId）：从UUID中提取的8位小写十六进制字符。sink仅接受匹配
^[a-z0-9]{4,64}$
格式的ID，该ID会作为日志文件名。
日志条目（每行一个JSON对象）：
字段客户端服务端
t
日志记录器启动后经过的毫秒数无；日志读取器通过
ts
对齐时间
ts
Unix 时间戳（毫秒）Unix 时间戳（毫秒）
kind
start
, 
server
, 
client
, 
error
, 
audio.in
, …
server.token
, …
src
无
"server"
其余字段钩子的字段（已脱敏）钩子的字段
脱敏规则（在客户端缓冲前执行）：对于音频类事件（
response.output_audio.delta
、
response.audio.delta
、
input_audio_buffer.append
），将
delta
/
audio
字段替换为
bytes
字段，值为base64解码后的字节数；长度超过400的字符串截断为400字符并追加
…[N chars]
（N为原长度）；嵌套深度超过4层的对象替换为
"[depth]"
；长度超过50的数组做截断处理。
客户端日志记录器（client logger）：对日志条目进行缓冲；每1秒或缓冲达到200条时，通过
POST <sink> {"sessionId","entries":[…]}
刷新到服务端；会话停止时使用keepalive机制刷新（或平台对应的“页面跳转后仍可执行”的等效机制）；吞掉所有传输错误，日志功能绝对不能抛出异常影响语音链路。开发环境下同时将日志输出到控制台。
日志接收端（sink）：接口为
POST /api/voice/log
，请求体为JSON。非开发环境且未设置
VOICE_LOG=1
时返回
404
。如果
sessionId
不匹配正则或
entries
不是数组，返回
400
。每个请求最多追加500条日志，丢弃长度超过16000字符的行，日志写入
.voice-logs/<sessionId>.ndjson
文件，目录不存在则自动创建。请求成功返回
204
。
任意服务端的伪代码实现：
handle POST /api/voice/log:
  if production and VOICE_LOG != "1": return 404
  body = parse json or return 400
  if not regex(body.sessionId) or not list(body.entries): return 400
  mkdir .voice-logs; append join(json(e) for e in body.entries[:500] if len < 16000) to .voice-logs/{sessionId}.ndjson
  return 204
客户端日志记录器的伪代码实现：
logger(sessionId, sink):
  buffer = []; started = now()
  log(kind, data): buffer.push({ ...redact(data), t: now() - started, ts: epoch_ms(), kind }); schedule flush (1 s timer, or immediately at 200 entries)
  server(event, extra): log("server", { ...redact_event(event), ...extra })   # never per audio delta
  client(event):        log("client", redact_event(event))                    # never per audio chunk
  error(where, err, extra): log("error", { where, name, message, ...extra })
  flush(final=false): POST sink {"sessionId","entries": buffer}; buffer = []; ignore all errors; keepalive when final
  close(): flush(final=true)

Hook points

钩子点

kind
and fields; the shape is the same in every language.
When
kind
and fields
Session start
start { url, target_rate }
Token fetched / failed
token.ok { ms }
/ 
error { where: "token", name, message, ms }
Mic granted / denied
mic.ok { ms, label, settings }
/ 
error { where: "mic", … }
Audio graph ready
env { ua, mic_rate, mic_state, play_rate, play_state, capture_frames, target_rate }
Socket
ws.connecting
, 
ws.open { ms }
, 
ws.error
, 
ws.close { code, reason, wasClean, by }
Every event sent, except audio chunks
client { …redacted event }
Every event received, except audio deltas
server { …redacted event, phase }
Phase change (dedupe)
phase { phase }
Mic chunks, aggregated per 2 s
audio.in { chunks, bytes, rms_max, rms_avg, pending, mic_state, phase }
Pre-open buffer sent on open
audio.flush { chunks }
First audio delta of a response
audio.out.first { response_id, bytes, since_response_created_ms, since_speech_stopped_ms, play_state }
response.done
audio.out { response_id, status, deltas, bytes, audio_ms, wall_ms, max_gap_ms, queued_ms, underruns, drain_ms_max }
Barge-in stop
play.stop { reason, dropped_ms }
User stop
stop { by: "client", phase }
, then flush
Token route (server)
server.token { ok, status, ms, upstream }
In the message handler: if the event is an audio delta, count bytes and gaps and play it; otherwise log it as 
server
with the current phase, then run the existing handling. Keep 
speechStoppedT
from 
input_audio_buffer.speech_stopped
and 
createdT
from 
response.created
; report both distances on the first delta. In the player, count an underrun when the next scheduled time is already in the past mid-response, track the largest drain, reset on 
response.created
.
UI: show the id in the voice status line (
Listening · session ab12cd34
) and append 
(voice session <id>)
to voice errors, so the user can name the run.
包含
kind
和对应字段；所有语言的字段结构保持一致。
触发时机
kind
及字段
会话启动
start { url, target_rate }
Token获取成功/失败
token.ok { ms }
/ 
error { where: "token", name, message, ms }
Mic权限授予/拒绝
mic.ok { ms, label, settings }
/ 
error { where: "mic", … }
音频图就绪
env { ua, mic_rate, mic_state, play_rate, play_state, capture_frames, target_rate }
套接字连接
ws.connecting
, 
ws.open { ms }
, 
ws.error
, 
ws.close { code, reason, wasClean, by }
每发送一个事件（音频块除外）
client { …redacted event }
每接收一个事件（音频delta除外）
server { …redacted event, phase }
阶段变化（去重后）
phase { phase }
Mic音频块（每2秒聚合一次）
audio.in { chunks, bytes, rms_max, rms_avg, pending, mic_state, phase }
连接建立时发送预开缓冲区
audio.flush { chunks }
响应的首个音频delta
audio.out.first { response_id, bytes, since_response_created_ms, since_speech_stopped_ms, play_state }
response.done
事件触发时
audio.out { response_id, status, deltas, bytes, audio_ms, wall_ms, max_gap_ms, queued_ms, underruns, drain_ms_max }
打断（barge-in）停止播放
play.stop { reason, dropped_ms }
用户主动停止
stop { by: "client", phase }
，随后刷新日志
Token路由（服务端）
server.token { ok, status, ms, upstream }
在消息处理逻辑中：如果是音频delta事件，仅统计字节数和间隔并播放；否则将其作为
server
类型事件连同当前阶段一起记录，再执行原有处理逻辑。保留
input_audio_buffer.speech_stopped
事件的
speechStoppedT
时间戳和
response.created
事件的
createdT
时间戳；在首个音频delta事件中上报这两个时间点的差值。在播放器中，当响应播放过程中下一个调度播放时间已过期时，计为一次欠载（underrun），跟踪最大排空时间（drain），并在
response.created
事件时重置。
UI层面：在语音状态栏中显示会话ID（例如
Listening · session ab12cd34
），并在语音错误消息后追加
(voice session <id>)
，方便用户标识本次运行。

4. Verify

4. 验证

bash
curl -s -o /dev/null -w "%{http_code}\n" -X POST localhost:<port>/api/voice/log \
  -H 'Content-Type: application/json' \
  -d '{"sessionId":"smoke001","entries":[{"t":0,"ts":0,"kind":"start"}]}'   # 204
curl -s -o /dev/null -w "%{http_code}\n" -X POST localhost:<port>/api/voice/log \
  -H 'Content-Type: application/json' -d '{"sessionId":"../x","entries":[]}' # 400
cat .voice-logs/smoke001.ndjson && rm .voice-logs/smoke001.ndjson
Then hand off: the user tests in the real app and reports what they said, what they heard, when it went wrong, and the session id.
bash
curl -s -o /dev/null -w "%{http_code}\n" -X POST localhost:<port>/api/voice/log \
  -H 'Content-Type: application/json' \
  -d '{"sessionId":"smoke001","entries":[{"t":0,"ts":0,"kind":"start"}]}'   # 204
curl -s -o /dev/null -w "%{http_code}\n" -X POST localhost:<port>/api/voice/log \
  -H 'Content-Type: application/json' -d '{"sessionId":"../x","entries":[]}' # 400
cat .voice-logs/smoke001.ndjson && rm .voice-logs/smoke001.ndjson
随后交接给用户：用户在真实应用中测试，并反馈他们说的内容、听到的内容、问题出现的时间以及会话ID。

Reading a log

日志读取

Any of these; none needs the app's toolchain:
bash
f=.voice-logs/<id>.ndjson
jq -r 'select(.kind|IN("start","token.ok","mic.ok","env","ws.open","ws.close","server.token","stop","error")) | "\(.t // .ts)ms \(.kind) \(.type // "") \(.message // "")"' $f   # milestones and errors
jq -r 'select(.kind=="server") | .type' $f | sort | uniq -c | sort -rn                      # server event counts
jq -c 'select(.kind|IN("audio.out.first","audio.out"))' $f                                  # per-turn latency, gaps, underruns
jq -c 'select(.kind=="audio.in")' $f                                                        # mic windows, rms
Or read the file; it is one object per line in time order. Server entries have only 
ts
; align them to the client clock with the 
start
entry's 
ts
. If the user wants a summary command, write one in the app's language that prints: milestones, server event counts, one line per 
audio.out
turn with its 
audio.out.first
latencies, mic window totals, errors and closes, and the last 25 entries.
可使用以下任意方式读取日志，均无需依赖应用的工具链：
bash
f=.voice-logs/<id>.ndjson
jq -r 'select(.kind|IN("start","token.ok","mic.ok","env","ws.open","ws.close","server.token","stop","error")) | "\(.t // .ts)ms \(.kind) \(.type // "") \(.message // "")"' $f   # milestones and errors
jq -r 'select(.kind=="server") | .type' $f | sort | uniq -c | sort -rn                      # server event counts
jq -c 'select(.kind|IN("audio.out.first","audio.out"))' $f                                  # per-turn latency, gaps, underruns
jq -c 'select(.kind=="audio.in")' $f                                                        # mic windows, rms
也可以直接读取文件；日志按时间顺序排列，每行一个JSON对象。服务端条目只有
ts
字段；可通过
start
条目的
ts
将服务端时间与客户端时钟对齐。如果用户需要摘要命令，用应用的语言实现一个，输出内容包括：关键里程碑事件、服务端事件计数、每个
audio.out
轮次的一行摘要（含对应
audio.out.first
的延迟）、mic窗口统计、错误与关闭事件，以及最后25条日志。

5. Fix loop

5. 修复循环

  1. Instrument (steps 1–4) if the app has no 
    .voice-logs
    pipeline yet.
  2. User tests in the real app and describes the run.
  3. Read the log around the failing 
    t
    .
  4. Match symptom → signature → fix (table below). No matching signature: add logging first, re-test, then fix.
  5. Fix one thing, re-test, confirm the signature is gone in a fresh log.
  6. Write it down: append a confirmed row under "Confirmed from sessions". If the fix changes how voice should be built, update 
    /add-voice
    too.
  1. 如果应用还没有
    .voice-logs
    日志管道，先埋点（执行步骤1-4）。
  2. 用户测试：在真实应用中测试并描述本次运行的情况。
  3. 读取日志：定位到故障发生的
    t
    时间点附近的日志。
  4. 匹配：症状 → 特征 → 修复方案（见下表）。如果没有匹配的特征，先补充日志，重新测试后再修复。
  5. 修复一个问题，重新测试，确认在新的日志中该特征已消失。
  6. 记录归档：在“会话确认记录”下追加一条已验证的记录。如果该修复改变了语音功能的构建方式，同时更新
    /add-voice
    技能。

Symptom → log signature → fix (starter rows)

症状 → 日志特征 → 修复方案（初始条目）

Symptom (user)Signature (log)Fix
Silence, but transcript appears
env.play_state
or 
audio.out.first.play_state
= 
suspended
Create and resume the playback audio context inside the user gesture; one context per session, not per turn
Assistant interrupts itself
speech_started
with 
phase=speaking
; 
audio.in.rms_max
rises only during playback		Echo. Confirm with headphones (if it stops, it is echo). Keep echo cancellation on, lower speaker volume, or gate mic sends while 
speaking
Choppy, stuttering
audio.out.underruns
> 0, 
drain_ms_max
high, 
max_gap_ms
far above chunk length		Schedule a small lead (150–250 ms) before the first chunk plays; do not rebuild the audio context per turn
Crackle, wrong pitch or speed
session.update
rate ≠ buffer rate; odd 
bytes
One rate everywhere (
audio.input/output.format.rate
, player buffer); even-byte alignment
Never connects, or closes at once
ws.close
before 
session.updated
; 
server.token.ok=false
Mint a token per click (300 s), protocol 
xai-client-secret.<token>
, 
model
in URL; read 
server.token.status
and 
upstream
Mic does nothing
audio.in.rms_max
≈ 0 in every window; 
mic.ok.label
unexpected		Wrong device or OS permission; check 
mic.ok.settings
, 
label
, 
mic_state
No user transcriptno 
conversation.item.input_audio_transcription.updated
Set 
audio.input.transcription.model: "grok-transcribe"
in 
session.update
Slow first word
audio.out.first.since_speech_stopped_ms
high		Try 
reasoning.effort: "none"
; shorter 
instructions
; check 
token.ok.ms
and 
ws.open.ms
for connect cost
User text appears after the reply
...transcription.updated
t
> 
response.created
t
Create the user row on 
input_audio_buffer.committed
(
item_id
), fill it on 
updated
First words cut off
ws.open.ms
large and 
audio.flush.chunks
at the buffer cap		Start mic before the socket, buffer early audio, raise the pre-open cap
用户反馈症状日志特征修复方案
有静音，但转录文本正常显示
env.play_state
或 
audio.out.first.play_state
为 
suspended
在用户手势交互中创建并恢复播放音频上下文；每个会话使用一个上下文，而非每轮对话新建一个
助手会打断自己说话出现
speech_started
事件且
phase=speaking
；
audio.in.rms_max
仅在播放时升高		回声问题。用耳机测试确认（如果戴耳机后问题消失，就是回声导致）。保持回声消除开启，降低扬声器音量，或在
speaking
阶段暂停mic发送
语音卡顿、断断续续
audio.out.underruns
> 0，
drain_ms_max
偏高，
max_gap_ms
远大于块长度		在首个音频块播放前预留一小段缓冲时间（150-250毫秒）；不要每轮对话重建音频上下文
有杂音、音调或速度异常
session.update
的采样率与缓冲区采样率不一致；
bytes
值异常		全链路使用统一采样率（
audio.input/output.format.rate
、播放器缓冲区）；保证字节对齐为偶数
无法连接，或连接后立即断开
session.updated
之前就出现
ws.close
；
server.token.ok=false
每次点击时生成一个token（有效期300秒），协议格式为
xai-client-secret.<token>
，URL中携带
model
参数；检查
server.token.status
和
upstream
Mic完全没反应每个窗口的
audio.in.rms_max
都≈0；
mic.ok.label
不符合预期		设备选择错误或系统权限问题；检查
mic.ok.settings
、
label
、
mic_state
没有用户转录文本没有
conversation.item.input_audio_transcription.updated
事件		在
session.update
中设置
audio.input.transcription.model: "grok-transcribe"
首词响应慢
audio.out.first.since_speech_stopped_ms
偏高		尝试设置
reasoning.effort: "none"
；缩短
instructions
长度；检查
token.ok.ms
和
ws.open.ms
排查连接耗时
用户文本在回复之后才出现
...transcription.updated
的
t
大于
response.created
的
t
在
input_audio_buffer.committed
事件时创建用户消息行（通过
item_id
），在
updated
事件时填充内容
首字被截断
ws.open.ms
较长且
audio.flush.chunks
达到缓冲区上限		在建立socket连接前启动mic采集，提前缓冲音频，提高预开缓冲区上限

Confirmed from sessions

会话确认记录

Append after a fix is verified in a fresh log. Format: 
YYYY-MM-DD · symptom · signature · fix · file(s)
.
在新日志中验证修复生效后，在此追加记录。格式：
YYYY-MM-DD · 症状 · 特征 · 修复方案 · 修改文件
。

Rules

规则

  • Plan first; no edits before the user aligns on the change list.
  • Never tokens, keys, or raw audio. Audio becomes byte counts.
  • Dev only. The sink returns 404 in production unless 
    VOICE_LOG=1
    . 
    .voice-logs/
    is gitignored.
  • Logging never throws into the voice path.
  • Aggregate audio; never log per chunk.
  • One change per re-test so the log tells you which fix worked.
  • Do not invent xAI event names; confirm in https://docs.x.ai/developers/model-capabilities/audio/speech-to-speech before adding a signature.
  • 先制定方案；在用户确认修改清单前，不修改任何文件。
  • 绝不记录token、密钥或原始音频。音频仅记录字节数。
  • 仅用于开发环境。生产环境下sink返回404，除非设置了
    VOICE_LOG=1
    。
    .voice-logs/
    目录已加入
    .gitignore
    。
  • 日志功能绝不抛出异常影响语音链路。
  • 音频数据做聚合处理；绝不记录单个音频块的内容。
  • 每次重新测试只改一个问题，这样才能通过日志确认哪个修复生效了。
  • 不要臆造xAI事件名称；添加新的特征前，先在https://docs.x.ai/developers/model-capabilities/audio/speech-to-speech 中确认。