realtime

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

ElevenLabs Realtime

Realtime models are ElevenLabs models built for low-latency voice conversation. There are two ways to use them:

An ElevenAgents agent (an agent ID, the dashboard, or an
```
elevenlabs agents
```
project with
```
agent_configs/
```
): set the agent's Voice mode to Realtime with
```
conversation_config.realtime_model
```
, and ElevenLabs manages its LLM, TTS model, and ASR settings. See Run an agent on a Realtime model.
A browser app built on OpenAI's Realtime SDK: keep the SDK and point it at the OpenAI-compatible ElevenLabs Realtime API,
```
wss://api.elevenlabs.io/v1/convai/realtime
```
. See Migrate an app from OpenAI Realtime.

When the user is starting from scratch and it's unclear which they want, ask. An agent brings a managed system prompt, tools, knowledge base, widget, and telephony; the Realtime API is the quicker path for code that already speaks OpenAI's realtime events. For agent features beyond the voice mode (prompt, tools, workflows, widget), use the

agents

skill.

Realtime 模型是 ElevenLabs 为低延迟语音对话构建的模型。有两种使用方式：

一个 ElevenAgents Agent（Agent ID、dashboard，或带
```
agent_configs/
```
的
```
elevenlabs agents
```
项目）：通过
```
conversation_config.realtime_model
```
将 Agent 的 Voice 模式设为 Realtime，由 ElevenLabs 管理其 LLM、TTS 模型和 ASR 设置。参见在 Realtime 模型上运行 Agent。
基于 OpenAI Realtime SDK 构建的浏览器应用：继续使用该 SDK，并将其指向兼容 OpenAI 的 ElevenLabs Realtime API
```
wss://api.elevenlabs.io/v1/convai/realtime
```
。参见从 OpenAI Realtime 迁移应用。

当用户从头开始且尚不明确需要哪种方案时，先询问。Agent 带有托管的 system prompt、tools、knowledge base、widget 和 telephony；对于已使用 OpenAI realtime 事件的代码，Realtime API 是更快捷的路径。对于超出 Voice 模式范围的 Agent 功能（prompt、tools、workflow、widget），使用

agents

skill。

Models

模型

Model	Model ID	Optimized for
Eleven Realtime v1 Mini	`eleven_realtime_v1_mini`	Lowest latency and cost. The default.
Eleven Realtime v1 Max	`eleven_realtime_v1_max`	Reasons before it responds, for complex instructions and multi-step tool use. Costs more.

Start with Mini. Choose Max when the work involves long or complex instructions or multi-step tool calls and Mini's responses aren't good enough, or when an app being migrated was on a larger OpenAI model for reasoning-heavy work. Agents add compliance and data residency constraints on the choice; see Choose the model.

模型	模型 ID	优化目标
Eleven Realtime v1 Mini	`eleven_realtime_v1_mini`	延迟和成本最低。默认选项。
Eleven Realtime v1 Max	`eleven_realtime_v1_max`	在响应前进行推理，适合复杂指令和多步骤工具调用。成本更高。

从 Mini 开始。当任务涉及长或复杂指令、多步骤工具调用且 Mini 的响应效果不够好时，选择 Max；或者当待迁移应用原本在推理密集型任务中使用更大的 OpenAI 模型时，也应选择 Max。Agent 还会对模型选择施加合规性和数据驻留约束；参见选择模型。

Run an agent on a Realtime model

在 Realtime 模型上运行 Agent

Every voice agent has a Voice mode. In Custom mode the agent's LLM, TTS model, and ASR settings are chosen individually. In Realtime mode the agent runs on a Realtime model and ElevenLabs manages those settings. The mode is one field:

conversation_config.realtime_model

set to a model ID means Realtime,

null

means Custom.

The system prompt, first message, voice, languages, temperature, tools, knowledge base, and conversation flow settings (turn eagerness, timeouts, interruptions) work the same in both modes.

每个语音 Agent 都有 Voice 模式。在 Custom 模式下，Agent 的 LLM、TTS 模型和 ASR 设置分别选择。在 Realtime 模式下，Agent 运行在 Realtime 模型上，由 ElevenLabs 管理这些设置。该模式只是一个字段：将

conversation_config.realtime_model

设置为模型 ID 即表示使用 Realtime，设置为

null

则表示使用 Custom。

system prompt、首条消息、voice、languages、temperature、tools、knowledge base 和对话流设置（turn eagerness、timeouts、interruptions）在两种模式下的行为相同。

Before you start

开始之前

These steps call the ElevenLabs API. The CLI and SDKs read
```
ELEVENLABS_API_KEY
```
from the environment; if it isn't set, use the
```
setup-api-key
```
skill. Never ask the user to paste the key into the conversation.
Realtime models are rolling out gradually. If the API rejects
```
realtime_model
```
with
```
Realtime models are not available for this workspace.
```
, the workspace doesn't have access yet. Stop and tell the user; no configuration change fixes it.

这些步骤会调用 ElevenLabs API。CLI 和 SDK 从环境变量读取
```
ELEVENLABS_API_KEY
```
；如果未设置，使用
```
setup-api-key
```
skill。绝不要让用户将密钥粘贴到对话中。
Realtime 模型正在逐步推出。如果 API 拒绝
```
realtime_model
```
并返回
```
Realtime models are not available for this workspace.
```
，说明该 workspace 尚无访问权限。请停止操作并告知用户；任何配置更改都无法解决此问题。

Step 1: Check the agent can use Realtime

第 1 步：检查 Agent 能否使用 Realtime

For an existing agent, fetch its configuration first (

elevenlabs agents pull --agent <agent-id>

, or

agents.get

in the SDK). Keep the agent in Custom mode, and tell the user why, if any of these apply:

Blocker	Where to look
Custom LLM or Amazon Bedrock	`conversation_config.agent.prompt.llm` is `custom-llm` or a Bedrock model
Created through another ElevenLabs product, such as Speech Engine	Ask the user if unsure; only ElevenAgents agents can use a Realtime model
Needs a specific LLM or TTS model	Ask the user whether the current `agent.prompt.llm` or `tts.model_id` is a requirement
Workflow nodes override a managed setting or `agent.prompt.max_tokens`	Overrides on the nodes under the agent's `workflow`
Transfers to or from agents in Custom mode	`transfer_to_agent` tools and `agent_transfer` workflow nodes on this agent, and on any agent that transfers into this one
Text-only agent	`conversation_config.conversation.text_only` is `true` ; Realtime models are built for voice

A workflow override can be removed and a transfer partner can be switched to Realtime too, but only with the user's agreement: both change how other parts of their setup behave.

对于现有 Agent，先获取其配置（

elevenlabs agents pull --agent <agent-id>

，或使用 SDK 中的

agents.get

）。让 Agent 保持 Custom 模式；如果出现以下任一情况，请向用户说明原因：

阻碍项	检查位置
自定义 LLM 或 Amazon Bedrock	`conversation_config.agent.prompt.llm` 为 `custom-llm` 或某个 Bedrock 模型
通过其他 ElevenLabs 产品创建，例如 Speech Engine	如果不确定，请询问用户；只有 ElevenAgents Agent 可以使用 Realtime 模型
需要特定的 LLM 或 TTS 模型	询问用户当前的 `agent.prompt.llm` 或 `tts.model_id` 是否为硬性要求
Workflow 节点覆盖了托管设置或 `agent.prompt.max_tokens`	Agent 的 `workflow` 下各节点上的覆盖项
与 Custom 模式 Agent 之间进行 transfer	此 Agent 上的 `transfer_to_agent` tools 和 `agent_transfer` workflow 节点，以及所有会转入此 Agent 的 Agent 上的对应项
纯文本 Agent	`conversation_config.conversation.text_only` 为 `true` ；Realtime 模型专为语音而构建

Workflow 覆盖项可以移除，transfer 对象也可以切换为 Realtime，但必须得到用户同意：这两项操作都会改变配置中其他部分的行为。

Step 2: Choose the model

第 2 步：选择模型

Start with

eleven_realtime_v1_mini

. Choose

eleven_realtime_v1_max

when the agent follows long or complex instructions or relies on multi-step tool calls and Mini isn't good enough, or when the agent must comply with Zero Retention Mode or HIPAA. Then check availability:

Environment	`eleven_realtime_v1_mini`	`eleven_realtime_v1_max`
US	Available	Available
EU data residency	Available	Available
India data residency	Available	Not available
Singapore data residency	Available	Not available
Zero Retention Mode and HIPAA	Not available	Available in the US and EU

Zero Retention Mode applies whether it's enforced on the workspace or enabled on the agent. Ask the user when the workspace's data residency or compliance settings aren't clear from the project. If neither model is available for the agent's environment, keep it in Custom mode.

Max also needs

agent.prompt.max_tokens

of at least 1024, or

-1

for no limit.

从

eleven_realtime_v1_mini

开始。当 Agent 需要遵循长或复杂指令、依赖多步骤工具调用而 Mini 的效果不够好时，选择

eleven_realtime_v1_max

；当待迁移应用原本在推理密集型任务中使用更大的 OpenAI 模型时，也应选择 Max。Agent 还会对模型选择施加合规性和数据驻留约束；参见选择模型。然后检查可用性：

环境	`eleven_realtime_v1_mini`	`eleven_realtime_v1_max`
美国	可用	可用
欧盟数据驻留	可用	可用
印度数据驻留	可用	不可用
新加坡数据驻留	可用	不可用
Zero Retention Mode 和 HIPAA	不可用	美国和欧盟可用

无论 Zero Retention Mode 是在 workspace 层强制执行，还是在 Agent 层启用，限制都同样适用。如果无法从项目中确定 workspace 的数据驻留或合规设置，请询问用户。如果 Agent 的环境不支持任何模型，请保持 Custom 模式。

Max 还要求

agent.prompt.max_tokens

至少为 1024，或设为

-1

表示不限制。

Step 3: Set the model

第 3 步：设置模型

On an existing agent, change only

realtime_model

; leave the rest of the configuration alone.

With the CLI, pull the agent, set the field in its file under

agent_configs/

, and push:

bash

elevenlabs agents pull --agent <agent-id>

json

{
  "conversation_config": {
    "realtime_model": "eleven_realtime_v1_mini"
  }
}

bash

elevenlabs agents push --agent <agent-id>

With the SDKs:

python

from elevenlabs import ElevenLabs

elevenlabs = ElevenLabs()

elevenlabs.conversational_ai.agents.update(
    agent_id="<agent-id>",
    conversation_config={
        "realtime_model": "eleven_realtime_v1_mini",
    },
)

typescript

import { ElevenLabsClient } from "@elevenlabs/elevenlabs-js";

const elevenlabs = new ElevenLabsClient();

await elevenlabs.conversationalAi.agents.update("<agent-id>", {
  conversationConfig: {
    realtimeModel: "eleven_realtime_v1_mini",
  },
});

For a new agent, put

realtime_model

(

realtimeModel

in TypeScript) in the

conversation_config

passed to

agents.create

, alongside the prompt, first message, and voice. Don't pick an LLM or TTS model for it; a Realtime agent ignores them.

To turn Realtime off, set

realtime_model

null

(

None

in Python). The agent goes back to the LLM, TTS model, and ASR settings stored on it.

对于现有 Agent，只更改

realtime_model

；其余配置保持不变。

使用 CLI 时，拉取 Agent，在其

agent_configs/

下的文件中设置该字段，然后推送：

bash

elevenlabs agents pull --agent <agent-id>

json

{
  "conversation_config": {
    "realtime_model": "eleven_realtime_v1_mini"
  }
}

bash

elevenlabs agents push --agent <agent-id>

使用 SDK 时：

python

from elevenlabs import ElevenLabs

elevenlabs = ElevenLabs()

elevenlabs.conversational_ai.agents.update(
    agent_id="<agent-id>",
    conversation_config={
        "realtime_model": "eleven_realtime_v1_mini",
    },
)

typescript

import { ElevenLabsClient } from "@elevenlabs/elevenlabs-js";

const elevenlabs = new ElevenLabsClient();

await elevenlabs.conversationalAi.agents.update("<agent-id>", {
  conversationConfig: {
    realtimeModel: "eleven_realtime_v1_mini",
  },
});

对于新 Agent，将

realtime_model

（TypeScript 中为

realtimeModel

）放入传给

agents.create

的

conversation_config

，并与 prompt、首条消息和 voice 一起提供。不要为其选择 LLM 或 TTS 模型；Realtime Agent 会忽略它们。

要关闭 Realtime，将

realtime_model

设置为

null

（Python 中为

None

）。Agent 将重新使用其存储的 LLM、TTS 模型和 ASR 设置。

Managed settings

托管设置

While a Realtime model is set, the agent ignores the stored values of these fields in

conversation_config

. They come back into use when the agent switches to Custom, so don't clear them, and don't edit them to tune a Realtime agent:

Setting	Field
LLM	`agent.prompt.llm`
Reasoning effort	`agent.prompt.reasoning_effort`
Thinking budget	`agent.prompt.thinking_budget`
Parallel tool calls	`agent.prompt.enable_parallel_tool_calls`
Cascade timeout	`agent.prompt.cascade_timeout_seconds`
TTS model	`tts.model_id`
Expressive mode	`tts.expressive_mode`
ASR provider	`asr.provider`

agent.prompt.max_tokens

keeps applying. Temperature does too, but the dashboard hides it in Realtime mode, so change it through the CLI or API. If the Realtime model errors or doesn't respond in time, the agent falls back to its backup LLMs (LLM cascading); the Realtime model sets how long it waits.

设置 Realtime 模型期间，Agent 会忽略

conversation_config

中以下字段的已存储值。切换回 Custom 后这些值会重新生效，因此不要清除它们，也不要编辑它们来调优 Realtime Agent：

设置	字段
LLM	`agent.prompt.llm`
Reasoning effort	`agent.prompt.reasoning_effort`
Thinking budget	`agent.prompt.thinking_budget`
Parallel tool calls	`agent.prompt.enable_parallel_tool_calls`
Cascade timeout	`agent.prompt.cascade_timeout_seconds`
TTS model	`tts.model_id`
Expressive mode	`tts.expressive_mode`
ASR provider	`asr.provider`

agent.prompt.max_tokens

仍然生效。Temperature 也会继续生效，但 dashboard 在 Realtime 模式下会隐藏它，因此要通过 CLI 或 API 修改。如果 Realtime 模型报错或未及时响应，Agent 会回退到备用 LLM（LLM cascading）；Realtime 模型决定等待时长。

Step 4: Publish and verify

第 4 步：发布并验证

The Step 1 rules are checked when the agent is published, not when a draft is saved: a draft that breaks one saves, then publishing fails with an error naming the setting to change.

Fetch the agent and confirm
```
conversation_config.realtime_model
```
holds the chosen model. If the push, update, or the user's publish in the dashboard fails, match the error below.
Find every agent that transfers into this one and confirm it's in Realtime mode too. A Custom-mode agent's transfer into a Realtime agent is only caught when that agent is next published; until then the transfer is rejected mid-conversation and the caller stays with the current agent.
Have the user hold a test conversation.

第 1 步中的规则是在发布 Agent 时检查，而不是保存草稿时检查：违反其中一项的草稿仍可保存，但随后发布会失败，并指出需要更改的设置。

获取 Agent，确认
```
conversation_config.realtime_model
```
持有所选模型。如果 push、update 或用户在 dashboard 中的 publish 失败，请对照下表匹配错误。
找出所有转入此 Agent 的 Agent，并确认它们也处于 Realtime 模式。Custom 模式 Agent 向 Realtime Agent 的 transfer，只有在该 Agent 下次发布时才会被发现；在此之前，该 transfer 会在对话中途被拒绝，调用方会留在当前 Agent 中。
让用户进行一次测试对话。

Errors

错误

Error	Cause and fix
`Realtime models are not available for this workspace.`	The workspace doesn't have access yet. Nothing to fix in the configuration.
`Transfers between default and Realtime model modes are not supported.`	A transfer to or from a Custom-mode agent ("default" means Custom). Put both agents in the same mode.
An error naming the Realtime model	The model isn't available under the agent's compliance or data residency settings. See Step 2.
Publishing fails naming a setting	One of the Step 1 blockers or Max's token limit. Change that setting, or keep the agent in Custom.

错误	原因与修复
`Realtime models are not available for this workspace.`	该 workspace 尚无访问权限。配置中没有任何内容需要修复。
`Transfers between default and Realtime model modes are not supported.`	向 Custom 模式 Agent（“default”指 Custom）发起 transfer 或从其发起 transfer。请将两个 Agent 设为同一模式。
出现指明 Realtime 模型的错误	根据 Agent 的合规性或数据驻留设置，该模型不可用。参见第 2 步。
发布失败并指明某项设置	存在第 1 步中的某个阻碍项，或触发了 Max 的 token 上限。更改该设置，或让 Agent 保持 Custom 模式。

Pricing

定价

A Realtime agent is billed the standard ElevenAgents per-minute rate plus its input, cached input, and output tokens at the Realtime model's rates; Max costs more per token than Mini. Point the user to https://elevenlabs.io/pricing rather than quoting rates.

Realtime Agent 的费用由 ElevenAgents 标准每分钟费率，加上按 Realtime 模型费率计费的输入、缓存输入和输出 token 构成；Max 的每 token 成本高于 Mini。请将用户引导至 https://elevenlabs.io/pricing，不要直接报价。

Migrate an app from OpenAI Realtime

从 OpenAI Realtime 迁移应用

ElevenLabs Realtime speaks the same wire protocol as OpenAI's GA Realtime API: the same client and server events over WebSocket. An app built on OpenAI's Realtime SDK keeps that SDK; the migration changes where credentials are minted, which URL the transport connects to, and the model, voice, and session configuration.

The one structural change: ElevenLabs Realtime is WebSocket-only. In a browser,

RealtimeSession

defaults to WebRTC, which wires the microphone and speaker for you. After switching to the WebSocket transport, the app has to capture and play audio itself.

This covers apps where the browser opens the realtime connection. It doesn't cover apps where a server opens the connection and streams audio to its clients over its own protocol, such as a bespoke WebSocket relay or a telephony bridge. Connecting to ElevenLabs Realtime from a server should work, but this skill has no steps for it. Tell the user so, and suggest they open an issue or a pull request at https://github.com/elevenlabs/skills describing their setup, rather than guessing at the server-side migration.

ElevenLabs Realtime 与 OpenAI GA Realtime API 使用相同的 wire protocol：通过 WebSocket 传输相同的客户端事件和服务端事件。基于 OpenAI Realtime SDK 构建的应用可以保留该 SDK；迁移会改变凭据的签发位置、transport 的连接 URL，以及模型、voice 和 session 配置。

唯一的结构性变化是：ElevenLabs Realtime 仅支持 WebSocket。在浏览器中，

RealtimeSession

默认使用 WebRTC，它会自动连接麦克风和扬声器。切换到 WebSocket transport 后，应用必须自行采集和播放音频。

本指南涵盖由浏览器建立 realtime 连接的应用。不涵盖由服务器建立连接、再通过自有协议将音频流式传输给客户端的应用，例如自定义 WebSocket relay 或 telephony bridge。从服务器连接 ElevenLabs Realtime 理论上应该可行，但本 skill 未提供相关步骤。请如实告知用户，并建议其在 https://github.com/elevenlabs/skills 提交 issue 或 pull request，描述其配置，而不是猜测服务器端迁移方式。

Step 0: Inventory the app

第 0 步：盘点应用

Find every place the app touches OpenAI Realtime before editing anything:

bash

grep -rnE "api\.openai\.com/v1/realtime|client_secrets|realtime/sessions|@openai/agents-realtime|@openai/agents/realtime|gpt-realtime|gpt-4o(-mini)?-realtime|OpenAIRealtimeWebRTC|OpenAIRealtimeWebSocket|RealtimeSession|OPENAI_API_KEY" --include='*.{ts,tsx,js,jsx,mjs,py}' .

Classify what you find, because each part migrates differently:

Credential minting: a server route that calls
```
/v1/realtime/client_secrets
```
(GA) or
```
/v1/realtime/sessions
```
(beta). Migrate in Step 1.

Browser session:

RealtimeSession

OpenAIRealtimeWebRTC

OpenAIRealtimeWebSocket

, or a hand-rolled

WebSocket

RTCPeerConnection

. Migrate in Steps 2 to 4.

Server-side realtime connections: a Node or Python process that opens the realtime socket itself, for example a relay that streams audio to clients over its own WebSocket protocol, a telephony bridge,
```
openai
```
Python's
```
client.realtime.connect()
```
, or a
```
RealtimeSession
```
running in Node with a full API key. Not covered; see the scope above. If the app also has a browser session, migrate only that part, and only with the user's agreement.
Features beyond a voice conversation: function tools, handoffs, guardrails, MCP servers, image input, SIP. ElevenLabs documents the voice conversation flow only. Leave this code in place, list it for the user as unverified against ElevenLabs, and have them test it.

OPENAI_API_KEY

often serves other OpenAI calls too. Remove it only if nothing else uses it.

在编辑任何内容之前，找出应用使用 OpenAI Realtime 的所有位置：

bash

grep -rnE "api\.openai\.com/v1/realtime|client_secrets|realtime/sessions|@openai/agents-realtime|@openai/agents/realtime|gpt-realtime|gpt-4o(-mini)?-realtime|OpenAIRealtimeWebRTC|OpenAIRealtimeWebSocket|RealtimeSession|OPENAI_API_KEY" --include='*.{ts,tsx,js,jsx,mjs,py}' .

对找到的内容进行分类，因为各部分的迁移方式不同：

Credential minting：调用
```
/v1/realtime/client_secrets
```
（GA）或
```
/v1/realtime/sessions
```
（beta）的服务器路由。在第 1 步迁移。

Browser session：

RealtimeSession

、

OpenAIRealtimeWebRTC

、

OpenAIRealtimeWebSocket

，或自行实现的

WebSocket

RTCPeerConnection

。在第 2 至 4 步迁移。

服务器端 realtime 连接：自行打开 realtime socket 的 Node 或 Python 进程，例如通过自有 WebSocket 协议向客户端流式传输音频的 relay、telephony bridge、
```
openai
```
Python 的
```
client.realtime.connect()
```
，或在 Node 中使用完整 API key 运行的
```
RealtimeSession
```
。不在本指南范围内；参见上述适用范围。如果应用还包含 browser session，仅迁移该部分，并且必须得到用户同意。
超出语音对话范围的功能：function tools、handoffs、guardrails、MCP servers、image input、SIP。ElevenLabs 仅记录语音对话流程。保留这些代码不动，将其列为尚未针对 ElevenLabs 验证的项目，并让用户进行测试。

OPENAI_API_KEY

通常也用于其他 OpenAI 调用。只有在没有其他用途时才将其移除。

Step 1: Mint the client secret from ElevenLabs

第 1 步：从 ElevenLabs 签发 Client Secret

The browser never sees the ElevenLabs API key. A server route mints a short-lived client secret and returns only its

value

What	OpenAI	ElevenLabs
URL	`POST https://api.openai.com/v1/realtime/client_secrets`	`POST https://api.elevenlabs.io/v1/convai/realtime/client_secrets`
Auth header	`Authorization: Bearer $OPENAI_API_KEY`	`xi-api-key: $ELEVENLABS_API_KEY`
`session.model`	`gpt-realtime` , …	`eleven_realtime_v1_mini` or `eleven_realtime_v1_max`
Response	`{ value, expires_at, session }`	Same shape

The request body keeps the GA shape,

{ session, expires_after }

. The session config sent here is the authoritative initial session for that credential. The

model

string selects the agent behavior directly; there is no

agent_id

Mint with the ElevenLabs SDK (

npm install @elevenlabs/elevenlabs-js

pip install elevenlabs

), which sends the

xi-api-key

header for you:

typescript

import { ElevenLabsClient } from "@elevenlabs/elevenlabs-js";

const elevenlabs = new ElevenLabsClient();

const clientSecret =
  await elevenlabs.conversationalAi.realtime.createClientSecret({
    session: {
      type: "realtime",
      model: "eleven_realtime_v1_mini",
      instructions:
        "You are a helpful, friendly voice assistant. Keep replies concise.",
      outputModalities: ["audio"],
      audio: {
        input: {
          format: { type: "audio/pcm", rate: 24000 },
          turnDetection: { type: "semantic_vad", eagerness: "medium" },
        },
        output: {
          format: { type: "audio/pcm" },
          voice: "brian",
        },
      },
    },
    expiresAfter: { anchor: "created_at", seconds: 600 },
  });

python

import os

from elevenlabs import (
    CreateClientSecretRequest,
    ExpiresAfter,
    OpenAiAudioConfigInput,
    OpenAiAudioInputConfigInput,
    OpenAiAudioInputFormat,
    OpenAiAudioOutputConfigInput,
    OpenAiAudioOutputFormat,
    OpenAiSessionConfigInput,
    OpenAiTurnDetection,
)
from elevenlabs.client import ElevenLabs

elevenlabs = ElevenLabs(api_key=os.environ["ELEVENLABS_API_KEY"])

client_secret = elevenlabs.conversational_ai.realtime.create_client_secret(
    request=CreateClientSecretRequest(
        session=OpenAiSessionConfigInput(
            type="realtime",
            model="eleven_realtime_v1_mini",
            instructions="You are a helpful, friendly voice assistant. Keep replies concise.",
            output_modalities=["audio"],
            audio=OpenAiAudioConfigInput(
                input=OpenAiAudioInputConfigInput(
                    format=OpenAiAudioInputFormat(type="audio/pcm", rate=24000),
                    turn_detection=OpenAiTurnDetection(type="semantic_vad", eagerness="medium"),
                ),
                output=OpenAiAudioOutputConfigInput(
                    format=OpenAiAudioOutputFormat(type="audio/pcm"),
                    voice="brian",
                ),
            ),
        ),
        expires_after=ExpiresAfter(anchor="created_at", seconds=600),
    ),
)

Return only

clientSecret.value

(

client_secret.value

in Python) to the browser.

Keep the route's existing framework, path, and response contract; only the upstream call changes. If the app minted with the beta

/v1/realtime/sessions

endpoint, its frontend reads

client_secret.value

. Either update the frontend to read the top-level

value

, or have the route return

{ client_secret: { value } }

so the frontend stays untouched.

浏览器绝不能看到 ElevenLabs API key。服务器路由负责签发短期 client secret，并且只将其

value

返回给浏览器。

项目	OpenAI	ElevenLabs
URL	`POST https://api.openai.com/v1/realtime/client_secrets`	`POST https://api.elevenlabs.io/v1/convai/realtime/client_secrets`
Auth header	`Authorization: Bearer $OPENAI_API_KEY`	`xi-api-key: $ELEVENLABS_API_KEY`
`session.model`	`gpt-realtime` 、…	`eleven_realtime_v1_mini` 或 `eleven_realtime_v1_max`
响应	`{ value, expires_at, session }`	相同结构

请求体保持 GA 结构

{ session, expires_after }

。此处发送的 session 配置是该凭据的权威初始 session。

model

字符串直接选择 Agent 行为；没有

agent_id

。

使用 ElevenLabs SDK 签发（

npm install @elevenlabs/elevenlabs-js

或

pip install elevenlabs

），它会替你发送

xi-api-key

header：

typescript

import { ElevenLabsClient } from "@elevenlabs/elevenlabs-js";

const elevenlabs = new ElevenLabsClient();

const clientSecret =
  await elevenlabs.conversationalAi.realtime.createClientSecret({
    session: {
      type: "realtime",
      model: "eleven_realtime_v1_mini",
      instructions:
        "You are a helpful, friendly voice assistant. Keep replies concise.",
      outputModalities: ["audio"],
      audio: {
        input: {
          format: { type: "audio/pcm", rate: 24000 },
          turnDetection: { type: "semantic_vad", eagerness: "medium" },
        },
        output: {
          format: { type: "audio/pcm" },
          voice: "brian",
        },
      },
    },
    expiresAfter: { anchor: "created_at", seconds: 600 },
  });

python

import os

from elevenlabs import (
    CreateClientSecretRequest,
    ExpiresAfter,
    OpenAiAudioConfigInput,
    OpenAiAudioInputConfigInput,
    OpenAiAudioInputFormat,
    OpenAiAudioOutputConfigInput,
    OpenAiAudioOutputFormat,
    OpenAiSessionConfigInput,
    OpenAiTurnDetection,
)
from elevenlabs.client import ElevenLabs

elevenlabs = ElevenLabs(api_key=os.environ["ELEVENLABS_API_KEY"])

client_secret = elevenlabs.conversational_ai.realtime.create_client_secret(
    request=CreateClientSecretRequest(
        session=OpenAiSessionConfigInput(
            type="realtime",
            model="eleven_realtime_v1_mini",
            instructions="You are a helpful, friendly voice assistant. Keep replies concise.",
            output_modalities=["audio"],
            audio=OpenAiAudioConfigInput(
                input=OpenAiAudioInputConfigInput(
                    format=OpenAiAudioInputFormat(type="audio/pcm", rate=24000),
                    turn_detection=OpenAiTurnDetection(type="semantic_vad", eagerness="medium"),
                ),
                output=OpenAiAudioOutputConfigInput(
                    format=OpenAiAudioOutputFormat(type="audio/pcm"),
                    voice="brian",
                ),
            ),
        ),
        expires_after=ExpiresAfter(anchor="created_at", seconds=600),
    ),
)

只将

clientSecret.value

（Python 中为

client_secret.value

）返回给浏览器。

保持路由现有的框架、路径和响应契约；仅更改上游调用。如果应用使用 beta

/v1/realtime/sessions

endpoint 签发凭据，其前端读取

client_secret.value

。要么将前端更新为读取顶层

value

，要么让路由返回

{ client_secret: { value } }

，这样前端无需改动。

Step 2: Convert the session config to the GA shape

第 2 步：将 Session 配置转换为 GA 结构

Send the GA session shape. Apps written against OpenAI's beta API use flat fields (

modalities

voice

input_audio_format

turn_detection

, …) that need converting, and ElevenLabs rejects the deprecated

modalities

field outright. This matters most for the mint request body and any hand-rolled

session.update

: the SDK's

config

option converts deprecated camelCase fields to the GA wire shape for you, though moving it to the GA form too removes the deprecation warnings.

Read Session config mapping for the full beta-to-GA field table, voice handling, and the camelCase form the SDK's

config

option uses.

发送 GA session 结构。针对 OpenAI beta API 编写的应用使用扁平字段（

modalities

、

voice

、

input_audio_format

、

turn_detection

等），这些字段需要转换，而 ElevenLabs 会直接拒绝已弃用的

modalities

字段。这一点对签发请求体和任何手动实现的

session.update

最为重要：SDK 的

config

选项可为你将已弃用的 camelCase 字段转换为 GA wire 结构，但迁移到 GA 结构本身也能消除弃用警告。

阅读 Session config 映射，了解完整的 beta 到 GA 字段对照表、voice 处理方式，以及 SDK 的

config

选项使用的 camelCase 形式。

Step 3: Point the browser session at ElevenLabs

第 3 步：将浏览器 Session 指向 ElevenLabs

Replace the transport with

OpenAIRealtimeWebSocket

aimed at the ElevenLabs URL, set the model on the session, and pass the minted

value

as the API key:

typescript

import {
  OpenAIRealtimeWebSocket,
  RealtimeAgent,
  RealtimeSession,
} from "@openai/agents-realtime";

const { value: clientSecret } = await fetch("/api/token").then((res) =>
  res.json(),
);

const agent = new RealtimeAgent({
  name: "Assistant",
  instructions:
    "You are a helpful, friendly voice assistant. Keep replies concise.",
});

const transport = new OpenAIRealtimeWebSocket({
  url: "wss://api.elevenlabs.io/v1/convai/realtime",
});

const session = new RealtimeSession(agent, {
  transport,
  model: "eleven_realtime_v1_mini",
  config: {
    outputModalities: ["audio"],
    audio: { output: { voice: "brian" } },
  },
});

await session.connect({ apiKey: clientSecret });

Keep the app's existing

RealtimeAgent

name, instructions, and event handlers. Drop WebRTC-only options (

audioElement

mediaStream

changePeerConnection

) and any

transport: "webrtc"

setting; they have no WebSocket equivalent.

将 transport 替换为指向 ElevenLabs URL 的

OpenAIRealtimeWebSocket

，在 session 上设置模型，并将签发的

value

作为 API key 传入：

typescript

import {
  OpenAIRealtimeWebSocket,
  RealtimeAgent,
  RealtimeSession,
} from "@openai/agents-realtime";

const { value: clientSecret } = await fetch("/api/token").then((res) =>
  res.json(),
);

const agent = new RealtimeAgent({
  name: "Assistant",
  instructions:
    "You are a helpful, friendly voice assistant. Keep replies concise.",
});

const transport = new OpenAIRealtimeWebSocket({
  url: "wss://api.elevenlabs.io/v1/convai/realtime",
});

const session = new RealtimeSession(agent, {
  transport,
  model: "eleven_realtime_v1_mini",
  config: {
    outputModalities: ["audio"],
    audio: { output: { voice: "brian" } },
  },
});

await session.connect({ apiKey: clientSecret });

保留应用现有的

RealtimeAgent

name、instructions 和 event handler。删除仅支持 WebRTC 的选项（

audioElement

、

mediaStream

、

changePeerConnection

）以及所有

transport: "webrtc"

设置；它们在 WebSocket 中没有对应项。

Step 4: Wire up audio (when the app was on WebRTC)

第 4 步：接入音频（当应用原来使用 WebRTC 时）

If the app previously used WebRTC (an explicit

OpenAIRealtimeWebRTC

transport: "webrtc"

, or no transport at all in a browser), it now has to:

Capture microphone audio as PCM16 at 24 kHz and stream it with
```
session.sendAudio(...)
```
.
Play PCM16 chunks from the session's
```
audio
```
event.
Stop playback on
```
audio_interrupted
```
. Without this, the assistant keeps talking over the user after a barge-in, because already-buffered audio keeps playing.
Release the microphone and close the session on teardown.

Read Browser audio over WebSocket for a working implementation with

wavtools

, the helper OpenAI uses in its own WebSocket examples, and its pitfalls.

Apps already on

OpenAIRealtimeWebSocket

have this wiring; check the sample rate is 24 kHz and keep it.

如果应用之前使用 WebRTC（显式使用

OpenAIRealtimeWebRTC

、

transport: "webrtc"

，或浏览器中完全未指定 transport），现在必须：

以 24 kHz 采集 PCM16 麦克风音频，并通过
```
session.sendAudio(...)
```
流式发送。
播放 session 的
```
audio
```
事件提供的 PCM16 数据块。
在
```
audio_interrupted
```
时停止播放。否则，用户 barge-in 后，Assistant 仍会继续盖过用户说话，因为已缓冲的音频会继续播放。
在 teardown 时释放麦克风并关闭 session。

阅读通过 WebSocket 处理浏览器音频，了解基于

wavtools

的可运行实现；这是 OpenAI 自己的 WebSocket 示例使用的 helper，以及其中的常见陷阱。

已经使用

OpenAIRealtimeWebSocket

的应用已有这套接线；检查采样率是否为 24 kHz 并保持不变。

Step 5: Verify

第 5 步：验证

Find where the server loads
```
OPENAI_API_KEY
```
from (
```
.env
```
,
```
.env.local
```
, deployment secrets) and ask the user to store
```
ELEVENLABS_API_KEY
```
there. Don't ask for the key in the conversation or write it into a file yourself. Add an empty
```
ELEVENLABS_API_KEY=
```
to
```
.env.example
```
if the project has one.
Once the user confirms the key is in place, run the app, start a conversation, and speak.

Confirm the

session.created

transport event reports the

eleven_realtime_*

model and the chosen voice:

typescript

session.on("transport_event", (event) => {
  if (event.type === "session.created") console.log(event);
});

Interrupt the assistant mid-sentence and confirm playback stops.
Tell the user which features from Step 0 remain unverified (tools, handoffs, guardrails, …).

找出服务器加载
```
OPENAI_API_KEY
```
的位置（
```
.env
```
、
```
.env.local
```
、部署 secrets），并请用户在那里存储
```
ELEVENLABS_API_KEY
```
。不要在对话中索要密钥，也不要自行将其写入文件。如果项目有
```
.env.example
```
，则添加空的
```
ELEVENLABS_API_KEY=
```
。
用户确认密钥已就位后，运行应用，开始对话并说话。

确认

session.created

transport event 报告

eleven_realtime_*

模型和所选 voice：

typescript

session.on("transport_event", (event) => {
  if (event.type === "session.created") console.log(event);
});

在 Assistant 说话途中打断它，并确认播放停止。
告知用户第 0 步中哪些功能仍未验证（tools、handoffs、guardrails 等）。

Troubleshooting

故障排除

Symptom	Cause and fix
WebSocket closes with code `4003`	Realtime isn't enabled for this ElevenLabs workspace. Contact ElevenLabs; the code can't fix this.
WebSocket closes with code `4001`	Client secret rejected: expired, or not the minted `value` passed as `apiKey` to `session.connect()` .
Mint returns 401	Still sending `Authorization: Bearer` ; ElevenLabs uses the `xi-api-key` header.
Mint or `session.update` is rejected	Beta-shaped fields such as `modalities` . Convert with the mapping.
Connects, but no audio either way	The app was on WebRTC and audio isn't wired yet. See Step 4.
Audio is noise, chipmunk-fast, or slowed down	Sample-rate or buffer mismatch. Capture and play at 24 kHz; see the pitfalls in the audio reference.
Assistant talks over the user after an interruption	Playback isn't stopped on `audio_interrupted` .

症状	原因与修复
WebSocket 以代码 `4003` 关闭	此 ElevenLabs workspace 尚未启用 Realtime。请联系 ElevenLabs；代码无法修复此问题。
WebSocket 以代码 `4001` 关闭	Client secret 被拒绝：已过期，或未将签发的 `value` 作为 `apiKey` 传给 `session.connect()` 。
签发返回 401	仍在发送 `Authorization: Bearer` ；ElevenLabs 使用 `xi-api-key` header。
签发或 `session.update` 被拒绝	使用了 `modalities` 等 beta 结构字段。请使用映射进行转换。
可以连接，但双向都没有音频	应用原来使用 WebRTC，但尚未接入音频。参见第 4 步。
音频嘈杂、速度异常地快或被放慢	采样率或缓冲区不匹配。以 24 kHz 采集和播放；参见音频参考中的常见陷阱。
打断后 Assistant 仍盖过用户说话	未在 `audio_interrupted` 时停止播放。

References

参考资料

Session config mapping
Browser audio over WebSocket

Session config 映射
通过 WebSocket 处理浏览器音频