ElevenLabs Realtime

Realtime models are ElevenLabs models built for low-latency voice conversation. There are two ways to use them:

An ElevenAgents agent (an agent ID, the dashboard, or an
```
elevenlabs agents
```
project with
```
agent_configs/
```
): set the agent's Voice mode to Realtime with
```
conversation_config.realtime_model
```
, and ElevenLabs manages its LLM, TTS model, and ASR settings. See Run an agent on a Realtime model.
A browser app built on OpenAI's Realtime SDK: keep the SDK and point it at the OpenAI-compatible ElevenLabs Realtime API,
```
wss://api.elevenlabs.io/v1/convai/realtime
```
. See Migrate an app from OpenAI Realtime.

When the user is starting from scratch and it's unclear which they want, ask. An agent brings a managed system prompt, tools, knowledge base, widget, and telephony; the Realtime API is the quicker path for code that already speaks OpenAI's realtime events. For agent features beyond the voice mode (prompt, tools, workflows, widget), use the

agents

skill.

Models

Model	Model ID	Optimized for
Eleven Realtime v1 Mini	`eleven_realtime_v1_mini`	Lowest latency and cost. The default.
Eleven Realtime v1 Max	`eleven_realtime_v1_max`	Reasons before it responds, for complex instructions and multi-step tool use. Costs more.

Start with Mini. Choose Max when the work involves long or complex instructions or multi-step tool calls and Mini's responses aren't good enough, or when an app being migrated was on a larger OpenAI model for reasoning-heavy work. Agents add compliance and data residency constraints on the choice; see Choose the model.

Run an agent on a Realtime model

Every voice agent has a Voice mode. In Custom mode the agent's LLM, TTS model, and ASR settings are chosen individually. In Realtime mode the agent runs on a Realtime model and ElevenLabs manages those settings. The mode is one field:

conversation_config.realtime_model

set to a model ID means Realtime,

null

means Custom.

The system prompt, first message, voice, languages, temperature, tools, knowledge base, and conversation flow settings (turn eagerness, timeouts, interruptions) work the same in both modes.

Before you start

These steps call the ElevenLabs API. The CLI and SDKs read
```
ELEVENLABS_API_KEY
```
from the environment; if it isn't set, use the
```
setup-api-key
```
skill. Never ask the user to paste the key into the conversation.
Realtime models are rolling out gradually. If the API rejects
```
realtime_model
```
with
```
Realtime models are not available for this workspace.
```
, the workspace doesn't have access yet. Stop and tell the user; no configuration change fixes it.

Step 1: Check the agent can use Realtime

For an existing agent, fetch its configuration first (

elevenlabs agents pull --agent <agent-id>

, or

agents.get

in the SDK). Keep the agent in Custom mode, and tell the user why, if any of these apply:

Blocker	Where to look
Custom LLM or Amazon Bedrock	`conversation_config.agent.prompt.llm` is `custom-llm` or a Bedrock model
Created through another ElevenLabs product, such as Speech Engine	Ask the user if unsure; only ElevenAgents agents can use a Realtime model
Needs a specific LLM or TTS model	Ask the user whether the current `agent.prompt.llm` or `tts.model_id` is a requirement
Workflow nodes override a managed setting or `agent.prompt.max_tokens`	Overrides on the nodes under the agent's `workflow`
Transfers to or from agents in Custom mode	`transfer_to_agent` tools and `agent_transfer` workflow nodes on this agent, and on any agent that transfers into this one
Text-only agent	`conversation_config.conversation.text_only` is `true` ; Realtime models are built for voice

A workflow override can be removed and a transfer partner can be switched to Realtime too, but only with the user's agreement: both change how other parts of their setup behave.

Step 2: Choose the model

Start with

eleven_realtime_v1_mini

. Choose

eleven_realtime_v1_max

when the agent follows long or complex instructions or relies on multi-step tool calls and Mini isn't good enough, or when the agent must comply with Zero Retention Mode or HIPAA. Then check availability:

Environment	`eleven_realtime_v1_mini`	`eleven_realtime_v1_max`
US	Available	Available
EU data residency	Available	Available
India data residency	Available	Not available
Singapore data residency	Available	Not available
Zero Retention Mode and HIPAA	Not available	Available in the US and EU

Zero Retention Mode applies whether it's enforced on the workspace or enabled on the agent. Ask the user when the workspace's data residency or compliance settings aren't clear from the project. If neither model is available for the agent's environment, keep it in Custom mode.

Max also needs

agent.prompt.max_tokens

of at least 1024, or

-1

for no limit.

Step 3: Set the model

On an existing agent, change only

realtime_model

; leave the rest of the configuration alone.

With the CLI, pull the agent, set the field in its file under

agent_configs/

, and push:

bash

elevenlabs agents pull --agent <agent-id>

json

{
  "conversation_config": {
    "realtime_model": "eleven_realtime_v1_mini"
  }
}

bash

elevenlabs agents push --agent <agent-id>

With the SDKs:

python

from elevenlabs import ElevenLabs

elevenlabs = ElevenLabs()

elevenlabs.conversational_ai.agents.update(
    agent_id="<agent-id>",
    conversation_config={
        "realtime_model": "eleven_realtime_v1_mini",
    },
)

typescript

import { ElevenLabsClient } from "@elevenlabs/elevenlabs-js";

const elevenlabs = new ElevenLabsClient();

await elevenlabs.conversationalAi.agents.update("<agent-id>", {
  conversationConfig: {
    realtimeModel: "eleven_realtime_v1_mini",
  },
});

For a new agent, put

realtime_model

(

realtimeModel

in TypeScript) in the

conversation_config

passed to

agents.create

, alongside the prompt, first message, and voice. Don't pick an LLM or TTS model for it; a Realtime agent ignores them.

To turn Realtime off, set

realtime_model

null

(

None

in Python). The agent goes back to the LLM, TTS model, and ASR settings stored on it.

Managed settings

While a Realtime model is set, the agent ignores the stored values of these fields in

conversation_config

. They come back into use when the agent switches to Custom, so don't clear them, and don't edit them to tune a Realtime agent:

Setting	Field
LLM	`agent.prompt.llm`
Reasoning effort	`agent.prompt.reasoning_effort`
Thinking budget	`agent.prompt.thinking_budget`
Parallel tool calls	`agent.prompt.enable_parallel_tool_calls`
Cascade timeout	`agent.prompt.cascade_timeout_seconds`
TTS model	`tts.model_id`
Expressive mode	`tts.expressive_mode`
ASR provider	`asr.provider`

agent.prompt.max_tokens

keeps applying. Temperature does too, but the dashboard hides it in Realtime mode, so change it through the CLI or API. If the Realtime model errors or doesn't respond in time, the agent falls back to its backup LLMs (LLM cascading); the Realtime model sets how long it waits.

Step 4: Publish and verify

The Step 1 rules are checked when the agent is published, not when a draft is saved: a draft that breaks one saves, then publishing fails with an error naming the setting to change.

Fetch the agent and confirm
```
conversation_config.realtime_model
```
holds the chosen model. If the push, update, or the user's publish in the dashboard fails, match the error below.
Find every agent that transfers into this one and confirm it's in Realtime mode too. A Custom-mode agent's transfer into a Realtime agent is only caught when that agent is next published; until then the transfer is rejected mid-conversation and the caller stays with the current agent.
Have the user hold a test conversation.

Errors

Error	Cause and fix
`Realtime models are not available for this workspace.`	The workspace doesn't have access yet. Nothing to fix in the configuration.
`Transfers between default and Realtime model modes are not supported.`	A transfer to or from a Custom-mode agent ("default" means Custom). Put both agents in the same mode.
An error naming the Realtime model	The model isn't available under the agent's compliance or data residency settings. See Step 2.
Publishing fails naming a setting	One of the Step 1 blockers or Max's token limit. Change that setting, or keep the agent in Custom.

Pricing

A Realtime agent is billed the standard ElevenAgents per-minute rate plus its input, cached input, and output tokens at the Realtime model's rates; Max costs more per token than Mini. Point the user to https://elevenlabs.io/pricing rather than quoting rates.

Migrate an app from OpenAI Realtime

ElevenLabs Realtime speaks the same wire protocol as OpenAI's GA Realtime API: the same client and server events over WebSocket. An app built on OpenAI's Realtime SDK keeps that SDK; the migration changes where credentials are minted, which URL the transport connects to, and the model, voice, and session configuration.

The one structural change: ElevenLabs Realtime is WebSocket-only. In a browser,

RealtimeSession

defaults to WebRTC, which wires the microphone and speaker for you. After switching to the WebSocket transport, the app has to capture and play audio itself.

This covers apps where the browser opens the realtime connection. It doesn't cover apps where a server opens the connection and streams audio to its clients over its own protocol, such as a bespoke WebSocket relay or a telephony bridge. Connecting to ElevenLabs Realtime from a server should work, but this skill has no steps for it. Tell the user so, and suggest they open an issue or a pull request at https://github.com/elevenlabs/skills describing their setup, rather than guessing at the server-side migration.

Step 0: Inventory the app

Find every place the app touches OpenAI Realtime before editing anything:

bash

grep -rnE "api\.openai\.com/v1/realtime|client_secrets|realtime/sessions|@openai/agents-realtime|@openai/agents/realtime|gpt-realtime|gpt-4o(-mini)?-realtime|OpenAIRealtimeWebRTC|OpenAIRealtimeWebSocket|RealtimeSession|OPENAI_API_KEY" --include='*.{ts,tsx,js,jsx,mjs,py}' .

Classify what you find, because each part migrates differently:

Credential minting: a server route that calls
```
/v1/realtime/client_secrets
```
(GA) or
```
/v1/realtime/sessions
```
(beta). Migrate in Step 1.

Browser session:

RealtimeSession

OpenAIRealtimeWebRTC

OpenAIRealtimeWebSocket

, or a hand-rolled

WebSocket

RTCPeerConnection

. Migrate in Steps 2 to 4.

Server-side realtime connections: a Node or Python process that opens the realtime socket itself, for example a relay that streams audio to clients over its own WebSocket protocol, a telephony bridge,
```
openai
```
Python's
```
client.realtime.connect()
```
, or a
```
RealtimeSession
```
running in Node with a full API key. Not covered; see the scope above. If the app also has a browser session, migrate only that part, and only with the user's agreement.
Features beyond a voice conversation: function tools, handoffs, guardrails, MCP servers, image input, SIP. ElevenLabs documents the voice conversation flow only. Leave this code in place, list it for the user as unverified against ElevenLabs, and have them test it.

OPENAI_API_KEY

often serves other OpenAI calls too. Remove it only if nothing else uses it.

Step 1: Mint the client secret from ElevenLabs

The browser never sees the ElevenLabs API key. A server route mints a short-lived client secret and returns only its

value

What	OpenAI	ElevenLabs
URL	`POST https://api.openai.com/v1/realtime/client_secrets`	`POST https://api.elevenlabs.io/v1/convai/realtime/client_secrets`
Auth header	`Authorization: Bearer $OPENAI_API_KEY`	`xi-api-key: $ELEVENLABS_API_KEY`
`session.model`	`gpt-realtime` , …	`eleven_realtime_v1_mini` or `eleven_realtime_v1_max`
Response	`{ value, expires_at, session }`	Same shape

The request body keeps the GA shape,

{ session, expires_after }

. The session config sent here is the authoritative initial session for that credential. The

model

string selects the agent behavior directly; there is no

agent_id

Mint with the ElevenLabs SDK (

npm install @elevenlabs/elevenlabs-js

pip install elevenlabs

), which sends the

xi-api-key

header for you:

typescript

import { ElevenLabsClient } from "@elevenlabs/elevenlabs-js";

const elevenlabs = new ElevenLabsClient();

const clientSecret =
  await elevenlabs.conversationalAi.realtime.createClientSecret({
    session: {
      type: "realtime",
      model: "eleven_realtime_v1_mini",
      instructions:
        "You are a helpful, friendly voice assistant. Keep replies concise.",
      outputModalities: ["audio"],
      audio: {
        input: {
          format: { type: "audio/pcm", rate: 24000 },
          turnDetection: { type: "semantic_vad", eagerness: "medium" },
        },
        output: {
          format: { type: "audio/pcm" },
          voice: "brian",
        },
      },
    },
    expiresAfter: { anchor: "created_at", seconds: 600 },
  });

python

import os

from elevenlabs import (
    CreateClientSecretRequest,
    ExpiresAfter,
    OpenAiAudioConfigInput,
    OpenAiAudioInputConfigInput,
    OpenAiAudioInputFormat,
    OpenAiAudioOutputConfigInput,
    OpenAiAudioOutputFormat,
    OpenAiSessionConfigInput,
    OpenAiTurnDetection,
)
from elevenlabs.client import ElevenLabs

elevenlabs = ElevenLabs(api_key=os.environ["ELEVENLABS_API_KEY"])

client_secret = elevenlabs.conversational_ai.realtime.create_client_secret(
    request=CreateClientSecretRequest(
        session=OpenAiSessionConfigInput(
            type="realtime",
            model="eleven_realtime_v1_mini",
            instructions="You are a helpful, friendly voice assistant. Keep replies concise.",
            output_modalities=["audio"],
            audio=OpenAiAudioConfigInput(
                input=OpenAiAudioInputConfigInput(
                    format=OpenAiAudioInputFormat(type="audio/pcm", rate=24000),
                    turn_detection=OpenAiTurnDetection(type="semantic_vad", eagerness="medium"),
                ),
                output=OpenAiAudioOutputConfigInput(
                    format=OpenAiAudioOutputFormat(type="audio/pcm"),
                    voice="brian",
                ),
            ),
        ),
        expires_after=ExpiresAfter(anchor="created_at", seconds=600),
    ),
)

Return only

clientSecret.value

(

client_secret.value

in Python) to the browser.

Keep the route's existing framework, path, and response contract; only the upstream call changes. If the app minted with the beta

/v1/realtime/sessions

endpoint, its frontend reads

client_secret.value

. Either update the frontend to read the top-level

value

, or have the route return

{ client_secret: { value } }

so the frontend stays untouched.

Step 2: Convert the session config to the GA shape

Send the GA session shape. Apps written against OpenAI's beta API use flat fields (

modalities

voice

input_audio_format

turn_detection

, …) that need converting, and ElevenLabs rejects the deprecated

modalities

field outright. This matters most for the mint request body and any hand-rolled

session.update

: the SDK's

config

option converts deprecated camelCase fields to the GA wire shape for you, though moving it to the GA form too removes the deprecation warnings.

Read Session config mapping for the full beta-to-GA field table, voice handling, and the camelCase form the SDK's

config

option uses.

Step 3: Point the browser session at ElevenLabs

Replace the transport with

OpenAIRealtimeWebSocket

aimed at the ElevenLabs URL, set the model on the session, and pass the minted

value

as the API key:

typescript

import {
  OpenAIRealtimeWebSocket,
  RealtimeAgent,
  RealtimeSession,
} from "@openai/agents-realtime";

const { value: clientSecret } = await fetch("/api/token").then((res) =>
  res.json(),
);

const agent = new RealtimeAgent({
  name: "Assistant",
  instructions:
    "You are a helpful, friendly voice assistant. Keep replies concise.",
});

const transport = new OpenAIRealtimeWebSocket({
  url: "wss://api.elevenlabs.io/v1/convai/realtime",
});

const session = new RealtimeSession(agent, {
  transport,
  model: "eleven_realtime_v1_mini",
  config: {
    outputModalities: ["audio"],
    audio: { output: { voice: "brian" } },
  },
});

await session.connect({ apiKey: clientSecret });

Keep the app's existing

RealtimeAgent

name, instructions, and event handlers. Drop WebRTC-only options (

audioElement

mediaStream

changePeerConnection

) and any

transport: "webrtc"

setting; they have no WebSocket equivalent.

Step 4: Wire up audio (when the app was on WebRTC)

If the app previously used WebRTC (an explicit

OpenAIRealtimeWebRTC

transport: "webrtc"

, or no transport at all in a browser), it now has to:

Capture microphone audio as PCM16 at 24 kHz and stream it with
```
session.sendAudio(...)
```
.
Play PCM16 chunks from the session's
```
audio
```
event.
Stop playback on
```
audio_interrupted
```
. Without this, the assistant keeps talking over the user after a barge-in, because already-buffered audio keeps playing.
Release the microphone and close the session on teardown.

Read Browser audio over WebSocket for a working implementation with

wavtools

, the helper OpenAI uses in its own WebSocket examples, and its pitfalls.

Apps already on

OpenAIRealtimeWebSocket

have this wiring; check the sample rate is 24 kHz and keep it.

Step 5: Verify

Find where the server loads
```
OPENAI_API_KEY
```
from (
```
.env
```
,
```
.env.local
```
, deployment secrets) and ask the user to store
```
ELEVENLABS_API_KEY
```
there. Don't ask for the key in the conversation or write it into a file yourself. Add an empty
```
ELEVENLABS_API_KEY=
```
to
```
.env.example
```
if the project has one.
Once the user confirms the key is in place, run the app, start a conversation, and speak.

Confirm the

session.created

transport event reports the

eleven_realtime_*

model and the chosen voice:

typescript

session.on("transport_event", (event) => {
  if (event.type === "session.created") console.log(event);
});

Interrupt the assistant mid-sentence and confirm playback stops.
Tell the user which features from Step 0 remain unverified (tools, handoffs, guardrails, …).

Troubleshooting

Symptom	Cause and fix
WebSocket closes with code `4003`	Realtime isn't enabled for this ElevenLabs workspace. Contact ElevenLabs; the code can't fix this.
WebSocket closes with code `4001`	Client secret rejected: expired, or not the minted `value` passed as `apiKey` to `session.connect()` .
Mint returns 401	Still sending `Authorization: Bearer` ; ElevenLabs uses the `xi-api-key` header.
Mint or `session.update` is rejected	Beta-shaped fields such as `modalities` . Convert with the mapping.
Connects, but no audio either way	The app was on WebRTC and audio isn't wired yet. See Step 4.
Audio is noise, chipmunk-fast, or slowed down	Sample-rate or buffer mismatch. Capture and play at 24 kHz; see the pitfalls in the audio reference.
Assistant talks over the user after an interruption	Playback isn't stopped on `audio_interrupted` .

References

Session config mapping
Browser audio over WebSocket

realtime

NPX Install

Tags

SKILL.md Content

ElevenLabs Realtime

Models

Run an agent on a Realtime model

Before you start

Step 1: Check the agent can use Realtime

Step 2: Choose the model

Step 3: Set the model

Managed settings

Step 4: Publish and verify

Errors

Pricing

Migrate an app from OpenAI Realtime

Step 0: Inventory the app

Step 1: Mint the client secret from ElevenLabs

Step 2: Convert the session config to the GA shape

Step 3: Point the browser session at ElevenLabs

Step 4: Wire up audio (when the app was on WebRTC)

Step 5: Verify

Troubleshooting

References