elevenlabs-agents

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

ElevenLabs Agents Platform

ElevenLabs Agents 平台

Overview

概述

ElevenLabs Agents Platform is a comprehensive solution for building production-ready conversational AI voice agents. The platform coordinates four core components:
  1. ASR (Automatic Speech Recognition) - Converts speech to text (32+ languages, sub-second latency)
  2. LLM (Large Language Model) - Reasoning and response generation (GPT, Claude, Gemini, custom models)
  3. TTS (Text-to-Speech) - Converts text to speech (5000+ voices, 31 languages, low latency)
  4. Turn-Taking Model - Proprietary model that handles conversation timing and interruptions
ElevenLabs Agents 平台是一个用于构建可投入生产环境的对话式AI语音代理的综合解决方案。该平台协调四个核心组件:
  1. ASR(自动语音识别) - 将语音转换为文本(支持32+种语言,亚秒级延迟)
  2. LLM(大语言模型) - 推理与响应生成(支持GPT、Claude、Gemini及自定义模型)
  3. TTS(文本转语音) - 将文本转换为语音(支持5000+种语音,31种语言,低延迟)
  4. 话轮转换模型 - 处理对话时序与打断的专有模型

🚨 Package Updates (November 2025)

🚨 包更新(2025年11月)

ElevenLabs migrated to new scoped packages in August 2025:
DEPRECATED (Do not use):
  • @11labs/react
    DEPRECATED
  • @11labs/client
    DEPRECATED
Current packages:
bash
npm install @elevenlabs/react@0.9.1        # React SDK
npm install @elevenlabs/client@0.9.1       # JavaScript SDK
npm install @elevenlabs/react-native@0.5.2 # React Native SDK
npm install @elevenlabs/elevenlabs-js@2.21.0 # Base SDK
npm install -g @elevenlabs/agents-cli@0.2.0  # CLI
If you have old packages installed, uninstall them first:
bash
npm uninstall @11labs/react @11labs/client
ElevenLabs在2025年8月迁移到了新的作用域包:
已弃用(请勿使用):
  • @11labs/react
    已弃用
  • @11labs/client
    已弃用
当前可用包:
bash
npm install @elevenlabs/react@0.9.1        # React SDK
npm install @elevenlabs/client@0.9.1       # JavaScript SDK
npm install @elevenlabs/react-native@0.5.2 # React Native SDK
npm install @elevenlabs/elevenlabs-js@2.21.0 # 基础SDK
npm install -g @elevenlabs/agents-cli@0.2.0  # CLI
如果已安装旧包,请先卸载:
bash
npm uninstall @11labs/react @11labs/client

When to Use This Skill

何时使用本技能

Use this skill when:
  • Building voice-enabled customer support agents
  • Creating interactive voice response (IVR) systems
  • Developing conversational AI applications
  • Integrating telephony (Twilio, SIP trunking)
  • Implementing voice chat in web/mobile apps
  • Configuring agents via CLI ("agents as code")
  • Setting up RAG/knowledge bases for agents
  • Integrating MCP (Model Context Protocol) servers
  • Building HIPAA/GDPR-compliant voice systems
  • Optimizing LLM costs with caching strategies
在以下场景中使用本技能:
  • 构建支持语音的客户支持代理
  • 创建交互式语音应答(IVR)系统
  • 开发对话式AI应用
  • 集成电话系统(Twilio、SIP中继)
  • 在Web/移动应用中实现语音聊天
  • 通过CLI配置代理(“代理即代码”)
  • 为代理设置RAG/知识库
  • 集成MCP(模型上下文协议)服务器
  • 构建符合HIPAA/GDPR规范的语音系统
  • 通过缓存策略优化LLM成本

Platform Capabilities

平台功能

Design & Configure:
  • Multi-step workflows with visual builder
  • System prompt engineering (6-component framework)
  • 5000+ voices across 31 languages
  • Pronunciation dictionaries (IPA/CMU formats)
  • Speed control (0.7x-1.2x)
  • RAG-powered knowledge bases
  • Dynamic variables and personalization
Connect & Deploy:
  • React SDK (
    @elevenlabs/react
    )
  • JavaScript SDK (
    @elevenlabs/client
    )
  • React Native SDK (
    @elevenlabs/react-native
    )
  • Swift SDK (iOS/macOS)
  • Embeddable widget
  • Telephony integration (Twilio, SIP)
  • Scribe (Real-Time Speech-to-Text) - Beta
Operate & Optimize:
  • Automated testing (scenario, tool call, load)
  • Conversation analysis and evaluation
  • Analytics dashboard (resolution rates, sentiment, compliance)
  • Privacy controls (GDPR, HIPAA, SOC 2)
  • Cost optimization (LLM caching, model swapping, burst pricing)
  • CLI for "agents as code" workflow

设计与配置:
  • 带可视化构建器的多步骤工作流
  • 系统提示工程(6组件框架)
  • 覆盖31种语言的5000+种语音
  • 发音词典(IPA/CMU格式)
  • 语速控制(0.7x-1.2x)
  • 基于RAG的知识库
  • 动态变量与个性化设置
连接与部署:
  • React SDK (
    @elevenlabs/react
    )
  • JavaScript SDK (
    @elevenlabs/client
    )
  • React Native SDK (
    @elevenlabs/react-native
    )
  • Swift SDK(iOS/macOS)
  • 可嵌入小部件
  • 电话系统集成(Twilio、SIP)
  • Scribe(实时语音转文本)- Beta版
运营与优化:
  • 自动化测试(场景测试、工具调用测试、负载测试)
  • 对话分析与评估
  • 分析仪表盘(解决率、情感分析、合规性)
  • 隐私控制(GDPR、HIPAA、SOC 2)
  • 成本优化(LLM缓存、模型切换、峰值定价)
  • 用于“代理即代码”工作流的CLI

1. Quick Start (3 Integration Paths)

1. 快速入门(3种集成路径)

Path A: React SDK (Embedded Voice Chat)

路径A:React SDK(嵌入式语音聊天)

For building voice chat interfaces in React applications.
Installation:
bash
npm install @elevenlabs/react zod
Basic Example:
typescript
import { useConversation } from '@elevenlabs/react';
import { z } from 'zod';

export default function VoiceChat() {
  const { startConversation, stopConversation, status } = useConversation({
    // Public agent (no API key needed)
    agentId: 'your-agent-id',

    // OR private agent (requires API key)
    apiKey: process.env.NEXT_PUBLIC_ELEVENLABS_API_KEY,

    // OR signed URL (server-generated, most secure)
    signedUrl: '/api/elevenlabs/auth',

    // Client-side tools (browser functions)
    clientTools: {
      updateCart: {
        description: "Update the shopping cart",
        parameters: z.object({
          item: z.string(),
          quantity: z.number()
        }),
        handler: async ({ item, quantity }) => {
          console.log('Updating cart:', item, quantity);
          return { success: true };
        }
      }
    },

    // Event handlers
    onConnect: () => console.log('Connected'),
    onDisconnect: () => console.log('Disconnected'),
    onEvent: (event) => {
      switch (event.type) {
        case 'transcript':
          console.log('User said:', event.data.text);
          break;
        case 'agent_response':
          console.log('Agent replied:', event.data.text);
          break;
      }
    },

    // Regional compliance (GDPR, data residency)
    serverLocation: 'us' // 'us' | 'global' | 'eu-residency' | 'in-residency'
  });

  return (
    <div>
      <button onClick={startConversation}>Start Conversation</button>
      <button onClick={stopConversation}>Stop</button>
      <p>Status: {status}</p>
    </div>
  );
}
用于在React应用中构建语音聊天界面。
安装:
bash
npm install @elevenlabs/react zod
基础示例:
typescript
import { useConversation } from '@elevenlabs/react';
import { z } from 'zod';

export default function VoiceChat() {
  const { startConversation, stopConversation, status } = useConversation({
    // 公共代理(无需API密钥)
    agentId: 'your-agent-id',

    // 或私有代理(需要API密钥)
    apiKey: process.env.NEXT_PUBLIC_ELEVENLABS_API_KEY,

    // 或签名URL(服务器生成,最安全)
    signedUrl: '/api/elevenlabs/auth',

    // 客户端工具(浏览器函数)
    clientTools: {
      updateCart: {
        description: "更新购物车",
        parameters: z.object({
          item: z.string(),
          quantity: z.number()
        }),
        handler: async ({ item, quantity }) => {
          console.log('正在更新购物车:', item, quantity);
          return { success: true };
        }
      }
    },

    // 事件处理器
    onConnect: () => console.log('已连接'),
    onDisconnect: () => console.log('已断开连接'),
    onEvent: (event) => {
      switch (event.type) {
        case 'transcript':
          console.log('用户说:', event.data.text);
          break;
        case 'agent_response':
          console.log('代理回复:', event.data.text);
          break;
      }
    },

    // 区域合规(GDPR、数据驻留)
    serverLocation: 'us' // 'us' | 'global' | 'eu-residency' | 'in-residency'
  });

  return (
    <div>
      <button onClick={startConversation}>开始对话</button>
      <button onClick={stopConversation}>停止</button>
      <p>状态: {status}</p>
    </div>
  );
}

Path B: CLI ("Agents as Code")

路径B:CLI(“代理即代码”)

For managing agents via code with version control and CI/CD.
Installation:
bash
npm install -g @elevenlabs/agents-cli
用于通过代码管理代理,支持版本控制与CI/CD。
安装:
bash
npm install -g @elevenlabs/agents-cli

or

pnpm install -g @elevenlabs/agents-cli

**Workflow**:
```bash
pnpm install -g @elevenlabs/agents-cli

**工作流**:
```bash

1. Authenticate

1. 认证

elevenlabs auth login
elevenlabs auth login

2. Initialize project (creates agents.json, tools.json, tests.json)

2. 初始化项目(创建agents.json、tools.json、tests.json)

elevenlabs agents init
elevenlabs agents init

3. Create agent from template

3. 从模板创建代理

elevenlabs agents add "Support Agent" --template customer-service
elevenlabs agents add "Support Agent" --template customer-service

4. Configure in agent_configs/support-agent.json

4. 在agent_configs/support-agent.json中配置

5. Push to platform

5. 推送到平台

elevenlabs agents push --env dev
elevenlabs agents push --env dev

6. Test

6. 测试

elevenlabs agents test "Support Agent"
elevenlabs agents test "Support Agent"

7. Deploy to production

7. 部署到生产环境

elevenlabs agents push --env prod

**Project Structure Created**:
your_project/ ├── agents.json # Agent registry ├── tools.json # Tool configurations ├── tests.json # Test configurations ├── agent_configs/ # Individual agent files ├── tool_configs/ # Tool configuration files └── test_configs/ # Test configuration files
undefined
elevenlabs agents push --env prod

**创建的项目结构**:
your_project/ ├── agents.json # 代理注册表 ├── tools.json # 工具配置 ├── tests.json # 测试配置 ├── agent_configs/ # 单个代理文件 ├── tool_configs/ # 工具配置文件 └── test_configs/ # 测试配置文件
undefined

Path C: API (Programmatic Agent Management)

路径C:API(程序化代理管理)

For creating agents dynamically (multi-tenant, SaaS platforms).
Installation:
bash
npm install elevenlabs
Example:
typescript
import { ElevenLabsClient } from 'elevenlabs';

const client = new ElevenLabsClient({
  apiKey: process.env.ELEVENLABS_API_KEY
});

// Create agent
const agent = await client.agents.create({
  name: 'Support Bot',
  conversation_config: {
    agent: {
      prompt: {
        prompt: "You are a helpful customer support agent.",
        llm: "gpt-4o",
        temperature: 0.7
      },
      first_message: "Hello! How can I help you today?",
      language: "en"
    },
    tts: {
      model_id: "eleven_turbo_v2_5",
      voice_id: "your-voice-id"
    }
  }
});

console.log('Agent created:', agent.agent_id);

用于动态创建代理(多租户、SaaS平台)。
安装:
bash
npm install elevenlabs
示例:
typescript
import { ElevenLabsClient } from 'elevenlabs';

const client = new ElevenLabsClient({
  apiKey: process.env.ELEVENLABS_API_KEY
});

// 创建代理
const agent = await client.agents.create({
  name: 'Support Bot',
  conversation_config: {
    agent: {
      prompt: {
        prompt: "你是一个乐于助人的客户支持代理。",
        llm: "gpt-4o",
        temperature: 0.7
      },
      first_message: "你好!今天我能帮你做什么?",
      language: "en"
    },
    tts: {
      model_id: "eleven_turbo_v2_5",
      voice_id: "your-voice-id"
    }
  }
});

console.log('代理已创建:', agent.agent_id);

2. Agent Configuration

2. 代理配置

System Prompt Architecture (6 Components)

系统提示架构(6组件)

ElevenLabs recommends structuring agent prompts using 6 components:
ElevenLabs推荐使用6组件结构来构建代理提示词:

1. Personality

1. 个性

Define the agent's identity, role, and character traits.
Example:
You are Alex, a friendly and knowledgeable customer support specialist at TechCorp.
You have 5 years of experience helping customers solve technical issues.
You're patient, empathetic, and always maintain a positive attitude.
定义代理的身份、角色和性格特征。
示例:
你是Alex,TechCorp的一位友好且知识渊博的客户支持专员。
你有5年帮助客户解决技术问题的经验。
你耐心、善解人意,始终保持积极的态度。

2. Environment

2. 环境

Describe the communication context (phone, web chat, video call).
Example:
You're speaking with customers over the phone. Communication is voice-only.
Customers may have background noise or poor connection quality.
Speak clearly and occasionally use thoughtful pauses for emphasis.
描述沟通场景(电话、网页聊天、视频通话)。
示例:
你正在通过电话与客户沟通。沟通仅通过语音进行。
客户可能有背景噪音或网络连接质量不佳。
说话要清晰,偶尔用深思熟虑的停顿来强调重点。

3. Tone

3. 语气

Specify formality, speech patterns, humor, and verbosity.
Example:
Tone: Professional yet warm. Use contractions ("I'm" instead of "I am") to sound natural.
Avoid jargon unless the customer uses it first. Keep responses concise (2-3 sentences max).
Use encouraging phrases like "I'll be happy to help with that" and "Let's get this sorted for you."
指定正式程度、说话方式、幽默感和 verbose程度。
示例:
语气:专业且热情。使用缩写(用“I'm”代替“I am”)让语气更自然。
除非客户先使用行话,否则避免使用行话。保持回复简洁(最多2-3句话)。
使用鼓励性短语,如“我很乐意帮你解决这个问题”和“让我们一起把这个问题解决好”。

4. Goal

4. 目标

Define objectives and success criteria.
Example:
Primary Goal: Resolve customer technical issues on the first call.
Secondary Goals:
- Verify customer identity securely
- Document issue details accurately
- Offer proactive solutions
- End calls with confirmation that the issue is resolved

Success Criteria: Customer verbally confirms their issue is resolved.
定义目标与成功标准。
示例:
主要目标:在首次通话中解决客户的技术问题。
次要目标:
- 安全验证客户身份
- 准确记录问题细节
- 提供主动解决方案
- 通话结束时确认问题已解决

成功标准:客户口头确认问题已解决。

5. Guardrails

5. 防护规则

Set boundaries, prohibited topics, and ethical constraints.
Example:
Guardrails:
- Never provide medical, legal, or financial advice
- Do not share confidential company information
- If asked about competitors, politely redirect to TechCorp's offerings
- Escalate to a human supervisor if customer becomes abusive
- Never make promises about refunds or credits without verification
设置边界、禁止话题和道德约束。
示例:
防护规则:
- 绝不提供医疗、法律或财务建议
- 不分享公司机密信息
- 如果被问及竞争对手,礼貌地将话题转回TechCorp的产品
- 如果客户变得粗鲁无礼,升级给人工主管
- 未经核实,绝不做出退款或信用额度的承诺

6. Tools

6. 工具

Describe available external capabilities and when to use them.
Example:
Available Tools:
1. lookup_order(order_id) - Fetch order details from database. Use when customer mentions an order number.
2. transfer_to_supervisor() - Escalate to human agent. Use when issue requires manager approval.
3. send_password_reset(email) - Trigger password reset email. Use when customer can't access account.

Always explain to the customer what you're doing before calling a tool.
Complete Template:
json
{
  "agent": {
    "prompt": {
      "prompt": "Personality:\nYou are Alex, a friendly customer support specialist.\n\nEnvironment:\nYou're speaking with customers over the phone.\n\nTone:\nProfessional yet warm. Keep responses concise.\n\nGoal:\nResolve technical issues on the first call.\n\nGuardrails:\n- Never provide medical/legal/financial advice\n- Escalate abusive customers\n\nTools:\n- lookup_order(order_id) - Fetch order details\n- transfer_to_supervisor() - Escalate to human",
      "llm": "gpt-4o",
      "temperature": 0.7,
      "max_tokens": 500
    }
  }
}
描述可用的外部功能以及何时使用它们。
示例:
可用工具:
1. lookup_order(order_id) - 从数据库获取订单详情。当客户提到订单号时使用。
2. transfer_to_supervisor() - 升级给人工代理。当问题需要经理批准时使用。
3. send_password_reset(email) - 触发密码重置邮件。当客户无法访问账户时使用。

在调用工具之前,务必向客户解释你要做什么。
完整模板:
json
{
  "agent": {
    "prompt": {
      "prompt": "个性:\n你是Alex,一位友好的客户支持专员。\n\n环境:\n你正在通过电话与客户沟通。\n\n语气:\n专业且热情。保持回复简洁。\n\n目标:\n在首次通话中解决技术问题。\n\n防护规则:\n- 绝不提供医疗/法律/财务建议\n- 将粗鲁无礼的客户升级给主管\n\n工具:\n- lookup_order(order_id) - 获取订单详情\n- transfer_to_supervisor() - 升级给人工代理",
      "llm": "gpt-4o",
      "temperature": 0.7,
      "max_tokens": 500
    }
  }
}

Turn-Taking Modes

话轮转换模式

Controls when the agent interrupts or waits for the user to finish speaking.
3 Modes:
ModeBehaviorBest For
EagerResponds quickly, jumps in at earliest opportunityFast-paced support, quick orders
NormalBalanced, waits for natural conversation breaksGeneral customer service (default)
PatientWaits longer, allows detailed user responsesInformation collection, therapy, tutoring
Configuration:
json
{
  "conversation_config": {
    "turn": {
      "mode": "patient" // "eager" | "normal" | "patient"
    }
  }
}
Use Cases:
  • Eager: Fast food ordering, quick FAQs, urgent notifications
  • Normal: General support, product inquiries, appointment booking
  • Patient: Detailed form filling, emotional support, educational tutoring
Gotchas:
  • Eager mode can feel interruptive to some users
  • Patient mode may feel slow in fast-paced contexts
  • Can be dynamically adjusted in workflows for context-aware behavior
控制代理何时打断用户或等待用户说完。
3种模式:
模式行为最佳适用场景
主动快速响应,尽早插话快节奏支持、快速下单
正常平衡模式,等待自然对话停顿通用客户服务(默认)
耐心等待更长时间,允许用户详细回复信息收集、心理咨询、辅导
配置:
json
{
  "conversation_config": {
    "turn": {
      "mode": "patient" // "eager" | "normal" | "patient"
    }
  }
}
使用场景:
  • 主动:快餐点餐、快速FAQ、紧急通知
  • 正常:通用支持、产品咨询、预约预订
  • 耐心:详细表单填写、情感支持、教育辅导
注意事项:
  • 主动模式可能会让部分用户感到被打断
  • 耐心模式在快节奏场景中可能显得缓慢
  • 可以在工作流中动态调整,实现上下文感知的行为

Workflows (Visual Builder)

工作流(可视化构建器)

Create branching conversation flows with subagent nodes and conditional routing.
Node Types:
  1. Subagent Nodes - Override base agent config (change prompt, voice, turn-taking)
  2. Tool Nodes - Guarantee tool execution (unlike tools in subagents)
Configuration:
json
{
  "workflow": {
    "nodes": [
      {
        "id": "node_1",
        "type": "subagent",
        "config": {
          "system_prompt": "You are now a technical support specialist. Ask detailed diagnostic questions.",
          "turn_eagerness": "patient",
          "voice_id": "tech_support_voice_id"
        }
      },
      {
        "id": "node_2",
        "type": "tool",
        "tool_name": "transfer_to_human"
      }
    ],
    "edges": [
      {
        "from": "node_1",
        "to": "node_2",
        "condition": "user_requests_escalation"
      }
    ]
  }
}
Use Cases:
  • Multi-department routing (sales → support → billing)
  • Decision trees ("press 1 for sales, 2 for support")
  • Role-playing scenarios (customer vs agent voices)
  • Escalation paths (bot → human transfer)
Gotchas:
  • Workflows add ~100-200ms latency per node transition
  • Tool nodes guarantee execution (subagents may skip tools)
  • Edges can create infinite loops if not tested properly
创建包含子代理节点和条件路由的分支对话流程。
节点类型:
  1. 子代理节点 - 覆盖基础代理配置(修改提示词、语音、话轮转换)
  2. 工具节点 - 确保工具执行(与子代理中的工具不同)
配置:
json
{
  "workflow": {
    "nodes": [
      {
        "id": "node_1",
        "type": "subagent",
        "config": {
          "system_prompt": "你现在是一位技术支持专员。询问详细的诊断问题。",
          "turn_eagerness": "patient",
          "voice_id": "tech_support_voice_id"
        }
      },
      {
        "id": "node_2",
        "type": "tool",
        "tool_name": "transfer_to_human"
      }
    ],
    "edges": [
      {
        "from": "node_1",
        "to": "node_2",
        "condition": "user_requests_escalation"
      }
    ]
  }
}
使用场景:
  • 多部门路由(销售 → 支持 → 计费)
  • 决策树(“按1选择销售,按2选择支持”)
  • 角色扮演场景(客户与代理语音)
  • 升级路径(机器人 → 人工转接)
注意事项:
  • 工作流在每次节点转换时会增加约100-200ms延迟
  • 工具节点确保执行(子代理可能会跳过工具)
  • 如果未测试,边可能会创建无限循环

Dynamic Variables & Personalization

动态变量与个性化

Inject runtime data into prompts, first messages, and tool parameters using
{{var_name}}
syntax.
System Variables (Auto-Available):
typescript
{{system__agent_id}}         // Current agent ID
{{system__conversation_id}}  // Conversation ID
{{system__caller_id}}        // Phone number (telephony only)
{{system__called_number}}    // Called number (telephony only)
{{system__call_duration_secs}} // Call duration
{{system__time_utc}}         // Current UTC time
{{system__call_sid}}         // Twilio call SID (Twilio only)
Custom Variables:
typescript
// Provide when starting conversation
const conversation = await client.conversations.create({
  agent_id: "agent_123",
  dynamic_variables: {
    user_name: "John",
    account_tier: "premium",
    order_id: "ORD-12345"
  }
});
Secret Variables (For API Keys):
{{secret__stripe_api_key}}
{{secret__database_password}}
Important: Secret variables only used in headers, never sent to LLM providers.
Usage in Prompts:
json
{
  "agent": {
    "prompt": {
      "prompt": "You are helping {{user_name}}, a {{account_tier}} customer."
    },
    "first_message": "Hello {{user_name}}! I see you're calling about order {{order_id}}."
  }
}
Gotcha: Missing variables cause "Missing required dynamic variables" error. Always provide all referenced variables when starting conversation.
使用
{{var_name}}
语法将运行时数据注入提示词、第一条消息和工具参数。
系统变量(自动可用):
typescript
{{system__agent_id}}         // 当前代理ID
{{system__conversation_id}}  // 对话ID
{{system__caller_id}}        // 电话号码(仅电话系统)
{{system__called_number}}    // 被叫号码(仅电话系统)
{{system__call_duration_secs}} // 通话时长
{{system__time_utc}}         // 当前UTC时间
{{system__call_sid}}         // Twilio通话SID(仅Twilio)
自定义变量:
typescript
// 启动对话时提供
const conversation = await client.conversations.create({
  agent_id: "agent_123",
  dynamic_variables: {
    user_name: "John",
    account_tier: "premium",
    order_id: "ORD-12345"
  }
});
机密变量(用于API密钥):
{{secret__stripe_api_key}}
{{secret__database_password}}
重要提示: 机密变量仅用于请求头,绝不会发送给LLM提供商。
在提示词中使用:
json
{
  "agent": {
    "prompt": {
      "prompt": "你正在帮助{{user_name}},一位{{account_tier}}客户。"
    },
    "first_message": "你好{{user_name}}!我看到你正在咨询订单{{order_id}}的问题。"
  }
}
注意事项: 缺少变量会导致“缺少必需的动态变量”错误。启动对话时务必提供所有引用的变量。

Authentication Patterns

认证模式

Option 1: Public Agents (No API Key)
typescript
const { startConversation } = useConversation({
  agentId: 'your-public-agent-id' // Anyone can use
});
Option 2: Private Agents with API Key
typescript
const { startConversation } = useConversation({
  agentId: 'your-private-agent-id',
  apiKey: process.env.NEXT_PUBLIC_ELEVENLABS_API_KEY
});
⚠️ Warning: Never expose API keys in client-side code. Use signed URLs instead.
Option 3: Signed URLs (Recommended for Production)
typescript
// Server-side (Next.js API route)
import { ElevenLabsClient } from 'elevenlabs';

export async function POST(req: Request) {
  const client = new ElevenLabsClient({
    apiKey: process.env.ELEVENLABS_API_KEY // Server-side only
  });

  const signedUrl = await client.convai.getSignedUrl({
    agent_id: 'your-agent-id'
  });

  return Response.json({ signedUrl });
}

// Client-side
const { startConversation } = useConversation({
  agentId: 'your-agent-id',
  signedUrl: await fetch('/api/elevenlabs/auth').then(r => r.json()).then(d => d.signedUrl)
});

选项1:公共代理(无需API密钥)
typescript
const { startConversation } = useConversation({
  agentId: 'your-public-agent-id' // 任何人都可以使用
});
选项2:带API密钥的私有代理
typescript
const { startConversation } = useConversation({
  agentId: 'your-private-agent-id',
  apiKey: process.env.NEXT_PUBLIC_ELEVENLABS_API_KEY
});
⚠️ 警告: 绝不要在客户端代码中暴露API密钥。请改用签名URL。
选项3:签名URL(生产环境推荐)
typescript
// 服务器端(Next.js API路由)
import { ElevenLabsClient } from 'elevenlabs';

export async function POST(req: Request) {
  const client = new ElevenLabsClient({
    apiKey: process.env.ELEVENLABS_API_KEY // 仅在服务器端使用
  });

  const signedUrl = await client.convai.getSignedUrl({
    agent_id: 'your-agent-id'
  });

  return Response.json({ signedUrl });
}

// 客户端
const { startConversation } = useConversation({
  agentId: 'your-agent-id',
  signedUrl: await fetch('/api/elevenlabs/auth').then(r => r.json()).then(d => d.signedUrl)
});

3. Voice & Language Features

3. 语音与语言功能

Multi-Voice Support

多语音支持

Dynamically switch between different voices during a single conversation.
Use Cases:
  • Multi-character storytelling (different voice per character)
  • Language tutoring (native speaker voices for each language)
  • Role-playing scenarios (customer vs agent)
  • Emotional agents (different voices for different moods)
Configuration:
json
{
  "agent": {
    "prompt": {
      "prompt": "When speaking as the customer, use voice_id 'customer_voice_abc123'. When speaking as the agent, use voice_id 'agent_voice_def456'."
    }
  }
}
Gotchas:
  • Voice switching adds ~200ms latency per switch
  • Requires careful prompt engineering to trigger switches correctly
  • Not all voices work equally well for all characters
在单次对话中动态切换不同语音。
使用场景:
  • 多角色叙事(每个角色使用不同语音)
  • 语言辅导(每种语言使用母语者语音)
  • 角色扮演场景(客户与代理)
  • 情感代理(不同情绪使用不同语音)
配置:
json
{
  "agent": {
    "prompt": {
      "prompt": "当以客户身份说话时,使用voice_id 'customer_voice_abc123'。当以代理身份说话时,使用voice_id 'agent_voice_def456'。"
    }
  }
}
注意事项:
  • 语音切换每次会增加约200ms延迟
  • 需要精心设计提示词以正确触发切换
  • 并非所有语音都能很好地适配所有角色

Pronunciation Dictionary

发音词典

Customize how the agent pronounces specific words or phrases.
Supported Formats:
  • IPA (International Phonetic Alphabet)
  • CMU (Carnegie Mellon University Pronouncing Dictionary)
  • Word Substitutions (replace words before TTS)
Configuration:
json
{
  "pronunciation_dictionary": [
    {
      "word": "ElevenLabs",
      "pronunciation": "ɪˈlɛvənlæbz",
      "format": "ipa"
    },
    {
      "word": "API",
      "pronunciation": "ey-pee-ay",
      "format": "cmu"
    },
    {
      "word": "AI",
      "substitution": "artificial intelligence"
    }
  ]
}
Use Cases:
  • Brand names (e.g., "IKEA" → "ee-KAY-uh")
  • Acronyms (e.g., "API" → "A-P-I" or "ay-pee-eye")
  • Technical terms
  • Character names in storytelling
Gotcha: Only Turbo v2/v2.5 models support phoneme-based pronunciation. Other models silently skip phoneme entries but still process word substitutions.
自定义代理发音特定单词或短语的方式。
支持格式:
  • IPA(国际音标)
  • CMU(卡内基梅隆大学发音词典)
  • 单词替换(在TTS之前替换单词)
配置:
json
{
  "pronunciation_dictionary": [
    {
      "word": "ElevenLabs",
      "pronunciation": "ɪˈlɛvənlæbz",
      "format": "ipa"
    },
    {
      "word": "API",
      "pronunciation": "ey-pee-ay",
      "format": "cmu"
    },
    {
      "word": "AI",
      "substitution": "artificial intelligence"
    }
  ]
}
使用场景:
  • 品牌名称(例如,"IKEA" → "ee-KAY-uh")
  • 首字母缩写(例如,"API" → "A-P-I"或"ay-pee-eye")
  • 技术术语
  • 叙事中的角色名称
注意事项: 仅Turbo v2/v2.5模型支持基于音素的发音。其他模型会静默跳过音素条目,但仍会处理单词替换。

Speed Control

语速控制

Adjust speaking speed dynamically (0.7x - 1.2x).
Configuration:
json
{
  "voice_settings": {
    "speed": 1.0 // 0.7 = slow, 1.0 = normal, 1.2 = fast
  }
}
Use Cases:
  • Slow (0.7x-0.9x): Accessibility, children, non-native speakers
  • Normal (1.0x): Default for most use cases
  • Fast (1.1x-1.2x): Urgent notifications, power users
Best Practices:
  • Use 0.9x-1.1x for natural-sounding adjustments
  • Extreme values (below 0.7 or above 1.2) degrade quality
  • Speed can be adjusted per agent, not per utterance
动态调整说话速度(0.7x - 1.2x)。
配置:
json
{
  "voice_settings": {
    "speed": 1.0 // 0.7 = 慢, 1.0 = 正常, 1.2 = 快
  }
}
使用场景:
  • 慢(0.7x-0.9x): 无障碍访问、儿童、非母语者
  • 正常(1.0x): 大多数场景的默认设置
  • 快(1.1x-1.2x): 紧急通知、高级用户
最佳实践:
  • 使用0.9x-1.1x以获得自然的调整效果
  • 极端值(低于0.7或高于1.2)会降低质量
  • 速度可按代理调整,不能按话语调整

Voice Design

语音设计

Create custom voices using ElevenLabs Voice Design tool.
Workflow:
  1. Navigate to Voice Library → Create Voice
  2. Use Voice Design (text-to-voice) or Voice Cloning (sample audio)
  3. Test voice with sample text
  4. Save voice to library
  5. Use
    voice_id
    in agent configuration
Voice Cloning Best Practices:
  • Use clean audio samples (no background noise, music, or pops)
  • Maintain consistent microphone distance
  • Avoid extreme volumes (whispering or shouting)
  • 1-2 minutes of audio recommended
Gotcha: Using English-trained voices for non-English languages causes pronunciation issues. Always use language-matched voices.
使用ElevenLabs语音设计工具创建自定义语音。
工作流:
  1. 导航到语音库 → 创建语音
  2. 使用语音设计(文本转语音)或语音克隆(样本音频)
  3. 使用示例文本测试语音
  4. 将语音保存到库中
  5. 在代理配置中使用
    voice_id
语音克隆最佳实践:
  • 使用干净的音频样本(无背景噪音、音乐或爆破音)
  • 保持一致的麦克风距离
  • 避免极端音量(低语或大喊)
  • 推荐1-2分钟的音频
注意事项: 将为英语训练的语音用于非英语语言会导致发音问题。始终使用与语言匹配的语音。

Language Configuration

语言配置

Support for 32+ languages with automatic detection and in-conversation switching.
Configuration:
json
{
  "agent": {
    "language": "en" // ISO 639-1 code
  }
}
Multi-Language Presets (Different Voice Per Language):
json
{
  "conversation_config": {
    "language_presets": [
      {
        "language": "en",
        "voice_id": "en_voice_id",
        "first_message": "Hello! How can I help you today?"
      },
      {
        "language": "es",
        "voice_id": "es_voice_id",
        "first_message": "¡Hola! ¿Cómo puedo ayudarte hoy?"
      },
      {
        "language": "fr",
        "voice_id": "fr_voice_id",
        "first_message": "Bonjour! Comment puis-je vous aider aujourd'hui?"
      }
    ]
  }
}
Automatic Language Detection: Agent detects user's language and switches automatically.
Supported Languages: English, Spanish, French, German, Italian, Portuguese, Dutch, Polish, Arabic, Chinese, Japanese, Korean, Hindi, and 18+ more.

支持32+种语言,支持自动检测与对话中切换。
配置:
json
{
  "agent": {
    "language": "en" // ISO 639-1代码
  }
}
多语言预设(每种语言使用不同语音):
json
{
  "conversation_config": {
    "language_presets": [
      {
        "language": "en",
        "voice_id": "en_voice_id",
        "first_message": "Hello! How can I help you today?"
      },
      {
        "language": "es",
        "voice_id": "es_voice_id",
        "first_message": "¡Hola! ¿Cómo puedo ayudarte hoy?"
      },
      {
        "language": "fr",
        "voice_id": "fr_voice_id",
        "first_message": "Bonjour! Comment puis-je vous aider aujourd'hui?"
      }
    ]
  }
}
自动语言检测: 代理检测用户语言并自动切换。
支持语言: 英语、西班牙语、法语、德语、意大利语、葡萄牙语、荷兰语、波兰语、阿拉伯语、中文、日语、韩语、印地语及18+种其他语言。

4. Knowledge Base & RAG

4. 知识库与RAG

RAG (Retrieval-Augmented Generation)

RAG(检索增强生成)

Enable agents to access large knowledge bases without loading entire documents into context.
How It Works:
  1. Upload documents (PDF, TXT, DOCX) to knowledge base
  2. ElevenLabs automatically computes vector embeddings
  3. During conversation, relevant chunks retrieved based on semantic similarity
  4. LLM uses retrieved context to generate responses
Configuration:
json
{
  "agent": {
    "prompt": {
      "knowledge_base": ["doc_id_1", "doc_id_2"]
    }
  }
}
Upload Documents via API:
typescript
import { ElevenLabsClient } from 'elevenlabs';

const client = new ElevenLabsClient({ apiKey: process.env.ELEVENLABS_API_KEY });

// Upload document
const doc = await client.knowledgeBase.upload({
  file: fs.createReadStream('support_docs.pdf'),
  name: 'Support Documentation'
});

// Compute RAG index
await client.knowledgeBase.computeRagIndex({
  document_id: doc.id,
  embedding_model: 'e5_mistral_7b' // or 'multilingual_e5_large'
});
Retrieval Configuration:
json
{
  "knowledge_base_config": {
    "max_chunks": 5,              // Number of chunks to retrieve
    "vector_distance_threshold": 0.8  // Similarity threshold
  }
}
Use Cases:
  • Product documentation agents
  • Customer support (FAQ, help center)
  • Educational tutors (textbooks, lecture notes)
  • Healthcare assistants (medical guidelines)
Gotchas:
  • RAG adds ~500ms latency per query
  • More chunks = higher cost but better context
  • Higher vector distance = more context but potentially less relevant
  • Documents must be indexed before use (can take minutes for large docs)

使代理能够访问大型知识库,而无需将整个文档加载到上下文中。
工作原理:
  1. 将文档(PDF、TXT、DOCX)上传到知识库
  2. ElevenLabs自动计算向量嵌入
  3. 在对话过程中,基于语义相似性检索相关片段
  4. LLM使用检索到的上下文生成回复
配置:
json
{
  "agent": {
    "prompt": {
      "knowledge_base": ["doc_id_1", "doc_id_2"]
    }
  }
}
通过API上传文档:
typescript
import { ElevenLabsClient } from 'elevenlabs';
import fs from 'fs';

const client = new ElevenLabsClient({ apiKey: process.env.ELEVENLABS_API_KEY });

// 上传文档
const doc = await client.knowledgeBase.upload({
  file: fs.createReadStream('support_docs.pdf'),
  name: 'Support Documentation'
});

// 计算RAG索引
await client.knowledgeBase.computeRagIndex({
  document_id: doc.id,
  embedding_model: 'e5_mistral_7b' // 或 'multilingual_e5_large'
});
检索配置:
json
{
  "knowledge_base_config": {
    "max_chunks": 5,              // 要检索的片段数量
    "vector_distance_threshold": 0.8  // 相似性阈值
  }
}
使用场景:
  • 产品文档代理
  • 客户支持(FAQ、帮助中心)
  • 教育辅导(教科书、课堂笔记)
  • 医疗助理(医疗指南)
注意事项:
  • RAG每次查询会增加约500ms延迟
  • 片段越多 = 成本越高但上下文越好
  • 向量距离越高 = 上下文越多但相关性可能越低
  • 文档在使用前必须建立索引(大型文档可能需要几分钟)

5. Tools (4 Types)

5. 工具(4种类型)

ElevenLabs supports 4 distinct tool types, each with different execution patterns.
ElevenLabs支持4种不同类型的工具,每种工具具有不同的执行模式。

A. Client Tools

A. 客户端工具

Execute operations on the client side (browser or mobile app).
Use Cases:
  • Update UI elements (shopping cart, notifications)
  • Trigger navigation (redirect user to page)
  • Access local storage
  • Control media playback
React Example:
typescript
import { useConversation } from '@elevenlabs/react';
import { z } from 'zod';

const { startConversation } = useConversation({
  clientTools: {
    updateCart: {
      description: "Update the shopping cart with new items",
      parameters: z.object({
        item: z.string().describe("The item name"),
        quantity: z.number().describe("Quantity to add")
      }),
      handler: async ({ item, quantity }) => {
        // Client-side logic
        const cart = getCart();
        cart.add(item, quantity);
        updateUI(cart);
        return { success: true, total: cart.total };
      }
    },
    navigate: {
      description: "Navigate to a different page",
      parameters: z.object({
        url: z.string().describe("The URL to navigate to")
      }),
      handler: async ({ url }) => {
        window.location.href = url;
        return { success: true };
      }
    }
  }
});
Gotchas:
  • Tool names are case-sensitive
  • Must return a value (agent reads the return value)
  • Handler can be async
在客户端(浏览器或移动应用)执行操作。
使用场景:
  • 更新UI元素(购物车、通知)
  • 触发导航(重定向用户到页面)
  • 访问本地存储
  • 控制媒体播放
React示例:
typescript
import { useConversation } from '@elevenlabs/react';
import { z } from 'zod';

const { startConversation } = useConversation({
  clientTools: {
    updateCart: {
      description: "用新商品更新购物车",
      parameters: z.object({
        item: z.string().describe("商品名称"),
        quantity: z.number().describe("要添加的数量")
      }),
      handler: async ({ item, quantity }) => {
        // 客户端逻辑
        const cart = getCart();
        cart.add(item, quantity);
        updateUI(cart);
        return { success: true, total: cart.total };
      }
    },
    navigate: {
      description: "导航到不同页面",
      parameters: z.object({
        url: z.string().describe("要导航到的URL")
      }),
      handler: async ({ url }) => {
        window.location.href = url;
        return { success: true };
      }
    }
  }
});
注意事项:
  • 工具名称区分大小写
  • 必须返回值(代理会读取返回值)
  • 处理器可以是异步的

B. Server Tools (Webhooks)

B. 服务器工具(Webhook)

Make HTTP requests to external APIs from ElevenLabs servers.
Use Cases:
  • Fetch real-time data (weather, stock prices)
  • Update CRM systems (Salesforce, HubSpot)
  • Process payments (Stripe, PayPal)
  • Send emails/SMS (SendGrid, Twilio)
Configuration via CLI:
bash
elevenlabs tools add-webhook "Get Weather" --config-path tool_configs/get-weather.json
tool_configs/get-weather.json:
json
{
  "name": "get_weather",
  "description": "Fetch current weather for a city",
  "url": "https://api.weather.com/v1/current",
  "method": "GET",
  "parameters": {
    "type": "object",
    "properties": {
      "city": {
        "type": "string",
        "description": "The city name (e.g., 'London', 'New York')"
      }
    },
    "required": ["city"]
  },
  "headers": {
    "Authorization": "Bearer {{secret__weather_api_key}}"
  }
}
Dynamic Variables in Tools:
json
{
  "url": "https://api.crm.com/customers/{{user_id}}",
  "headers": {
    "X-API-Key": "{{secret__crm_api_key}}"
  }
}
Gotchas:
  • Secret variables only work in headers (not URL or body)
  • Schema description guides LLM on when to use tool
从ElevenLabs服务器向外部API发送HTTP请求。
使用场景:
  • 获取实时数据(天气、股票价格)
  • 更新CRM系统(Salesforce、HubSpot)
  • 处理支付(Stripe、PayPal)
  • 发送电子邮件/SMS(SendGrid、Twilio)
通过CLI配置:
bash
elevenlabs tools add-webhook "Get Weather" --config-path tool_configs/get-weather.json
tool_configs/get-weather.json:
json
{
  "name": "get_weather",
  "description": "获取城市当前天气",
  "url": "https://api.weather.com/v1/current",
  "method": "GET",
  "parameters": {
    "type": "object",
    "properties": {
      "city": {
        "type": "string",
        "description": "城市名称(例如,'London'、'New York')"
      }
    },
    "required": ["city"]
  },
  "headers": {
    "Authorization": "Bearer {{secret__weather_api_key}}"
  }
}
工具中的动态变量:
json
{
  "url": "https://api.crm.com/customers/{{user_id}}",
  "headers": {
    "X-API-Key": "{{secret__crm_api_key}}"
  }
}
注意事项:
  • 机密变量仅在请求头中有效(不在URL或请求体中)
  • 架构描述指导LLM何时使用工具

C. MCP Tools (Model Context Protocol)

C. MCP工具(模型上下文协议)

Connect to external MCP servers for standardized tool access.
Use Cases:
  • Access databases (PostgreSQL, MongoDB)
  • Query knowledge bases (Pinecone, Weaviate)
  • Integrate with IDEs (VS Code, Cursor)
  • Connect to data sources (Google Drive, Notion)
Configuration:
  1. Navigate to MCP server integrations in dashboard
  2. Click "Add Custom MCP Server"
  3. Configure:
    • Name: Server identifier
    • Server URL: SSE or HTTP endpoint
    • Secret Token: Optional auth header
  4. Test connectivity and discover tools
  5. Add to agents (public or private)
Approval Modes:
  • Always Ask: Maximum security, requires permission per tool call
  • Fine-Grained: Per-tool approval settings
  • No Approval: Auto-execute all tools
Gotchas:
  • Only SSE and HTTP streamable transport supported
  • MCP servers must be publicly accessible or behind auth
  • Not available for Zero Retention Mode
  • Not compatible with HIPAA compliance
Example: Using ElevenLabs MCP Server in Claude Desktop:
json
{
  "mcpServers": {
    "ElevenLabs": {
      "command": "uvx",
      "args": ["elevenlabs-mcp"],
      "env": {
        "ELEVENLABS_API_KEY": "<your-key>",
        "ELEVENLABS_MCP_OUTPUT_MODE": "files"
      }
    }
  }
}
连接到外部MCP服务器以实现标准化工具访问。
使用场景:
  • 访问数据库(PostgreSQL、MongoDB)
  • 查询知识库(Pinecone、Weaviate)
  • 与IDE集成(VS Code、Cursor)
  • 连接到数据源(Google Drive、Notion)
配置:
  1. 在仪表板中导航到MCP服务器集成
  2. 点击“添加自定义MCP服务器”
  3. 配置:
    • 名称: 服务器标识符
    • 服务器URL: SSE或HTTP端点
    • 机密令牌: 可选的认证请求头
  4. 测试连接并发现工具
  5. 添加到代理(公共或私有)
审批模式:
  • 始终询问: 最高安全性,每次工具调用都需要权限
  • 细粒度: 按工具设置审批权限
  • 无需审批: 自动执行所有工具
注意事项:
  • 仅支持SSE和HTTP可流式传输的传输方式
  • MCP服务器必须可公开访问或通过认证
  • 在零保留模式下不可用
  • 与HIPAA合规性不兼容
示例:在Claude Desktop中使用ElevenLabs MCP服务器:
json
{
  "mcpServers": {
    "ElevenLabs": {
      "command": "uvx",
      "args": ["elevenlabs-mcp"],
      "env": {
        "ELEVENLABS_API_KEY": "<your-key>",
        "ELEVENLABS_MCP_OUTPUT_MODE": "files"
      }
    }
  }
}

D. System Tools

D. 系统工具

Modify the internal state of the conversation without external calls.
Use Cases:
  • Update conversation context
  • Switch between workflow nodes
  • Modify agent behavior mid-conversation
  • Track conversation state
Built-in System Tools:
  • end_call
    - End the conversation
  • detect_language
    - Detect user's language
  • transfer_agent
    - Switch to different agent/workflow node
  • transfer_to_number
    - Transfer to external phone number (telephony only)
  • dtmf_playpad
    - Display DTMF keypad (telephony only)
  • voicemail_detection
    - Detect voicemail (telephony only)
Configuration:
json
{
  "system_tools": [
    {
      "name": "update_conversation_state",
      "description": "Update the conversation context with new information",
      "parameters": {
        "key": { "type": "string" },
        "value": { "type": "string" }
      }
    }
  ]
}
Gotchas:
  • System tools don't trigger external APIs
  • Changes are ephemeral (lost after conversation ends)
  • Useful for workflows and state management

修改对话的内部状态,无需外部调用。
使用场景:
  • 更新对话上下文
  • 在工作流节点之间切换
  • 在对话过程中修改代理行为
  • 跟踪对话状态
内置系统工具:
  • end_call
    - 结束对话
  • detect_language
    - 检测用户语言
  • transfer_agent
    - 切换到不同代理/工作流节点
  • transfer_to_number
    - 转接至外部电话号码(仅电话系统)
  • dtmf_playpad
    - 显示DTMF键盘(仅电话系统)
  • voicemail_detection
    - 检测语音信箱(仅电话系统)
配置:
json
{
  "system_tools": [
    {
      "name": "update_conversation_state",
      "description": "用新信息更新对话上下文",
      "parameters": {
        "key": { "type": "string" },
        "value": { "type": "string" }
      }
    }
  ]
}
注意事项:
  • 系统工具不会触发外部API
  • 更改是临时的(对话结束后丢失)
  • 对工作流和状态管理很有用

6. SDK Integration

6. SDK集成

React SDK (
@elevenlabs/react
)

React SDK (
@elevenlabs/react
)

Installation:
bash
npm install @elevenlabs/react zod
Complete Example:
typescript
import { useConversation } from '@elevenlabs/react';
import { z } from 'zod';
import { useState } from 'react';

export default function VoiceAgent() {
  const [transcript, setTranscript] = useState<string[]>([]);

  const {
    startConversation,
    stopConversation,
    status,
    isSpeaking
  } = useConversation({
    agentId: 'your-agent-id',

    // Authentication (choose one)
    apiKey: process.env.NEXT_PUBLIC_ELEVENLABS_API_KEY,

    // Client tools
    clientTools: {
      updateCart: {
        description: "Update shopping cart",
        parameters: z.object({
          item: z.string(),
          quantity: z.number()
        }),
        handler: async ({ item, quantity }) => {
          console.log('Cart updated:', item, quantity);
          return { success: true };
        }
      }
    },

    // Events
    onConnect: () => {
      console.log('Connected to agent');
      setTranscript([]);
    },
    onDisconnect: () => console.log('Disconnected'),
    onEvent: (event) => {
      if (event.type === 'transcript') {
        setTranscript(prev => [...prev, `User: ${event.data.text}`]);
      } else if (event.type === 'agent_response') {
        setTranscript(prev => [...prev, `Agent: ${event.data.text}`]);
      }
    },
    onError: (error) => console.error('Error:', error),

    // Regional compliance
    serverLocation: 'us'
  });

  return (
    <div>
      <div>
        <button onClick={startConversation} disabled={status === 'connected'}>
          Start Conversation
        </button>
        <button onClick={stopConversation} disabled={status !== 'connected'}>
          Stop
        </button>
      </div>

      <div>Status: {status}</div>
      <div>{isSpeaking && 'Agent is speaking...'}</div>

      <div>
        <h3>Transcript</h3>
        {transcript.map((line, i) => (
          <p key={i}>{line}</p>
        ))}
      </div>
    </div>
  );
}
安装:
bash
npm install @elevenlabs/react zod
完整示例:
typescript
import { useConversation } from '@elevenlabs/react';
import { z } from 'zod';
import { useState } from 'react';

export default function VoiceAgent() {
  const [transcript, setTranscript] = useState<string[]>([]);

  const {
    startConversation,
    stopConversation,
    status,
    isSpeaking
  } = useConversation({
    agentId: 'your-agent-id',

    // 认证(选择一种)
    apiKey: process.env.NEXT_PUBLIC_ELEVENLABS_API_KEY,

    // 客户端工具
    clientTools: {
      updateCart: {
        description: "更新购物车",
        parameters: z.object({
          item: z.string(),
          quantity: z.number()
        }),
        handler: async ({ item, quantity }) => {
          console.log('购物车已更新:', item, quantity);
          return { success: true };
        }
      }
    },

    // 事件
    onConnect: () => {
      console.log('已连接到代理');
      setTranscript([]);
    },
    onDisconnect: () => console.log('已断开连接'),
    onEvent: (event) => {
      if (event.type === 'transcript') {
        setTranscript(prev => [...prev, `用户: ${event.data.text}`]);
      } else if (event.type === 'agent_response') {
        setTranscript(prev => [...prev, `代理: ${event.data.text}`]);
      }
    },
    onError: (error) => console.error('错误:', error),

    // 区域合规
    serverLocation: 'us'
  });

  return (
    <div>
      <div>
        <button onClick={startConversation} disabled={status === 'connected'}>
          开始对话
        </button>
        <button onClick={stopConversation} disabled={status !== 'connected'}>
          停止
        </button>
      </div>

      <div>状态: {status}</div>
      <div>{isSpeaking && '代理正在说话...'}</div>

      <div>
        <h3>对话记录</h3>
        {transcript.map((line, i) => (
          <p key={i}>{line}</p>
        ))}
      </div>
    </div>
  );
}

JavaScript SDK (
@elevenlabs/client
)

JavaScript SDK (
@elevenlabs/client
)

For vanilla JavaScript projects (no React).
Installation:
bash
npm install @elevenlabs/client
Example:
javascript
import { Conversation } from '@elevenlabs/client';

const conversation = new Conversation({
  agentId: 'your-agent-id',
  apiKey: process.env.ELEVENLABS_API_KEY,

  onConnect: () => console.log('Connected'),
  onDisconnect: () => console.log('Disconnected'),

  onEvent: (event) => {
    switch (event.type) {
      case 'transcript':
        document.getElementById('user-text').textContent = event.data.text;
        break;
      case 'agent_response':
        document.getElementById('agent-text').textContent = event.data.text;
        break;
    }
  }
});

// Start conversation
document.getElementById('start-btn').addEventListener('click', async () => {
  await conversation.start();
});

// Stop conversation
document.getElementById('stop-btn').addEventListener('click', async () => {
  await conversation.stop();
});
适用于原生JavaScript项目(无React)。
安装:
bash
npm install @elevenlabs/client
示例:
javascript
import { Conversation } from '@elevenlabs/client';

const conversation = new Conversation({
  agentId: 'your-agent-id',
  apiKey: process.env.ELEVENLABS_API_KEY,

  onConnect: () => console.log('已连接'),
  onDisconnect: () => console.log('已断开连接'),

  onEvent: (event) => {
    switch (event.type) {
      case 'transcript':
        document.getElementById('user-text').textContent = event.data.text;
        break;
      case 'agent_response':
        document.getElementById('agent-text').textContent = event.data.text;
        break;
    }
  }
});

// 开始对话
document.getElementById('start-btn').addEventListener('click', async () => {
  await conversation.start();
});

// 停止对话
document.getElementById('stop-btn').addEventListener('click', async () => {
  await conversation.stop();
});

Connection Types: WebRTC vs WebSocket

连接类型:WebRTC vs WebSocket

ElevenLabs SDKs support two connection types with different characteristics.
Comparison Table:
FeatureWebSocketWebRTC
Authentication
signedUrl
conversationToken
Audio FormatConfigurablePCM_48000 (hardcoded)
Sample RateConfigurable (16k, 24k, 48k)48000 (hardcoded)
LatencyStandardLower
Device SwitchingFlexibleLimited (format locked)
Best ForGeneral use, flexibilityLow-latency requirements
WebSocket Configuration (default):
typescript
import { useConversation } from '@elevenlabs/react';

const { startConversation } = useConversation({
  agentId: 'your-agent-id',

  // WebSocket uses signed URL
  signedUrl: async () => {
    const response = await fetch('/api/elevenlabs/auth');
    const { signedUrl } = await response.json();
    return signedUrl;
  },

  // Connection type (optional, defaults to 'websocket')
  connectionType: 'websocket',

  // Audio config (flexible)
  audioConfig: {
    sampleRate: 24000, // 16000, 24000, or 48000
    format: 'PCM_24000'
  }
});
WebRTC Configuration:
typescript
const { startConversation } = useConversation({
  agentId: 'your-agent-id',

  // WebRTC uses conversation token (different auth flow)
  conversationToken: async () => {
    const response = await fetch('/api/elevenlabs/token');
    const { token } = await response.json();
    return token;
  },

  // Connection type
  connectionType: 'webrtc',

  // Audio format is HARDCODED to PCM_48000 (not configurable)
  // audioConfig ignored for WebRTC
});
Backend Token Endpoints:
typescript
// WebSocket signed URL (GET /v1/convai/conversation/get-signed-url)
app.get('/api/elevenlabs/auth', async (req, res) => {
  const response = await fetch(
    `https://api.elevenlabs.io/v1/convai/conversation/get-signed-url?agent_id=${AGENT_ID}`,
    { headers: { 'xi-api-key': ELEVENLABS_API_KEY } }
  );
  const { signed_url } = await response.json();
  res.json({ signedUrl: signed_url });
});

// WebRTC conversation token (GET /v1/convai/conversation/token)
app.get('/api/elevenlabs/token', async (req, res) => {
  const response = await fetch(
    `https://api.elevenlabs.io/v1/convai/conversation/token?agent_id=${AGENT_ID}`,
    { headers: { 'xi-api-key': ELEVENLABS_API_KEY } }
  );
  const { conversation_token } = await response.json();
  res.json({ token: conversation_token });
});
When to Use Each:
Use WebSocket WhenUse WebRTC When
Need flexible audio formatsNeed lowest possible latency
Switching between audio devices frequentlyAudio format can be locked to 48kHz
Standard latency is acceptableBuilding real-time applications
Need maximum configuration controlPerformance is critical
Gotchas:
  • WebRTC hardcodes PCM_48000 - no way to change format
  • Device switching in WebRTC limited by fixed format
  • Different authentication methods (signedUrl vs conversationToken)
  • WebRTC may have better performance but less flexibility
ElevenLabs SDK支持两种连接类型,具有不同特性。
对比表:
特性WebSocketWebRTC
认证
signedUrl
conversationToken
音频格式可配置PCM_48000(硬编码)
采样率可配置(16k、24k、48k)48000(硬编码)
延迟标准更低
设备切换灵活有限(格式锁定)
最佳适用场景通用场景、灵活性低延迟需求
WebSocket配置(默认):
typescript
import { useConversation } from '@elevenlabs/react';

const { startConversation } = useConversation({
  agentId: 'your-agent-id',

  // WebSocket使用签名URL
  signedUrl: async () => {
    const response = await fetch('/api/elevenlabs/auth');
    const { signedUrl } = await response.json();
    return signedUrl;
  },

  // 连接类型(可选,默认为'websocket')
  connectionType: 'websocket',

  // 音频配置(灵活)
  audioConfig: {
    sampleRate: 24000, // 16000、24000或48000
    format: 'PCM_24000'
  }
});
WebRTC配置:
typescript
const { startConversation } = useConversation({
  agentId: 'your-agent-id',

  // WebRTC使用对话令牌(不同的认证流程)
  conversationToken: async () => {
    const response = await fetch('/api/elevenlabs/token');
    const { token } = await response.json();
    return token;
  },

  // 连接类型
  connectionType: 'webrtc',

  // 音频格式硬编码为PCM_48000(不可配置)
  // audioConfig对WebRTC无效
});
后端令牌端点:
typescript
// WebSocket签名URL(GET /v1/convai/conversation/get-signed-url)
app.get('/api/elevenlabs/auth', async (req, res) => {
  const response = await fetch(
    `https://api.elevenlabs.io/v1/convai/conversation/get-signed-url?agent_id=${AGENT_ID}`,
    { headers: { 'xi-api-key': ELEVENLABS_API_KEY } }
  );
  const { signed_url } = await response.json();
  res.json({ signedUrl: signed_url });
});

// WebRTC对话令牌(GET /v1/convai/conversation/token)
app.get('/api/elevenlabs/token', async (req, res) => {
  const response = await fetch(
    `https://api.elevenlabs.io/v1/convai/conversation/token?agent_id=${AGENT_ID}`,
    { headers: { 'xi-api-key': ELEVENLABS_API_KEY } }
  );
  const { conversation_token } = await response.json();
  res.json({ token: conversation_token });
});
何时使用每种类型:
使用WebSocket的场景使用WebRTC的场景
需要灵活的音频格式需要最低可能的延迟
频繁在音频设备之间切换音频格式可以锁定为48kHz
标准延迟可接受构建实时应用
需要最大配置控制性能至关重要
注意事项:
  • WebRTC硬编码PCM_48000 - 无法更改格式
  • WebRTC中的设备切换受固定格式限制
  • 不同的认证方法(signedUrl vs conversationToken)
  • WebRTC可能性能更好但灵活性更低

React Native SDK (Expo)

React Native SDK(Expo)

Installation:
bash
npx expo install @elevenlabs/react-native @livekit/react-native @livekit/react-native-webrtc livekit-client
Requirements:
  • Expo SDK 47+
  • iOS 14.0+ / macOS 11.0+
  • Custom dev build required (Expo Go not supported)
Example:
typescript
import { useConversation } from '@elevenlabs/react-native';
import { View, Button, Text } from 'react-native';
import { z } from 'zod';

export default function App() {
  const { startConversation, stopConversation, status } = useConversation({
    agentId: 'your-agent-id',
    signedUrl: 'https://api.elevenlabs.io/v1/convai/auth/...',

    clientTools: {
      updateProfile: {
        description: "Update user profile",
        parameters: z.object({
          name: z.string()
        }),
        handler: async ({ name }) => {
          console.log('Updating profile:', name);
          return { success: true };
        }
      }
    }
  });

  return (
    <View style={{ padding: 20 }}>
      <Button title="Start" onPress={startConversation} />
      <Button title="Stop" onPress={stopConversation} />
      <Text>Status: {status}</Text>
    </View>
  );
}
Gotchas:
  • Requires custom dev build (not Expo Go)
  • iOS/macOS only (Android support via Kotlin SDK, not yet officially released)
安装:
bash
npx expo install @elevenlabs/react-native @livekit/react-native @livekit/react-native-webrtc livekit-client
要求:
  • Expo SDK 47+
  • iOS 14.0+ / macOS 11.0+
  • 需要自定义开发构建(不支持Expo Go)
示例:
typescript
import { useConversation } from '@elevenlabs/react-native';
import { View, Button, Text } from 'react-native';
import { z } from 'zod';

export default function App() {
  const { startConversation, stopConversation, status } = useConversation({
    agentId: 'your-agent-id',
    signedUrl: 'https://api.elevenlabs.io/v1/convai/auth/...',

    clientTools: {
      updateProfile: {
        description: "更新用户资料",
        parameters: z.object({
          name: z.string()
        }),
        handler: async ({ name }) => {
          console.log('正在更新资料:', name);
          return { success: true };
        }
      }
    }
  });

  return (
    <View style={{ padding: 20 }}>
      <Button title="开始" onPress={startConversation} />
      <Button title="停止" onPress={stopConversation} />
      <Text>状态: {status}</Text>
    </View>
  );
}
注意事项:
  • 需要自定义开发构建(不支持Expo Go)
  • 仅支持iOS/macOS(Android支持通过Kotlin SDK,尚未正式发布)

Swift SDK (iOS/macOS)

Swift SDK(iOS/macOS)

Installation (Swift Package Manager):
swift
dependencies: [
  .package(url: "https://github.com/elevenlabs/elevenlabs-swift-sdk", from: "1.0.0")
]
Requirements:
  • iOS 14.0+ / macOS 11.0+
  • Swift 5.9+
Use Cases:
  • Native iOS apps
  • macOS applications
  • watchOS (with limitations)
安装(Swift包管理器):
swift
dependencies: [
  .package(url: "https://github.com/elevenlabs/elevenlabs-swift-sdk", from: "1.0.0")
]
要求:
  • iOS 14.0+ / macOS 11.0+
  • Swift 5.9+
使用场景:
  • 原生iOS应用
  • macOS应用
  • watchOS(有局限性)

Widget (Embeddable Web Component)

小部件(可嵌入Web组件)

Installation: Copy-paste embed code from dashboard or use this template:
html
<script src="https://elevenlabs.io/convai-widget/index.js"></script>
<script>
  ElevenLabsWidget.init({
    agentId: 'your-agent-id',

    // Theming
    theme: {
      primaryColor: '#3B82F6',
      backgroundColor: '#1F2937',
      textColor: '#F9FAFB'
    },

    // Position
    position: 'bottom-right', // 'bottom-left' | 'bottom-right'

    // Custom branding
    branding: {
      logo: 'https://example.com/logo.png',
      name: 'Support Agent'
    }
  });
</script>
Use Cases:
  • Customer support chat bubbles
  • Website assistants
  • Lead capture forms
安装: 从仪表板复制粘贴嵌入代码或使用以下模板:
html
<script src="https://elevenlabs.io/convai-widget/index.js"></script>
<script>
  ElevenLabsWidget.init({
    agentId: 'your-agent-id',

    // 主题
    theme: {
      primaryColor: '#3B82F6',
      backgroundColor: '#1F2937',
      textColor: '#F9FAFB'
    },

    // 位置
    position: 'bottom-right', // 'bottom-left' | 'bottom-right'

    // 自定义品牌
    branding: {
      logo: 'https://example.com/logo.png',
      name: 'Support Agent'
    }
  });
</script>
使用场景:
  • 客户支持聊天气泡
  • 网站助手
  • 潜在客户捕获表单

Scribe (Real-Time Speech-to-Text)

Scribe(实时语音转文本)

Status: Closed Beta (requires sales contact) Release: 2025
Scribe is ElevenLabs' real-time speech-to-text service for low-latency transcription.
Capabilities:
  • Microphone streaming (real-time transcription)
  • Pre-recorded audio file transcription
  • Partial (interim) and final transcripts
  • Word-level timestamps
  • Voice Activity Detection (VAD)
  • Manual and automatic commit strategies
  • Language detection
  • PCM_16000 and PCM_24000 audio formats
Authentication: Uses single-use tokens (not API keys):
typescript
// Fetch token from backend
const response = await fetch('/api/scribe/token');
const { token } = await response.json();

// Backend endpoint
const token = await client.scribe.getToken();
return { token };
React Hook (
useScribe
)
:
typescript
import { useScribe } from '@elevenlabs/react';

export default function Transcription() {
  const {
    connect,
    disconnect,
    startRecording,
    stopRecording,
    status,
    transcript,
    partialTranscript
  } = useScribe({
    token: async () => {
      const response = await fetch('/api/scribe/token');
      const { token } = await response.json();
      return token;
    },

    // Commit strategy
    commitStrategy: 'vad', // 'vad' (automatic) or 'manual'

    // Audio format
    sampleRate: 16000, // 16000 or 24000

    // Events
    onConnect: () => console.log('Connected to Scribe'),
    onDisconnect: () => console.log('Disconnected'),

    onPartialTranscript: (text) => {
      console.log('Interim:', text);
    },

    onFinalTranscript: (text, timestamps) => {
      console.log('Final:', text);
      console.log('Timestamps:', timestamps); // Word-level timing
    },

    onError: (error) => console.error('Error:', error)
  });

  return (
    <div>
      <button onClick={connect}>Connect</button>
      <button onClick={startRecording}>Start Recording</button>
      <button onClick={stopRecording}>Stop Recording</button>
      <button onClick={disconnect}>Disconnect</button>

      <p>Status: {status}</p>
      <p>Partial: {partialTranscript}</p>
      <p>Final: {transcript}</p>
    </div>
  );
}
JavaScript SDK (
Scribe.connect
)
:
javascript
import { Scribe } from '@elevenlabs/client';

const connection = await Scribe.connect({
  token: 'your-single-use-token',

  sampleRate: 16000,
  commitStrategy: 'vad',

  onPartialTranscript: (text) => {
    document.getElementById('interim').textContent = text;
  },

  onFinalTranscript: (text, timestamps) => {
    const finalDiv = document.getElementById('final');
    finalDiv.textContent += text + ' ';

    // timestamps: [{ word: 'hello', start: 0.5, end: 0.8 }, ...]
  },

  onError: (error) => {
    console.error('Scribe error:', error);
  }
});

// Start recording from microphone
await connection.startRecording();

// Stop recording
await connection.stopRecording();

// Manual commit (if commitStrategy: 'manual')
await connection.commit();

// Disconnect
await connection.disconnect();
Transcribing Pre-Recorded Files:
typescript
import { Scribe } from '@elevenlabs/client';

const connection = await Scribe.connect({ token });

// Send audio buffer
const audioBuffer = fs.readFileSync('recording.pcm');
await connection.sendAudioData(audioBuffer);

// Manually commit to get final transcript
await connection.commit();

// Wait for final transcript event
Event Types:
  • SESSION_STARTED
    : Connection established
  • PARTIAL_TRANSCRIPT
    : Interim transcription (unbuffered)
  • FINAL_TRANSCRIPT
    : Complete sentence/phrase
  • FINAL_TRANSCRIPT_WITH_TIMESTAMPS
    : Final + word timing
  • ERROR
    : Transcription error
  • AUTH_ERROR
    : Authentication failed
  • OPEN
    : WebSocket opened
  • CLOSE
    : WebSocket closed
Commit Strategies:
StrategyDescriptionUse When
vad
(automatic)
Voice Activity Detection auto-commits on silenceReal-time transcription
manual
Call
connection.commit()
explicitly
Pre-recorded files, controlled commits
Audio Formats:
  • PCM_16000
    (16kHz, 16-bit PCM)
  • PCM_24000
    (24kHz, 16-bit PCM)
Gotchas:
  • Token is single-use (expires after one connection)
  • Closed beta - requires sales contact
  • Language detection automatic (no manual override)
  • No speaker diarization yet
When to Use Scribe:
  • Building custom transcription UI
  • Real-time captions/subtitles
  • Voice note apps
  • Meeting transcription
  • Accessibility features
When NOT to Use: Use Agents Platform instead if you need:
  • Conversational AI (LLM + TTS)
  • Two-way voice interaction
  • Agent responses

状态: 封闭Beta版(需要联系销售) 发布: 2025年
Scribe是ElevenLabs的实时语音转文本服务,用于低延迟转录。
功能:
  • 麦克风流(实时转录)
  • 预录音频文件转录
  • 部分(临时)和最终转录本
  • 单词级时间戳
  • 语音活动检测(VAD)
  • 手动和自动提交策略
  • 语言检测
  • PCM_16000和PCM_24000音频格式
认证: 使用一次性令牌(而非API密钥):
typescript
// 从后端获取令牌
const response = await fetch('/api/scribe/token');
const { token } = await response.json();

// 后端端点
const token = await client.scribe.getToken();
return { token };
React Hook (
useScribe
)
:
typescript
import { useScribe } from '@elevenlabs/react';

export default function Transcription() {
  const {
    connect,
    disconnect,
    startRecording,
    stopRecording,
    status,
    transcript,
    partialTranscript
  } = useScribe({
    token: async () => {
      const response = await fetch('/api/scribe/token');
      const { token } = await response.json();
      return token;
    },

    // 提交策略
    commitStrategy: 'vad', // 'vad'(自动)或 'manual'

    // 音频格式
    sampleRate: 16000, // 16000或24000

    // 事件
    onConnect: () => console.log('已连接到Scribe'),
    onDisconnect: () => console.log('已断开连接'),

    onPartialTranscript: (text) => {
      console.log('临时转录:', text);
    },

    onFinalTranscript: (text, timestamps) => {
      console.log('最终转录:', text);
      console.log('时间戳:', timestamps); // 单词级时序
    },

    onError: (error) => console.error('错误:', error)
  });

  return (
    <div>
      <button onClick={connect}>连接</button>
      <button onClick={startRecording}>开始录音</button>
      <button onClick={stopRecording}>停止录音</button>
      <button onClick={disconnect}>断开连接</button>

      <p>状态: {status}</p>
      <p>临时转录: {partialTranscript}</p>
      <p>最终转录: {transcript}</p>
    </div>
  );
}
JavaScript SDK (
Scribe.connect
)
:
javascript
import { Scribe } from '@elevenlabs/client';

const connection = await Scribe.connect({
  token: 'your-single-use-token',

  sampleRate: 16000,
  commitStrategy: 'vad',

  onPartialTranscript: (text) => {
    document.getElementById('interim').textContent = text;
  },

  onFinalTranscript: (text, timestamps) => {
    const finalDiv = document.getElementById('final');
    finalDiv.textContent += text + ' ';

    // timestamps: [{ word: 'hello', start: 0.5, end: 0.8 }, ...]
  },

  onError: (error) => {
    console.error('Scribe错误:', error);
  }
});

// 开始从麦克风录音
await connection.startRecording();

// 停止录音
await connection.stopRecording();

// 手动提交(如果commitStrategy: 'manual')
await connection.commit();

// 断开连接
await connection.disconnect();
转录预录文件:
typescript
import { Scribe } from '@elevenlabs/client';
import fs from 'fs';

const connection = await Scribe.connect({ token });

// 发送音频缓冲区
const audioBuffer = fs.readFileSync('recording.pcm');
await connection.sendAudioData(audioBuffer);

// 手动提交以获取最终转录本
await connection.commit();

// 等待最终转录事件
事件类型:
  • SESSION_STARTED
    : 连接已建立
  • PARTIAL_TRANSCRIPT
    : 临时转录(无缓冲)
  • FINAL_TRANSCRIPT
    : 完整句子/短语
  • FINAL_TRANSCRIPT_WITH_TIMESTAMPS
    : 最终转录 + 单词时序
  • ERROR
    : 转录错误
  • AUTH_ERROR
    : 认证失败
  • OPEN
    : WebSocket已打开
  • CLOSE
    : WebSocket已关闭
提交策略:
策略描述使用场景
vad
(自动)
语音活动检测在静音时自动提交实时转录
manual
显式调用
connection.commit()
预录文件、受控提交
音频格式:
  • PCM_16000
    (16kHz,16位PCM)
  • PCM_24000
    (24kHz,16位PCM)
注意事项:
  • 令牌是一次性的(连接后过期)
  • 封闭Beta版 - 需要联系销售
  • 语言检测自动进行(无手动覆盖)
  • 尚不支持说话人分离
何时使用Scribe:
  • 构建自定义转录UI
  • 实时字幕/副标题
  • 语音笔记应用
  • 会议转录
  • 无障碍功能
何时不使用: 如果您需要以下功能,请改用Agents Platform:
  • 对话式AI(LLM + TTS)
  • 双向语音交互
  • 代理回复

7. Testing & Evaluation

7. 测试与评估

Scenario Testing (LLM-Based Evaluation)

场景测试(基于LLM的评估)

Simulate full conversations and evaluate against success criteria.
Configuration via CLI:
bash
elevenlabs tests add "Refund Request Test" --template basic-llm
test_configs/refund-request-test.json:
json
{
  "name": "Refund Request Test",
  "scenario": "Customer requests refund for defective product",
  "user_input": "I want a refund for order #12345. The product was broken when it arrived.",
  "success_criteria": [
    "Agent acknowledges the request empathetically",
    "Agent asks for order number (which was already provided)",
    "Agent verifies order details",
    "Agent provides refund timeline or next steps"
  ],
  "evaluation_type": "llm"
}
Run Test:
bash
elevenlabs agents test "Support Agent"
模拟完整对话并根据成功标准进行评估。
通过CLI配置:
bash
elevenlabs tests add "Refund Request Test" --template basic-llm
test_configs/refund-request-test.json:
json
{
  "name": "退款请求测试",
  "scenario": "客户要求为有缺陷的产品退款",
  "user_input": "我想为订单#12345退款。产品到货时已损坏。",
  "success_criteria": [
    "代理感同身受地确认请求",
    "代理询问订单号(已提供)",
    "代理验证订单详情",
    "代理提供退款时间线或下一步操作"
  ],
  "evaluation_type": "llm"
}
运行测试:
bash
elevenlabs agents test "Support Agent"

Tool Call Testing

工具调用测试

Verify that agents correctly use tools with the right parameters.
Configuration:
json
{
  "name": "Account Balance Test",
  "scenario": "Customer requests account balance",
  "expected_tool_call": {
    "tool_name": "get_account_balance",
    "parameters": {
      "account_id": "ACC-12345"
    }
  }
}
验证代理是否正确使用工具并传入正确参数。
配置:
json
{
  "name": "账户余额测试",
  "scenario": "客户查询账户余额",
  "expected_tool_call": {
    "tool_name": "get_account_balance",
    "parameters": {
      "account_id": "ACC-12345"
    }
  }
}

Load Testing

负载测试

Test agent capacity under high concurrency.
Configuration:
bash
undefined
测试代理在高并发下的容量。
配置:
bash
undefined

Spawn 100 users, 1 per second, test for 10 minutes

生成100个用户,每秒1个,测试10分钟

elevenlabs test load
--users 100
--spawn-rate 1
--duration 600

**Gotchas**:
- Load testing consumes real API credits
- Use burst pricing for expected traffic spikes
- Requires careful planning to avoid hitting rate limits
elevenlabs test load
--users 100
--spawn-rate 1
--duration 600

**注意事项**:
- 负载测试会消耗真实API积分
- 对预期流量峰值使用峰值定价
- 需要仔细规划以避免触发速率限制

Simulation API (Programmatic Testing)

模拟API(程序化测试)

API Endpoint:
bash
POST /v1/convai/agents/:agent_id/simulate
Example:
typescript
const simulation = await client.agents.simulate({
  agent_id: 'agent_123',
  scenario: 'Customer requests refund',
  user_messages: [
    "I want a refund for order #12345",
    "I ordered it last week",
    "Yes, please process it"
  ],
  success_criteria: [
    "Agent acknowledges request",
    "Agent asks for order details",
    "Agent provides refund timeline"
  ]
});

console.log('Test passed:', simulation.passed);
console.log('Criteria met:', simulation.evaluation.criteria_met);
Use Cases:
  • CI/CD integration (test before deploy)
  • Regression testing
  • Load testing preparation

API端点:
bash
POST /v1/convai/agents/:agent_id/simulate
示例:
typescript
const simulation = await client.agents.simulate({
  agent_id: 'agent_123',
  scenario: '客户要求退款',
  user_messages: [
    "我想为订单#12345退款",
    "我上周下单的",
    "是的,请处理"
  ],
  success_criteria: [
    "代理确认请求",
    "代理询问订单详情",
    "代理提供退款时间线"
  ]
});

console.log('测试通过:', simulation.passed);
console.log('符合的标准:', simulation.evaluation.criteria_met);
使用场景:
  • CI/CD集成(部署前测试)
  • 回归测试
  • 负载测试准备

8. Analytics & Monitoring

8. 分析与监控

Conversation Analysis

对话分析

Extract structured data from conversation transcripts.
Features:
从对话转录本中提取结构化数据。
功能:

Success Evaluation (LLM-Based)

基于LLM的成功评估

json
{
  "evaluation_criteria": {
    "resolution": "Was the customer's issue resolved?",
    "sentiment": "Was the conversation tone positive?",
    "compliance": "Did the agent follow company policies?"
  }
}
json
{
  "evaluation_criteria": {
    "resolution": "客户的问题是否已解决?",
    "sentiment": "对话语气是否积极?",
    "compliance": "代理是否遵循公司政策?"
  }
}

Data Collection

数据收集

json
{
  "data_collection": {
    "fields": [
      { "name": "customer_name", "type": "string" },
      { "name": "issue_type", "type": "enum", "values": ["billing", "technical", "other"] },
      { "name": "satisfaction", "type": "number", "range": [1, 5] }
    ]
  }
}
Access:
  • Via Post-call Webhooks (real-time)
  • Via Analytics Dashboard (batch)
  • Via API (on-demand)
json
{
  "data_collection": {
    "fields": [
      { "name": "customer_name", "type": "string" },
      { "name": "issue_type", "type": "enum", "values": ["billing", "technical", "other"] },
      { "name": "satisfaction", "type": "number", "range": [1, 5] }
    ]
  }
}
访问方式:
  • 通过通话后Webhook(实时)
  • 通过分析仪表盘(批量)
  • 通过API(按需)

Analytics Dashboard

分析仪表盘

Metrics:
  • Resolution Rates: % of issues resolved
  • CX Metrics: Sentiment, satisfaction, CSAT
  • Compliance: Policy adherence, guardrail violations
  • Performance: Response time, call duration, concurrency
  • Tool Usage: Tool call frequency, success rates
  • LLM Costs: Track costs per agent/conversation
Access: Dashboard → Analytics tab

指标:
  • 解决率: 问题已解决的百分比
  • CX指标: 情感、满意度、CSAT
  • 合规性: 政策遵守情况、防护规则违规
  • 性能: 响应时间、通话时长、并发性
  • 工具使用: 工具调用频率、成功率
  • LLM成本: 按代理/对话跟踪成本
访问: 仪表盘 → 分析标签页

9. Privacy & Compliance

9. 隐私与合规

Data Retention

数据保留

Default: 2 years (GDPR-compliant)
Configuration:
json
{
  "privacy": {
    "transcripts": {
      "retention_days": 730  // 2 years (GDPR)
    },
    "audio": {
      "retention_days": 2190  // 6 years (HIPAA)
    }
  }
}
Compliance Recommendations:
  • GDPR: Align with data processing purposes (typically 1-2 years)
  • HIPAA: Minimum 6 years for medical records
  • SOC 2: Encryption in transit and at rest (automatic)
默认: 2年(符合GDPR)
配置:
json
{
  "privacy": {
    "transcripts": {
      "retention_days": 730  // 2年(GDPR)
    },
    "audio": {
      "retention_days": 2190  // 6年(HIPAA)
    }
  }
}
合规建议:
  • GDPR: 与数据处理目的保持一致(通常1-2年)
  • HIPAA: 医疗记录至少保留6年
  • SOC 2: 传输中与静态数据加密(自动)

Encryption

加密

  • In Transit: TLS 1.3
  • At Rest: AES-256
  • Regional Compliance: Data residency (US, EU, India)
Regional Configuration:
typescript
const { startConversation } = useConversation({
  serverLocation: 'eu-residency' // 'us' | 'global' | 'eu-residency' | 'in-residency'
});
  • 传输中: TLS 1.3
  • 静态: AES-256
  • 区域合规: 数据驻留(美国、欧盟、印度)
区域配置:
typescript
const { startConversation } = useConversation({
  serverLocation: 'eu-residency' // 'us' | 'global' | 'eu-residency' | 'in-residency'
});

Zero Retention Mode

零保留模式

For maximum privacy, enable zero retention to immediately delete all conversation data.
Limitations:
  • No conversation history
  • No analytics
  • No post-call webhooks
  • No MCP tool integrations

为了最大程度的隐私,启用零保留模式可立即删除所有对话数据。
局限性:
  • 无对话历史
  • 无分析
  • 无通话后Webhook
  • 无MCP工具集成

10. Cost Optimization

10. 成本优化

LLM Caching

LLM缓存

Reduce costs by caching repeated inputs.
How It Works:
  • First request: Full cost (
    input_cache_write
    )
  • Subsequent requests: Reduced cost (
    input_cache_read
    )
  • Automatic cache invalidation after TTL
Configuration:
json
{
  "llm_config": {
    "caching": {
      "enabled": true,
      "ttl_seconds": 3600  // 1 hour
    }
  }
}
Use Cases:
  • Repeated system prompts
  • Large knowledge bases
  • Frequent tool definitions
Savings: Up to 90% on cached inputs
通过缓存重复输入降低成本。
工作原理:
  • 首次请求: 全额费用 (
    input_cache_write
    )
  • 后续请求: 降低费用 (
    input_cache_read
    )
  • 自动缓存失效 在TTL后
配置:
json
{
  "llm_config": {
    "caching": {
      "enabled": true,
      "ttl_seconds": 3600  // 1小时
    }
  }
}
使用场景:
  • 重复的系统提示词
  • 大型知识库
  • 频繁的工具定义
节省: 缓存输入最多可节省90%

Model Swapping

模型切换

Switch between models based on cost/performance needs.
Available Models:
  • GPT-4o (high cost, high quality)
  • GPT-4o-mini (medium cost, good quality)
  • Claude Sonnet 4.5 (high cost, best reasoning)
  • Gemini 2.5 Flash (low cost, fast)
Configuration:
json
{
  "llm_config": {
    "model": "gpt-4o-mini"  // Swap anytime via dashboard or API
  }
}
根据成本/性能需求在模型之间切换。
可用模型:
  • GPT-4o(高成本,高质量)
  • GPT-4o-mini(中等成本,良好质量)
  • Claude Sonnet 4.5(高成本,最佳推理)
  • Gemini 2.5 Flash(低成本,快速)
配置:
json
{
  "llm_config": {
    "model": "gpt-4o-mini"  // 随时通过仪表盘或API切换
  }
}

Burst Pricing

峰值定价

Temporarily exceed concurrency limits during high-demand periods.
How It Works:
  • Normal: Your subscription concurrency limit (e.g., 10 simultaneous calls)
  • Burst: Up to 3x your limit (e.g., 30 simultaneous calls)
  • Cost: 2x the standard rate for burst calls
Configuration:
json
{
  "call_limits": {
    "burst_pricing_enabled": true
  }
}
Use Cases:
  • Black Friday traffic spikes
  • Product launches
  • Seasonal demand (holidays)
Gotchas:
  • Burst calls cost 2x (plan accordingly)
  • Not unlimited (3x cap)

在高需求期间临时超过并发限制。
工作原理:
  • 正常: 您的订阅并发限制(例如,10个同时通话)
  • 峰值: 最多达到限制的3倍(例如,30个同时通话)
  • 成本: 峰值通话的费用是标准费率的2倍
配置:
json
{
  "call_limits": {
    "burst_pricing_enabled": true
  }
}
使用场景:
  • 黑色星期五流量峰值
  • 产品发布
  • 季节性需求(节假日)
注意事项:
  • 峰值通话费用是2倍(请相应规划)
  • 并非无限制(3倍上限)

11. Advanced Features

11. 高级功能

Events (WebSocket/SSE)

事件(WebSocket/SSE)

Real-time event streaming for live transcription, agent responses, and tool calls.
Event Types:
  • audio
    - Audio stream chunks
  • transcript
    - Real-time transcription
  • agent_response
    - Agent's text response
  • tool_call
    - Tool execution status
  • conversation_state
    - State updates
Example:
typescript
const { startConversation } = useConversation({
  onEvent: (event) => {
    switch (event.type) {
      case 'transcript':
        console.log('User said:', event.data.text);
        break;
      case 'agent_response':
        console.log('Agent replied:', event.data.text);
        break;
      case 'tool_call':
        console.log('Tool called:', event.data.tool_name);
        break;
    }
  }
});
用于实时转录、代理回复和工具调用的实时事件流。
事件类型:
  • audio
    - 音频流片段
  • transcript
    - 实时转录
  • agent_response
    - 代理的文本回复
  • tool_call
    - 工具执行状态
  • conversation_state
    - 状态更新
示例:
typescript
const { startConversation } = useConversation({
  onEvent: (event) => {
    switch (event.type) {
      case 'transcript':
        console.log('用户说:', event.data.text);
        break;
      case 'agent_response':
        console.log('代理回复:', event.data.text);
        break;
      case 'tool_call':
        console.log('已调用工具:', event.data.tool_name);
        break;
    }
  }
});

Custom Models (Bring Your Own LLM)

自定义模型(自带LLM)

Use your own OpenAI API key or custom LLM server.
Configuration:
json
{
  "llm_config": {
    "custom": {
      "endpoint": "https://api.openai.com/v1/chat/completions",
      "api_key": "{{secret__openai_api_key}}",
      "model": "gpt-4"
    }
  }
}
Use Cases:
  • Custom fine-tuned models
  • Private LLM deployments (Ollama, LocalAI)
  • Cost control (use your own credits)
  • Compliance (on-premise models)
Gotchas:
  • Endpoint must be OpenAI-compatible
  • No official support for non-OpenAI-compatible models
使用您自己的OpenAI API密钥或自定义LLM服务器。
配置:
json
{
  "llm_config": {
    "custom": {
      "endpoint": "https://api.openai.com/v1/chat/completions",
      "api_key": "{{secret__openai_api_key}}",
      "model": "gpt-4"
    }
  }
}
使用场景:
  • 自定义微调模型
  • 私有LLM部署(Ollama、LocalAI)
  • 成本控制(使用您自己的积分)
  • 合规性(本地部署模型)
注意事项:
  • 端点必须与OpenAI兼容
  • 官方不支持非OpenAI兼容模型

Post-Call Webhooks

通话后Webhook

Receive notifications when a call ends and analysis completes.
Configuration:
json
{
  "webhooks": {
    "post_call": {
      "url": "https://api.example.com/webhook",
      "headers": {
        "Authorization": "Bearer {{secret__webhook_auth_token}}"
      }
    }
  }
}
Payload:
json
{
  "conversation_id": "conv_123",
  "agent_id": "agent_456",
  "transcript": "...",
  "duration_seconds": 120,
  "analysis": {
    "sentiment": "positive",
    "resolution": true,
    "extracted_data": {
      "customer_name": "John Doe",
      "issue_type": "billing"
    }
  }
}
Security (HMAC Verification):
typescript
import crypto from 'crypto';

export async function POST(req: Request) {
  const signature = req.headers.get('elevenlabs-signature');
  const payload = await req.text();

  const hmac = crypto
    .createHmac('sha256', process.env.WEBHOOK_SECRET!)
    .update(payload)
    .digest('hex');

  if (signature !== hmac) {
    return new Response('Invalid signature', { status: 401 });
  }

  // Process webhook
  const data = JSON.parse(payload);
  console.log('Conversation ended:', data.conversation_id);

  // MUST return 200
  return new Response('OK', { status: 200 });
}
Gotchas:
  • Must return 200 status code
  • Auto-disabled after 10 consecutive failures (7+ days since last success)
  • Retry logic: 3 attempts with exponential backoff
通话结束且分析完成时接收通知。
配置:
json
{
  "webhooks": {
    "post_call": {
      "url": "https://api.example.com/webhook",
      "headers": {
        "Authorization": "Bearer {{secret__webhook_auth_token}}"
      }
    }
  }
}
负载:
json
{
  "conversation_id": "conv_123",
  "agent_id": "agent_456",
  "transcript": "...",
  "duration_seconds": 120,
  "analysis": {
    "sentiment": "positive",
    "resolution": true,
    "extracted_data": {
      "customer_name": "John Doe",
      "issue_type": "billing"
    }
  }
}
安全性(HMAC验证):
typescript
import crypto from 'crypto';

export async function POST(req: Request) {
  const signature = req.headers.get('elevenlabs-signature');
  const payload = await req.text();

  const hmac = crypto
    .createHmac('sha256', process.env.WEBHOOK_SECRET!)
    .update(payload)
    .digest('hex');

  if (signature !== hmac) {
    return new Response('无效签名', { status: 401 });
  }

  // 处理Webhook
  const data = JSON.parse(payload);
  console.log('对话已结束:', data.conversation_id);

  // 必须返回200
  return new Response('OK', { status: 200 });
}
注意事项:
  • 必须返回200状态码
  • 连续10次失败后自动禁用(自上次成功以来7+天)
  • 重试逻辑: 3次尝试,指数退避

Chat Mode (Text-Only)

聊天模式(纯文本)

Disable voice, use text-only conversations.
Configuration:
json
{
  "conversation_config": {
    "chat_mode": true  // Disables audio input/output
  }
}
Benefits:
  • Faster response times (~200ms saved)
  • Lower costs (no ASR/TTS charges)
  • Easier testing (no microphone required)
Use Cases:
  • Testing agents without audio
  • Building text chat interfaces
  • Accessibility (text-only users)
禁用语音,使用纯文本对话。
配置:
json
{
  "conversation_config": {
    "chat_mode": true  // 禁用音频输入/输出
  }
}
优势:
  • 更快的响应时间(节省约200ms)
  • 更低的成本(无ASR/TTS费用)
  • 更容易测试(无需麦克风)
使用场景:
  • 无需音频测试代理
  • 构建文本聊天界面
  • 无障碍访问(纯文本用户)

Telephony Integration

电话系统集成

SIP Trunking:
SIP Endpoint: sip-static.rtc.elevenlabs.io
TLS Transport: Recommended for production
SRTP Encryption: Supported
Supported Providers: Twilio, Vonage, RingCentral, Sinch, Infobip, Telnyx, Exotel, Plivo, Bandwidth
Native Twilio Integration:
json
{
  "telephony": {
    "provider": "twilio",
    "phone_number": "+1234567890",
    "account_sid": "{{secret__twilio_account_sid}}",
    "auth_token": "{{secret__twilio_auth_token}}"
  }
}
Use Cases:
  • Customer support hotlines
  • Appointment scheduling
  • Order status inquiries
  • IVR systems

SIP中继:
SIP端点: sip-static.rtc.elevenlabs.io
TLS传输: 生产环境推荐
SRTP加密: 支持
支持提供商: Twilio、Vonage、RingCentral、Sinch、Infobip、Telnyx、Exotel、Plivo、Bandwidth
原生Twilio集成:
json
{
  "telephony": {
    "provider": "twilio",
    "phone_number": "+1234567890",
    "account_sid": "{{secret__twilio_account_sid}}",
    "auth_token": "{{secret__twilio_auth_token}}"
  }
}
使用场景:
  • 客户支持热线
  • 预约预订
  • 订单状态查询
  • IVR系统

12. CLI & DevOps ("Agents as Code")

12. CLI与DevOps(“代理即代码”)

Installation & Authentication

安装与认证

bash
undefined
bash
undefined

Install globally

全局安装

npm install -g @elevenlabs/cli
npm install -g @elevenlabs/cli

Authenticate

认证

elevenlabs auth login
elevenlabs auth login

Set residency (for GDPR compliance)

设置驻留(用于GDPR合规)

elevenlabs auth residency eu-residency # or 'in-residency' | 'global'
elevenlabs auth residency eu-residency # 或 'in-residency' | 'global'

Check current user

检查当前用户

elevenlabs auth whoami

**Environment Variables** (For CI/CD):
```bash
export ELEVENLABS_API_KEY=your-api-key
elevenlabs auth whoami

**环境变量**(用于CI/CD):
```bash
export ELEVENLABS_API_KEY=your-api-key

Project Structure

项目结构

Initialize Project:
bash
elevenlabs agents init
Directory Structure Created:
your_project/
├── agents.json              # Agent registry
├── tools.json               # Tool configurations
├── tests.json               # Test configurations
├── agent_configs/           # Individual agent files (.json)
├── tool_configs/            # Tool configuration files
└── test_configs/            # Test configuration files
初始化项目:
bash
elevenlabs agents init
创建的目录结构:
your_project/
├── agents.json              # 代理注册表
├── tools.json               # 工具配置
├── tests.json               # 测试配置
├── agent_configs/           # 单个代理文件 (.json)
├── tool_configs/            # 工具配置文件
└── test_configs/            # 测试配置文件

Agent Management Commands

代理管理命令

bash
undefined
bash
undefined

Create agent

创建代理

elevenlabs agents add "Support Agent" --template customer-service
elevenlabs agents add "Support Agent" --template customer-service

Deploy to platform

部署到平台

elevenlabs agents push elevenlabs agents push --agent "Support Agent" elevenlabs agents push --env prod elevenlabs agents push --dry-run # Preview changes
elevenlabs agents push elevenlabs agents push --agent "Support Agent" elevenlabs agents push --env prod elevenlabs agents push --dry-run # 预览更改

Import existing agents

导入现有代理

elevenlabs agents pull
elevenlabs agents pull

List agents

列出代理

elevenlabs agents list
elevenlabs agents list

Check sync status

检查同步状态

elevenlabs agents status
elevenlabs agents status

Delete agent

删除代理

elevenlabs agents delete <agent_id>
undefined
elevenlabs agents delete <agent_id>
undefined

Tool Management Commands

工具管理命令

bash
undefined
bash
undefined

Create webhook tool

创建Webhook工具

elevenlabs tools add-webhook "Get Weather" --config-path tool_configs/get-weather.json
elevenlabs tools add-webhook "Get Weather" --config-path tool_configs/get-weather.json

Create client tool

创建客户端工具

elevenlabs tools add-client "Update Cart" --config-path tool_configs/update-cart.json
elevenlabs tools add-client "Update Cart" --config-path tool_configs/update-cart.json

Deploy tools

部署工具

elevenlabs tools push
elevenlabs tools push

Import existing tools

导入现有工具

elevenlabs tools pull
elevenlabs tools pull

Delete tools

删除工具

elevenlabs tools delete <tool_id> elevenlabs tools delete --all
undefined
elevenlabs tools delete <tool_id> elevenlabs tools delete --all
undefined

Testing Commands

测试命令

bash
undefined
bash
undefined

Create test

创建测试

elevenlabs tests add "Refund Test" --template basic-llm
elevenlabs tests add "Refund Test" --template basic-llm

Deploy tests

部署测试

elevenlabs tests push
elevenlabs tests push

Import tests

导入测试

elevenlabs tests pull
elevenlabs tests pull

Run test

运行测试

elevenlabs agents test "Support Agent"
undefined
elevenlabs agents test "Support Agent"
undefined

Multi-Environment Deployment

多环境部署

Pattern:
bash
undefined
模式:
bash
undefined

Development

开发环境

elevenlabs agents push --env dev
elevenlabs agents push --env dev

Staging

staging环境

elevenlabs agents push --env staging
elevenlabs agents push --env staging

Production (with confirmation)

生产环境(带确认)

elevenlabs agents push --env prod --dry-run
elevenlabs agents push --env prod --dry-run

Review changes...

查看更改...

elevenlabs agents push --env prod

**Environment-Specific Configs**:
agent_configs/ ├── support-bot.json # Base config ├── support-bot.dev.json # Dev overrides ├── support-bot.staging.json # Staging overrides └── support-bot.prod.json # Prod overrides
undefined
elevenlabs agents push --env prod

**环境特定配置**:
agent_configs/ ├── support-bot.json # 基础配置 ├── support-bot.dev.json # 开发环境覆盖 ├── support-bot.staging.json # staging环境覆盖 └── support-bot.prod.json # 生产环境覆盖
undefined

CI/CD Integration

CI/CD集成

GitHub Actions Example:
yaml
name: Deploy Agent
on:
  push:
    branches: [main]
    paths:
      - 'agent_configs/**'
      - 'tool_configs/**'

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Install CLI
        run: npm install -g @elevenlabs/cli

      - name: Test Configs
        run: elevenlabs agents push --dry-run --env prod
        env:
          ELEVENLABS_API_KEY: ${{ secrets.ELEVENLABS_API_KEY_PROD }}

      - name: Deploy
        run: elevenlabs agents push --env prod
        env:
          ELEVENLABS_API_KEY: ${{ secrets.ELEVENLABS_API_KEY_PROD }}
GitHub Actions示例:
yaml
name: Deploy Agent
on:
  push:
    branches: [main]
    paths:
      - 'agent_configs/**'
      - 'tool_configs/**'

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Install CLI
        run: npm install -g @elevenlabs/cli

      - name: Test Configs
        run: elevenlabs agents push --dry-run --env prod
        env:
          ELEVENLABS_API_KEY: ${{ secrets.ELEVENLABS_API_KEY_PROD }}

      - name: Deploy
        run: elevenlabs agents push --env prod
        env:
          ELEVENLABS_API_KEY: ${{ secrets.ELEVENLABS_API_KEY_PROD }}

Version Control Best Practices

版本控制最佳实践

Commit:
  • agent_configs/*.json
  • tool_configs/*.json
  • test_configs/*.json
  • agents.json
    ,
    tools.json
    ,
    tests.json
Ignore:
undefined
提交:
  • agent_configs/*.json
  • tool_configs/*.json
  • test_configs/*.json
  • agents.json
    ,
    tools.json
    ,
    tests.json
忽略:
undefined

.gitignore

.gitignore

.env .elevenlabs/ *.secret.json

---
.env .elevenlabs/ *.secret.json

---

13. Common Errors & Solutions

13. 常见错误与解决方案

Error 1: Missing Required Dynamic Variables

错误1:缺少必需的动态变量

Symptom: "Missing required dynamic variables" error, no transcript generated
Cause: Dynamic variables referenced in prompts/messages but not provided at conversation start
Solution:
typescript
const conversation = await client.conversations.create({
  agent_id: "agent_123",
  dynamic_variables: {
    user_name: "John",
    account_tier: "premium",
    // Provide ALL variables referenced in prompts
  }
});
症状: “缺少必需的动态变量”错误,无转录本生成
原因: 提示词/消息中引用了动态变量,但在对话开始时未提供
解决方案:
typescript
const conversation = await client.conversations.create({
  agent_id: "agent_123",
  dynamic_variables: {
    user_name: "John",
    account_tier: "premium",
    // 提供提示词中引用的所有变量
  }
});

Error 2: Case-Sensitive Tool Names

错误2:工具名称区分大小写

Symptom: Tool not executing, agent says "tool not found"
Cause: Tool name in config doesn't match registered name (case-sensitive)
Solution:
json
// agent_configs/bot.json
{
  "agent": {
    "prompt": {
      "tool_ids": ["orderLookup"]  // Must match exactly
    }
  }
}

// tool_configs/order-lookup.json
{
  "name": "orderLookup"  // Match case exactly
}
症状: 工具未执行,代理说“工具未找到”
原因: 配置中的工具名称与注册名称不匹配(区分大小写)
解决方案:
json
// agent_configs/bot.json
{
  "agent": {
    "prompt": {
      "tool_ids": ["orderLookup"]  // 必须完全匹配
    }
  }
}

// tool_configs/order-lookup.json
{
  "name": "orderLookup"  // 完全匹配大小写
}

Error 3: Webhook Authentication Failures

错误3:Webhook认证失败

Symptom: Webhook auto-disabled after failures
Cause:
  • Incorrect HMAC signature verification
  • Not returning 200 status code
  • 10+ consecutive failures
Solution:
typescript
// Always verify HMAC signature
import crypto from 'crypto';

const signature = req.headers['elevenlabs-signature'];
const payload = JSON.stringify(req.body);

const hmac = crypto
  .createHmac('sha256', process.env.WEBHOOK_SECRET)
  .update(payload)
  .digest('hex');

if (signature !== hmac) {
  return res.status(401).json({ error: 'Invalid signature' });
}

// Process webhook
// ...

// MUST return 200
res.status(200).json({ success: true });
症状: Webhook在失败后自动禁用
原因:
  • HMAC签名验证不正确
  • 未返回200状态码
  • 连续10+次失败
解决方案:
typescript
// 始终验证HMAC签名
import crypto from 'crypto';

const signature = req.headers['elevenlabs-signature'];
const payload = JSON.stringify(req.body);

const hmac = crypto
  .createHmac('sha256', process.env.WEBHOOK_SECRET)
  .update(payload)
  .digest('hex');

if (signature !== hmac) {
  return res.status(401).json({ error: '无效签名' });
}

// 处理Webhook
// ...

// 必须返回200
res.status(200).json({ success: true });

Error 4: Voice Consistency Issues

错误4:语音一致性问题

Symptom: Generated audio varies in volume/tone
Cause:
  • Background noise in voice clone training data
  • Inconsistent microphone distance
  • Whispering or shouting in samples
Solution:
  • Use clean audio samples (no music, noise, pops)
  • Maintain consistent microphone distance
  • Avoid extreme volumes
  • Test voice settings before deployment
症状: 生成的音频音量/语气不一致
原因:
  • 语音克隆训练数据中有背景噪音
  • 麦克风距离不一致
  • 样本中有低语或大喊
解决方案:
  • 使用干净的音频样本(无音乐、噪音、爆破音)
  • 保持一致的麦克风距离
  • 避免极端音量
  • 部署前测试语音设置

Error 5: Wrong Language Voice

错误5:语言与语音不匹配

Symptom: Unpredictable pronunciation, accent issues
Cause: Using English-trained voice for non-English language
Solution:
json
{
  "language_presets": [
    {
      "language": "es",
      "voice_id": "spanish_trained_voice_id"  // Must be Spanish-trained
    }
  ]
}
症状: 发音不可预测,口音问题
原因: 使用为英语训练的语音处理非英语语言
解决方案:
json
{
  "language_presets": [
    {
      "language": "es",
      "voice_id": "spanish_trained_voice_id"  // 必须是为西班牙语训练的语音
    }
  ]
}

Error 6: Restricted API Keys Not Supported (CLI)

错误6:受限制的API密钥不被支持(CLI)

Symptom: CLI authentication fails
Cause: Using restricted API key (not currently supported)
Solution: Use unrestricted API key for CLI operations
症状: CLI认证失败
原因: 使用受限制的API密钥(当前不支持)
解决方案: 使用无限制的API密钥进行CLI操作

Error 7: Agent Configuration Push Conflicts

错误7:代理配置推送冲突

Symptom: Changes not reflected after push
Cause: Hash-based change detection missed modification
Solution:
bash
undefined
症状: 推送后更改未反映
原因: 基于哈希的变更检测未检测到修改
解决方案:
bash
undefined

Force re-sync

强制重新同步

elevenlabs agents init --override elevenlabs agents pull # Re-import from platform
elevenlabs agents init --override elevenlabs agents pull # 从平台重新导入

Make changes

进行更改

elevenlabs agents push
undefined
elevenlabs agents push
undefined

Error 8: Tool Parameter Schema Mismatch

错误8:工具参数架构不匹配

Symptom: Tool called but parameters empty or incorrect
Cause: Schema definition doesn't match actual usage
Solution:
json
// tool_configs/order-lookup.json
{
  "parameters": {
    "type": "object",
    "properties": {
      "order_id": {
        "type": "string",
        "description": "The order ID to look up (format: ORD-12345)"  // Clear description
      }
    },
    "required": ["order_id"]
  }
}
症状: 工具被调用但参数为空或不正确
原因: 架构定义与实际使用不匹配
解决方案:
json
// tool_configs/order-lookup.json
{
  "parameters": {
    "type": "object",
    "properties": {
      "order_id": {
        "type": "string",
        "description": "要查询的订单ID(格式: ORD-12345)"  // 清晰的描述
      }
    },
    "required": ["order_id"]
  }
}

Error 9: RAG Index Not Ready

错误9:RAG索引未就绪

Symptom: Agent doesn't use knowledge base
Cause: RAG index still computing (can take minutes for large documents)
Solution:
typescript
// Check index status before using
const index = await client.knowledgeBase.getRagIndex({
  document_id: 'doc_123'
});

if (index.status !== 'ready') {
  console.log('Index still computing...');
}
症状: 代理未使用知识库
原因: RAG索引仍在计算中(大型文档可能需要几分钟)
解决方案:
typescript
// 使用前检查索引状态
const index = await client.knowledgeBase.getRagIndex({
  document_id: 'doc_123'
});

if (index.status !== 'ready') {
  console.log('索引仍在计算中...');
}

Error 10: WebSocket Protocol Error (1002)

错误10:WebSocket协议错误(1002)

Symptom: Intermittent "protocol error" when using WebSocket connections
Cause: Network instability or incompatible browser
Solution:
  • Use WebRTC instead of WebSocket (more resilient)
  • Implement reconnection logic
  • Check browser compatibility
症状: 使用WebSocket连接时间歇性出现“协议错误”
原因: 网络不稳定或浏览器不兼容
解决方案:
  • 改用WebRTC而非WebSocket(更具弹性)
  • 实现重连逻辑
  • 检查浏览器兼容性

Error 11: 401 Unauthorized in Production

错误11:生产环境中401未授权

Symptom: Works locally but fails in production
Cause: Agent visibility settings or API key configuration
Solution:
  • Check agent visibility (public vs private)
  • Verify API key is set in production environment
  • Check allowlist configuration if enabled
症状: 在本地工作但在生产环境中失败
原因: 代理可见性设置或API密钥配置
解决方案:
  • 检查代理可见性(公共 vs 私有)
  • 验证API密钥是否在生产环境中设置
  • 如果启用了允许列表,检查允许列表配置

Error 12: Allowlist Connection Errors

错误12:允许列表连接错误

Symptom: "Host elevenlabs.io is not allowed to connect to this agent"
Cause: Agent has allowlist enabled but using shared link
Solution:
  • Configure agent allowlist with correct domains
  • Or disable allowlist for testing
症状: “Host elevenlabs.io is not allowed to connect to this agent”
原因: 代理启用了允许列表但使用共享链接
解决方案:
  • 使用正确的域名配置代理允许列表
  • 或为测试禁用允许列表

Error 13: Workflow Infinite Loops

错误13:工作流无限循环

Symptom: Agent gets stuck in workflow, never completes
Cause: Edge conditions creating loops
Solution:
  • Add max iteration limits
  • Test all edge paths
  • Add explicit exit conditions
症状: 代理卡在工作流中,永远无法完成
原因: 边缘条件导致循环
解决方案:
  • 添加最大迭代限制
  • 测试所有边缘路径
  • 添加明确的退出条件

Error 14: Burst Pricing Not Enabled

错误14:未启用峰值定价

Symptom: Calls rejected during traffic spikes
Cause: Burst pricing not enabled in agent settings
Solution:
json
{
  "call_limits": {
    "burst_pricing_enabled": true
  }
}
症状: 流量峰值期间通话被拒绝
原因: 代理设置中未启用峰值定价
解决方案:
json
{
  "call_limits": {
    "burst_pricing_enabled": true
  }
}

Error 15: MCP Server Timeout

错误15:MCP服务器超时

Symptom: MCP tools not responding
Cause: MCP server slow or unreachable
Solution:
  • Check MCP server URL is accessible
  • Verify transport type (SSE vs HTTP)
  • Check authentication token
  • Monitor MCP server logs
症状: MCP工具无响应
原因: MCP服务器缓慢或不可达
解决方案:
  • 检查MCP服务器URL是否可访问
  • 验证传输类型(SSE vs HTTP)
  • 检查认证令牌
  • 监控MCP服务器日志

Error 16: First Message Cutoff on Android

错误16:Android上第一条消息被截断

Symptom: First message from agent gets cut off on Android devices (works fine on iOS/web)
Cause: Android devices need time to switch to correct audio mode after connection
Solution:
typescript
import { useConversation } from '@elevenlabs/react';

const { startConversation } = useConversation({
  agentId: 'your-agent-id',

  // Add connection delay for Android
  connectionDelay: {
    android: 3_000,  // 3 seconds (default)
    ios: 0,          // No delay needed
    default: 0       // Other platforms
  },

  // Rest of config...
});
Explanation:
  • Android needs 3 seconds to switch audio routing mode
  • Without delay, first audio chunk is lost
  • iOS and web don't have this issue
  • Adjust delay if 3 seconds isn't sufficient
Testing:
bash
undefined
症状: 代理的第一条消息在Android设备上被截断(在iOS/Web上正常工作)
原因: Android设备在连接后需要时间切换到正确的音频模式
解决方案:
typescript
import { useConversation } from '@elevenlabs/react';

const { startConversation } = useConversation({
  agentId: 'your-agent-id',

  // 为Android添加连接延迟
  connectionDelay: {
    android: 3_000,  // 3秒(默认)
    ios: 0,          // 无需延迟
    default: 0       // 其他平台
  },

  // 其余配置...
});
解释:
  • Android需要3秒切换音频路由模式
  • 无延迟的话,第一个音频片段会丢失
  • iOS和Web没有这个问题
  • 如果3秒不够,调整延迟
测试:
bash
undefined

Test on Android device

在Android设备上测试

npm run android
npm run android

First message should now be complete

现在第一条消息应该完整了

undefined
undefined

Error 17: CSP (Content Security Policy) Violations

错误17:CSP(内容安全策略)违规

Symptom: "Refused to load the script because it violates the following Content Security Policy directive" errors in browser console
Cause: Applications with strict Content Security Policy don't allow
data:
or
blob:
URLs in
script-src
directive. ElevenLabs SDK uses Audio Worklets that are loaded as blobs by default.
Solution - Self-Host Worklet Files:
Step 1: Copy worklet files to your public directory:
bash
undefined
症状: 浏览器控制台中出现“Refused to load the script because it violates the following Content Security Policy directive”错误
原因: 具有严格内容安全策略的应用不允许
data:
blob:
URL在
script-src
指令中。ElevenLabs SDK默认使用作为blob加载的Audio Worklet。
解决方案 - 自托管Worklet文件:
步骤1: 将worklet文件复制到您的公共目录:
bash
undefined

Copy from node_modules

从node_modules复制

cp node_modules/@elevenlabs/client/dist/worklets/*.js public/elevenlabs/

**Step 2**: Configure SDK to use self-hosted worklets:
```typescript
import { useConversation } from '@elevenlabs/react';

const { startConversation } = useConversation({
  agentId: 'your-agent-id',

  // Point to self-hosted worklet files
  workletPaths: {
    'rawAudioProcessor': '/elevenlabs/rawAudioProcessor.worklet.js',
    'audioConcatProcessor': '/elevenlabs/audioConcatProcessor.worklet.js',
  },

  // Rest of config...
});
Step 3: Update CSP headers to allow self-hosted scripts:
nginx
undefined
cp node_modules/@elevenlabs/client/dist/worklets/*.js public/elevenlabs/

**步骤2**: 配置SDK使用自托管的worklet文件:
```typescript
import { useConversation } from '@elevenlabs/react';

const { startConversation } = useConversation({
  agentId: 'your-agent-id',

  // 指向自托管的worklet文件
  workletPaths: {
    'rawAudioProcessor': '/elevenlabs/rawAudioProcessor.worklet.js',
    'audioConcatProcessor': '/elevenlabs/audioConcatProcessor.worklet.js',
  },

  // 其余配置...
});
步骤3: 更新CSP头以允许自托管脚本:
nginx
undefined

nginx example

nginx示例

add_header Content-Security-Policy " default-src 'self'; script-src 'self' https://elevenlabs.io; connect-src 'self' https://api.elevenlabs.io wss://api.elevenlabs.io; worker-src 'self'; " always;

**Worklet Files Location**:
node_modules/@elevenlabs/client/dist/worklets/ ├── rawAudioProcessor.worklet.js └── audioConcatProcessor.worklet.js

**Gotchas**:
- Worklet files must be served from same origin (CORS restriction)
- Update worklet files when upgrading `@elevenlabs/client`
- Paths must match exactly (case-sensitive)

**When You Need This**:
- Enterprise applications with strict CSP
- Government/financial apps
- Apps with security audits
- Any app blocking `blob:` URLs

---
add_header Content-Security-Policy " default-src 'self'; script-src 'self' https://elevenlabs.io; connect-src 'self' https://api.elevenlabs.io wss://api.elevenlabs.io; worker-src 'self'; " always;

**Worklet文件位置**:
node_modules/@elevenlabs/client/dist/worklets/ ├── rawAudioProcessor.worklet.js └── audioConcatProcessor.worklet.js

**注意事项**:
- Worklet文件必须从同一源提供(CORS限制)
- 升级`@elevenlabs/client`时更新worklet文件
- 路径必须完全匹配(区分大小写)

**何时需要此方案**:
- 具有严格CSP的企业应用
- 政府/金融应用
- 有安全审计的应用
- 任何阻止`blob:`URL的应用

---

Integration with Existing Skills

与现有技能集成

This skill composes well with:
  • cloudflare-worker-base → Deploy agents on Cloudflare Workers edge network
  • cloudflare-workers-ai → Use Cloudflare LLMs as custom models in agents
  • cloudflare-durable-objects → Persistent conversation state and session management
  • cloudflare-kv → Cache agent configurations and user preferences
  • nextjs → React SDK integration in Next.js applications
  • ai-sdk-core → Vercel AI SDK provider for unified AI interface
  • clerk-auth → Authenticated voice sessions with user identity
  • hono-routing → API routes for webhooks and server tools

本技能可与以下技能很好地组合:
  • cloudflare-worker-base → 在Cloudflare Workers边缘网络部署代理
  • cloudflare-workers-ai → 使用Cloudflare LLM作为代理中的自定义模型
  • cloudflare-durable-objects → 持久对话状态与会话管理
  • cloudflare-kv → 缓存代理配置与用户偏好
  • nextjs → Next.js应用中的React SDK集成
  • ai-sdk-core → Vercel AI SDK提供统一AI接口
  • clerk-auth → 带用户身份的认证语音会话
  • hono-routing → Webhook与服务器工具的API路由

Additional Resources

其他资源

Official Documentation:
Examples:
Community:

Production Tested: WordPress Auditor, Customer Support Agents Last Updated: 2025-11-03 Package Versions: elevenlabs@1.59.0, @elevenlabs/cli@0.2.0
官方文档:
示例:
社区:

生产环境测试: WordPress审核员、客户支持代理 最后更新: 2025-11-03 包版本: elevenlabs@1.59.0, @elevenlabs/cli@0.2.0