ag2-live

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Realtime voice / live audio agents

实时语音/直播音频代理

When to use

适用场景

  • The user wants a talking agent: speak into a mic, the agent replies with voice, hands-free, low latency (Gemini Live or OpenAI Realtime).
  • They want live transcription of the user's speech while the agent talks.
  • They want a voice pipeline over an existing text
    Agent
    : transcribe a recorded clip → run the agent → (optionally) speak the reply.
  • They want to add spoken output (TTS) to an otherwise text-only streaming
    Agent
    .
If the user only needs to send a recorded audio file into an agent as one input (not a live session), use
ag2-multimodal-input
(
AudioInput
) instead.
Hardware / keys caveat. A real session needs (a) a provider API key and (b) a working microphone + speaker. Neither is available headless. Everything in this skill constructs without hardware; the parts that actually open a socket or audio device are marked [needs keys + audio] below.
  • 用户需要对话代理:对着麦克风说话,代理以语音回复,支持免提、低延迟(基于Gemini Live或OpenAI Realtime)。
  • 用户需要在代理说话的同时实时转写自己的语音。
  • 用户需要为现有文本
    Agent
    搭建语音流水线:转写录制片段 → 运行代理 →(可选)语音回复。
  • 用户需要为纯文本流式
    Agent
    添加语音输出(TTS)
如果用户仅需将录制的音频文件作为单次输入发送给代理(而非实时会话),请使用
ag2-multimodal-input
AudioInput
)。
硬件/密钥注意事项。实时会话需要:(a) 服务商API密钥;(b) 可用的麦克风和扬声器。无外设环境下无法使用。本技能的所有功能均可离线构建;实际打开套接字或音频设备的部分标记为**[需要密钥+音频设备]**。

Installation

安装

The live module splits across optional extras — install the ones you need:
bash
undefined
直播模块拆分为可选扩展包,请按需安装:
bash
undefined

OpenAI Realtime + OpenAI TTS/STT

OpenAI Realtime + OpenAI TTS/STT

pip install "ag2[openai-realtime]"
pip install "ag2[openai-realtime]"

Gemini Live

Gemini Live

pip install "ag2[gemini-realtime]"
pip install "ag2[gemini-realtime]"

Local microphone / speaker I/O (SoundDeviceRecorder / SoundDevicePlayer)

本地麦克风/扬声器输入输出(SoundDeviceRecorder / SoundDevicePlayer)

pip install "sounddevice[numpy]"

A typical OpenAI voice app on the local sound card: `pip install "ag2[openai-realtime]" "sounddevice[numpy]"`.

> Required. Run the relevant install before delivering code. Without the matching extra, the public symbol resolves to a placeholder that raises `ImportError` (`openai`/`gemini`) or the additional-dependency error (`sounddevice[numpy]`) the moment you use it.
pip install "sounddevice[numpy]"

本地声卡上运行的典型OpenAI语音应用:`pip install "ag2[openai-realtime]" "sounddevice[numpy]"`。

> 必须操作。在运行代码前完成对应安装。若未安装匹配的扩展包,公共符号会解析为占位符,在首次使用时触发`ImportError`(针对`openai`/`gemini`)或额外依赖错误(针对`sounddevice[numpy]`)。

Public API

公开API

All exported from
ag2.live
:
SymbolRole
LiveAgent
Wraps a prompt + realtime config;
agent.run()
opens the bidirectional session
GeminiRealTimeConfig
Gemini Live realtime provider config
OpenAIRealTimeConfig
OpenAI Realtime provider config
OpenAITTSConfig
Text → speech (PCM bytes)
OpenAITranscriber
One-shot speech → text (transcription)
OpenAITranslationTranscriber
One-shot speech → English text (translation)
SoundDeviceRecorder
Mic capture →
RecordedAudioEvent
on the stream
SoundDevicePlayer
Plays
SynthesizedAudioEvent
PCM out the speaker
TTSObserver
Observer that speaks a text
Agent
's streamed tokens via a TTS config
所有API均从
ag2.live
导出:
符号作用
LiveAgent
封装提示词与实时配置;
agent.run()
开启双向会话
GeminiRealTimeConfig
Gemini Live实时服务商配置
OpenAIRealTimeConfig
OpenAI Realtime实时服务商配置
OpenAITTSConfig
文本转语音(PCM字节)
OpenAITranscriber
一次性语音转文本(转写)
OpenAITranslationTranscriber
一次性语音转英文文本(翻译)
SoundDeviceRecorder
麦克风捕获 → 流中生成
RecordedAudioEvent
SoundDevicePlayer
SynthesizedAudioEvent
的PCM数据播放至扬声器
TTSObserver
观察者,通过TTS配置将文本
Agent
的流式令牌转换为语音播放

How a live session works

实时会话工作原理

LiveAgent
is built around an event stream (a
ConversationContext
). The recorder, the provider session, and the player all share that one context:
mic ──SoundDeviceRecorder──▶ RecordedAudioEvent ──▶ provider session (Gemini/OpenAI)
                                       SynthesizedAudioEvent (assistant audio)
                                              SoundDevicePlayer ──▶ speaker
Along the way the provider also emits
TranscriptionChunkEvent
/
TranscriptionCompletedEvent
(your speech),
ModelMessageChunk
(assistant text),
ToolCallEvent
/
ToolResultEvent
, and
UsageEvent
.
agent.run()
is an async context manager that yields the shared
ConversationContext
so you can attach the recorder and player to it.
LiveAgent
围绕事件
ConversationContext
)构建。录制器、服务商会话和播放器共享同一个上下文:
麦克风 ──SoundDeviceRecorder──▶ RecordedAudioEvent ──▶ 服务商会话(Gemini/OpenAI)
                                       SynthesizedAudioEvent(助手音频)
                                              SoundDevicePlayer ──▶ 扬声器
过程中服务商还会输出
TranscriptionChunkEvent
/
TranscriptionCompletedEvent
(用户语音转写)、
ModelMessageChunk
(助手文本)、
ToolCallEvent
/
ToolResultEvent
以及
UsageEvent
agent.run()
是异步上下文管理器,会返回共享的
ConversationContext
,以便将录制器和播放器绑定到该上下文。

Minimal recipe — full duplex voice loop [needs keys + audio]

最简示例——全双工语音循环 [需要密钥+音频设备]

python
import asyncio

from ag2.live import (
    LiveAgent,
    OpenAIRealTimeConfig,
    SoundDevicePlayer,
    SoundDeviceRecorder,
)
from ag2.live.openai import AudioOutput, InputConfig

agent = LiveAgent(
    "voice_bot",
    "You are a friendly realtime voice assistant. Keep replies short and conversational.",
    config=OpenAIRealTimeConfig(
        "gpt-realtime",
        output=AudioOutput(voice="marin"),
        # Transcribe the user's speech so we can see it too.
        input=InputConfig(transcription={"model": "gpt-4o-mini-transcribe"}),
    ),
)

async def main() -> None:
    # run() yields the shared ConversationContext.
    async with agent.run() as ctx:
        # Recorder and player bind to the SAME context so audio flows on one bus.
        async with (
            SoundDeviceRecorder(context=ctx),   # mic  -> RecordedAudioEvent
            SoundDevicePlayer(context=ctx),     # SynthesizedAudioEvent -> speaker
        ):
            print("Talk to the agent. Ctrl-C to stop.")
            await asyncio.Event().wait()  # keep the session open

if __name__ == "__main__":
    asyncio.run(main())
Set
OPENAI_API_KEY
(or pass
client=AsyncOpenAI(...)
). For Gemini, set
GEMINI_API_KEY
/
GOOGLE_API_KEY
(or pass
client=
).
Swap to Gemini by changing only the config:
python
from ag2.live import GeminiRealTimeConfig
from ag2.live.gemini import AudioOutput, InputConfig

agent = LiveAgent(
    "voice_bot",
    "You are a friendly realtime voice assistant.",
    config=GeminiRealTimeConfig(
        "gemini-2.5-flash-native-audio-preview-12-2025",
        output=AudioOutput(voice="Kore"),
        input=InputConfig(transcribe=True),   # user-speech transcription
    ),
)
python
import asyncio

from ag2.live import (
    LiveAgent,
    OpenAIRealTimeConfig,
    SoundDevicePlayer,
    SoundDeviceRecorder,
)
from ag2.live.openai import AudioOutput, InputConfig

agent = LiveAgent(
    "voice_bot",
    "You are a friendly realtime voice assistant. Keep replies short and conversational.",
    config=OpenAIRealTimeConfig(
        "gpt-realtime",
        output=AudioOutput(voice="marin"),
        # Transcribe the user's speech so we can see it too.
        input=InputConfig(transcription={"model": "gpt-4o-mini-transcribe"}),
    ),
)

async def main() -> None:
    # run() yields the shared ConversationContext.
    async with agent.run() as ctx:
        # Recorder and player bind to the SAME context so audio flows on one bus.
        async with (
            SoundDeviceRecorder(context=ctx),   # mic  -> RecordedAudioEvent
            SoundDevicePlayer(context=ctx),     # SynthesizedAudioEvent -> speaker
        ):
            print("Talk to the agent. Ctrl-C to stop.")
            await asyncio.Event().wait()  # keep the session open

if __name__ == "__main__":
    asyncio.run(main())
设置
OPENAI_API_KEY
(或传入
client=AsyncOpenAI(...)
)。使用Gemini时,设置
GEMINI_API_KEY
/
GOOGLE_API_KEY
(或传入
client=
)。
只需修改配置即可切换为Gemini:
python
from ag2.live import GeminiRealTimeConfig
from ag2.live.gemini import AudioOutput, InputConfig

agent = LiveAgent(
    "voice_bot",
    "You are a friendly realtime voice assistant.",
    config=GeminiRealTimeConfig(
        "gemini-2.5-flash-native-audio-preview-12-2025",
        output=AudioOutput(voice="Kore"),
        input=InputConfig(transcribe=True),   # user-speech transcription
    ),
)

Provider matrix — Gemini Live vs OpenAI Realtime

服务商对比——Gemini Live vs OpenAI Realtime

Gemini (
GeminiRealTimeConfig
)
OpenAI (
OpenAIRealTimeConfig
)
Import the knob types from
ag2.live.gemini
ag2.live.openai
Output modes
AudioOutput
(default) /
TextOutput
AudioOutput
(default) /
TextOutput
Voices (
AudioOutput(voice=...)
)
Aoede
,
Charon
,
Fenrir
,
Kore
(default),
Leda
,
Orus
,
Puck
,
Zephyr
alloy
(default),
ash
,
ballad
,
coral
,
echo
,
sage
,
shimmer
,
verse
,
marin
,
cedar
Example models (first positional arg)
gemini-2.5-flash-native-audio-preview-12-2025
,
gemini-3.1-flash-live-preview
,
gemini-live-2.5-flash-preview
,
gemini-2.0-flash-live-001
gpt-realtime
,
gpt-realtime-mini
,
gpt-audio-1.5
,
gpt-4o-realtime-preview-2024-10-01
User-speech transcription
InputConfig(transcribe=True, transcription_languages=[...])
InputConfig(transcription={"model": "gpt-4o-mini-transcribe"})
Turn detection / VAD
InputConfig(automatic_activity_detection=..., activity_handling=..., turn_coverage=...)
InputConfig(turn_detection={"type": "semantic_vad", ...})
(default on)
Audio I/O sample ratein 16 kHz / out 24 kHz (fixed by API)configurable via
AudioOutput.format
/
InputConfig.format
(default PCM 24 kHz)
Generation knobs
temperature=
,
max_output_tokens=
max_output_tokens=
(int or
"inf"
),
tool_choice=
,
tracing=
Escape hatch for raw provider config
config=<LiveConnectConfigDict>
session=<RealtimeSessionCreateRequestParam>
Bring your own client
client=google.genai.Client(...)
client=openai.AsyncOpenAI(...)
The model name is a positional arg; everything else is keyword-only. The
Literal
model/voice lists above are accepted, but any string is allowed too (forward-compatible with new models).
AudioOutput
adds
language_code=
(Gemini) and
speed=
+
format=
(OpenAI).
TextOutput()
takes no args — it returns text only (
ModelMessageChunk
), no audio playback.
Gemini(
GeminiRealTimeConfig
OpenAI(
OpenAIRealTimeConfig
配置类型导入自
ag2.live.gemini
ag2.live.openai
输出模式
AudioOutput
(默认)/
TextOutput
AudioOutput
(默认)/
TextOutput
语音选项(
AudioOutput(voice=...)
Aoede
,
Charon
,
Fenrir
,
Kore
(默认),
Leda
,
Orus
,
Puck
,
Zephyr
alloy
(默认),
ash
,
ballad
,
coral
,
echo
,
sage
,
shimmer
,
verse
,
marin
,
cedar
示例模型(第一个位置参数)
gemini-2.5-flash-native-audio-preview-12-2025
,
gemini-3.1-flash-live-preview
,
gemini-live-2.5-flash-preview
,
gemini-2.0-flash-live-001
gpt-realtime
,
gpt-realtime-mini
,
gpt-audio-1.5
,
gpt-4o-realtime-preview-2024-10-01
用户语音转写
InputConfig(transcribe=True, transcription_languages=[...])
InputConfig(transcription={"model": "gpt-4o-mini-transcribe"})
说话检测/VAD
InputConfig(automatic_activity_detection=..., activity_handling=..., turn_coverage=...)
InputConfig(turn_detection={"type": "semantic_vad", ...})
(默认开启)
音频输入输出采样率输入16 kHz / 输出24 kHz(API固定)可通过
AudioOutput.format
/
InputConfig.format
配置(默认PCM 24 kHz)
生成控制参数
temperature=
,
max_output_tokens=
max_output_tokens=
(整数或
"inf"
),
tool_choice=
,
tracing=
原始服务商配置入口
config=<LiveConnectConfigDict>
session=<RealtimeSessionCreateRequestParam>
自定义客户端
client=google.genai.Client(...)
client=openai.AsyncOpenAI(...)
模型名称为位置参数;其余均为关键字参数。上述
Literal
类型的模型/语音列表均可直接使用,同时也支持任意字符串(兼容未来新增模型)。
AudioOutput
支持
language_code=
(Gemini)和
speed=
+
format=
(OpenAI)。
TextOutput()
无参数——仅返回文本(
ModelMessageChunk
),无音频播放。

Tools, HITL, observers on a
LiveAgent

LiveAgent的工具、人工介入、观察者

LiveAgent(...)
accepts the same surface as a normal
Agent
:
tools=
,
hitl_hook=
,
middleware=
,
observers=
,
dependencies=
,
variables=
,
plugins=
. Tool calls from the model arrive as
ToolCallEvent
and results are forwarded back into the session automatically.
run()
can override
config=
,
prompt=
,
tools=
,
observers=
,
hitl_hook=
per session.
Note:
LiveAgent
only supports function tools over realtime — provider server-side tool types raise
NotImplementedError
in both backends.
LiveAgent(...)
支持与普通
Agent
相同的参数:
tools=
,
hitl_hook=
,
middleware=
,
observers=
,
dependencies=
,
variables=
,
plugins=
。模型生成的工具调用会以
ToolCallEvent
形式返回,结果会自动转发回会话。
run()
可针对单次会话覆盖
config=
,
prompt=
,
tools=
,
observers=
,
hitl_hook=
参数。
注意
LiveAgent
仅支持实时会话中的函数工具——服务商端工具类型在两个后端均会触发
NotImplementedError

Audio I/O —
SoundDeviceRecorder
/
SoundDevicePlayer

音频输入输出——SoundDeviceRecorder / SoundDevicePlayer

Both are async context managers (
async with
) and bind to a
ConversationContext
via
context=
. The device opens on
__aenter__
[needs audio]; construction alone touches no hardware.
python
from ag2.live import SoundDeviceRecorder, SoundDevicePlayer
两者均为异步上下文管理器(
async with
),通过
context=
绑定到
ConversationContext
。设备会在
__aenter__
时打开**[需要音频设备]**;仅构建实例不会操作硬件。
python
from ag2.live import SoundDeviceRecorder, SoundDevicePlayer

Recorder: mic -> RecordedAudioEvent on the stream.

录制器:麦克风 → 流中生成RecordedAudioEvent

recorder = SoundDeviceRecorder(context=ctx, sample_rate=16000, channels=1) # block_size optional
recorder = SoundDeviceRecorder(context=ctx, sample_rate=16000, channels=1) # block_size可选

Player: subscribes to SynthesizedAudioEvent, writes PCM to the speaker (24 kHz mono).

播放器:订阅SynthesizedAudioEvent,将PCM数据输出到扬声器(24 kHz单声道)

player = SoundDevicePlayer(context=ctx)

`SoundDeviceRecorder` also has a **one-shot blocking** `record(duration)` that returns a `VoiceInput` (16-bit PCM) — handy to feed the STT pipeline below without running a live session:

```python
voice = recorder.record(duration=4.0)   # blocks 4s, returns VoiceInput  [needs audio]
player = SoundDevicePlayer(context=ctx)

`SoundDeviceRecorder`还提供**一次性阻塞式**`record(duration)`方法,返回`VoiceInput`(16位PCM)——无需运行实时会话即可为下文的STT流水线提供输入:

```python
voice = recorder.record(duration=4.0)   # 阻塞4秒,返回VoiceInput  [需要音频设备]

One-shot speech-to-text + voice pipeline [needs keys; audio only if recording live]

一次性语音转文本+语音流水线 [需要密钥;录制实时音频时需音频设备]

OpenAITranscriber
(transcription) and
OpenAITranslationTranscriber
(translate to English) take a
VoiceInput
and return text.
.pipe(agent)
builds a
VoicePipeline
that transcribes then runs a normal text
Agent
:
python
from ag2 import Agent
from ag2.config import OpenAIConfig
from ag2.live import OpenAITranscriber, SoundDeviceRecorder
from ag2.live.stt import VoiceInput

agent = Agent("assistant", "You answer questions.", config=OpenAIConfig(model="gpt-4o-mini"))
pipeline = OpenAITranscriber("gpt-4o-transcribe").pipe(agent)
OpenAITranscriber
(转写)和
OpenAITranslationTranscriber
(翻译为英文)接收
VoiceInput
并返回文本。
.pipe(agent)
可构建
VoicePipeline
,实现转写后运行普通文本
Agent
python
from ag2 import Agent
from ag2.config import OpenAIConfig
from ag2.live import OpenAITranscriber, SoundDeviceRecorder
from ag2.live.stt import VoiceInput

agent = Agent("assistant", "You answer questions.", config=OpenAIConfig(model="gpt-4o-mini"))
pipeline = OpenAITranscriber("gpt-4o-transcribe").pipe(agent)

Record a clip, transcribe it, and ask the agent — in one call. [needs keys + audio]

录制片段、转写并询问代理——一步完成 [需要密钥+音频设备]

voice: VoiceInput = SoundDeviceRecorder().record(4.0) reply = await pipeline.ask(voice) print(await reply.content())
voice: VoiceInput = SoundDeviceRecorder().record(4.0) reply = await pipeline.ask(voice) print(await reply.content())

Continue the conversation with another clip:

录制另一个片段继续对话:

followup = await reply.ask(SoundDeviceRecorder().record(4.0))

`VoiceInput(content: bytes, frame_rate: int, channels: int)` wraps 16-bit PCM; build it directly from any PCM source if you aren't using the recorder.

`OpenAITranslationTranscriber` is identical but always outputs English (useful for non-English speech in → English text out).
followup = await reply.ask(SoundDeviceRecorder().record(4.0))

`VoiceInput(content: bytes, frame_rate: int, channels: int)`封装16位PCM数据;若不使用录制器,可直接从任意PCM源构建该对象。

`OpenAITranslationTranscriber`功能相同,但始终输出英文(适用于非英文语音输入转英文文本输出的场景)。

Text-to-speech —
OpenAITTSConfig
and
TTSObserver

文本转语音——OpenAITTSConfig和TTSObserver

OpenAITTSConfig
synthesizes text into PCM bytes:
python
from ag2.live import OpenAITTSConfig

tts = OpenAITTSConfig("gpt-4o-mini-tts", voice="alloy", speed=1.0)
pcm: bytes = await tts.synthesize("Hello there!")   # [needs keys]
TTSObserver
turns any text streaming
Agent
into a talking one: attach it as an observer and it accumulates
ModelMessageChunk
tokens, synthesizes complete sentences, and emits
SynthesizedAudioEvent
— which a
SoundDevicePlayer
on the same stream plays aloud. This is the bridge between a normal text agent and live audio output (no realtime provider needed).
python
from ag2 import Agent, MemoryStream
from ag2.config import OpenAIConfig
from ag2.context import ConversationContext
from ag2.live import OpenAITTSConfig, SoundDevicePlayer, TTSObserver

agent = Agent(
    "narrator",
    "You are a helpful assistant.",
    config=OpenAIConfig(model="gpt-4o-mini"),
    observers=[TTSObserver(OpenAITTSConfig("gpt-4o-mini-tts", voice="nova"))],
)
OpenAITTSConfig
将文本合成为PCM字节:
python
from ag2.live import OpenAITTSConfig

tts = OpenAITTSConfig("gpt-4o-mini-tts", voice="alloy", speed=1.0)
pcm: bytes = await tts.synthesize("Hello there!")   # [需要密钥]
TTSObserver
可将任意文本流式
Agent
转换为语音代理:将其作为观察者附加后,会累积
ModelMessageChunk
令牌,合成完整句子并输出
SynthesizedAudioEvent
——同一流中的
SoundDevicePlayer
会播放该音频。这是普通文本代理与实时音频输出之间的桥梁(无需实时服务商)。
python
from ag2 import Agent, MemoryStream
from ag2.config import OpenAIConfig
from ag2.context import ConversationContext
from ag2.live import OpenAITTSConfig, SoundDevicePlayer, TTSObserver

agent = Agent(
    "narrator",
    "You are a helpful assistant.",
    config=OpenAIConfig(model="gpt-4o-mini"),
    observers=[TTSObserver(OpenAITTSConfig("gpt-4o-mini-tts", voice="nova"))],
)

Agent and Player must share ONE stream: agent.ask(stream=...) and

Agent和Player必须共享同一个流:agent.ask(stream=...)和

SoundDevicePlayer(context=...) whose .stream is that same stream.

SoundDevicePlayer(context=...)的.stream必须为同一个流

stream = MemoryStream() ctx = ConversationContext(stream=stream)
async with SoundDevicePlayer(context=ctx): # plays the synthesized speech [needs audio] reply = await agent.ask("Tell me a one-line joke.", stream=stream) print(await reply.content())

`TTSObserver(config)` returns a `CompositeObserver`. It flushes any remaining buffered text on the final `ModelMessage`, so trailing partial sentences are still spoken.
stream = MemoryStream() ctx = ConversationContext(stream=stream)
async with SoundDevicePlayer(context=ctx): # 播放合成语音 [需要音频设备] reply = await agent.ask("Tell me a one-line joke.", stream=stream) print(await reply.content())

`TTSObserver(config)`返回`CompositeObserver`。它会在最终`ModelMessage`时刷新所有缓冲文本,确保末尾的不完整句子也能被播放。

Usage reporting

使用报告

await LiveAgent.usage_report(ctx)
aggregates token usage over the live session's event log into a
UsageReport
.
await LiveAgent.usage_report(ctx)
会将实时会话事件日志中的令牌使用情况汇总为
UsageReport

What is construction-tested vs needs live keys + hardware

可离线构建测试 vs 需要实时密钥+硬件的功能

Run by this skill's
references/test_samples.py
(all green) — constructed / exercised offline:
  • All 9 public symbols import as real classes (not missing-dependency placeholders).
  • GeminiRealTimeConfig
    /
    OpenAIRealTimeConfig
    construction with
    AudioOutput
    /
    TextOutput
    /
    InputConfig
    , voices, temperature/max-tokens, and
    _build_session(instructions=...)
    merge.
  • OpenAITTSConfig
    ,
    OpenAITranscriber
    ,
    OpenAITranslationTranscriber
    construction.
  • LiveAgent
    construction (string prompt, list prompt, with
    tools=
    /
    observers=
    ).
  • SoundDeviceRecorder
    /
    SoundDevicePlayer
    construction (no device opened).
  • TTSObserver
    returns a
    CompositeObserver
    .
  • OpenAITranscriber(...).pipe(agent)
    VoicePipeline
    ;
    VoiceInput
    dataclass.
Requires real API keys + microphone/speaker (NOT runnable headless):
  • agent.run()
    opening a live websocket to Gemini/OpenAI.
  • async with SoundDeviceRecorder(...)
    /
    SoundDevicePlayer(...)
    (opens the sound card on
    __aenter__
    ).
  • SoundDeviceRecorder.record(duration)
    (blocks on the mic).
  • OpenAITTSConfig.synthesize(...)
    ,
    OpenAITranscriber.transcribe(...)
    , and
    VoicePipeline.ask(...)
    (hit the OpenAI API).
本技能的
references/test_samples.py
可运行以下离线构建/测试功能(全部通过):
  • 所有9个公开符号均可导入为真实类(而非缺失依赖的占位符)。
  • GeminiRealTimeConfig
    /
    OpenAIRealTimeConfig
    的构建,包括
    AudioOutput
    /
    TextOutput
    /
    InputConfig
    、语音选项、温度/最大令牌数,以及
    _build_session(instructions=...)
    合并逻辑。
  • OpenAITTSConfig
    OpenAITranscriber
    OpenAITranslationTranscriber
    的构建。
  • LiveAgent
    的构建(字符串提示词、列表提示词,包含
    tools=
    /
    observers=
    )。
  • SoundDeviceRecorder
    /
    SoundDevicePlayer
    的构建(不打开设备)。
  • TTSObserver
    返回
    CompositeObserver
  • OpenAITranscriber(...).pipe(agent)
    生成
    VoicePipeline
    VoiceInput
    数据类。
需要真实API密钥+麦克风/扬声器(无外设环境无法运行)
  • agent.run()
    打开与Gemini/OpenAI的实时WebSocket连接。
  • async with SoundDeviceRecorder(...)
    /
    SoundDevicePlayer(...)
    (在
    __aenter__
    时打开声卡)。
  • SoundDeviceRecorder.record(duration)
    (阻塞等待麦克风输入)。
  • OpenAITTSConfig.synthesize(...)
    OpenAITranscriber.transcribe(...)
    以及
    VoicePipeline.ask(...)
    (调用OpenAI API)。

Common pitfalls

常见陷阱

  • Wrong extra installed.
    pip install "ag2[openai-realtime]"
    for OpenAI,
    "ag2[gemini-realtime]"
    for Gemini,
    "sounddevice[numpy]"
    for local audio. The symbol imports fine but raises on first use otherwise.
  • numpy
    missing.
    SoundDeviceRecorder
    /
    SoundDevicePlayer
    need numpy —
    sounddevice[numpy]
    pulls it in. Without it you get an additional-dependency
    ImportError
    .
  • Importing knob types from the wrong module.
    AudioOutput
    /
    TextOutput
    /
    InputConfig
    are provider-specific:
    ag2.live.gemini
    vs
    ag2.live.openai
    . They are not re-exported from
    ag2.live
    .
  • Recorder, player, and
    run()
    not sharing one context.
    Pass the
    ctx
    yielded by
    agent.run()
    into
    SoundDeviceRecorder(context=ctx)
    /
    SoundDevicePlayer(context=ctx)
    , or audio events won't reach the session/speaker.
  • Voice not in the provider's list. Gemini and OpenAI have different voice names (see matrix). A bare string is accepted but an unknown voice is rejected by the provider at session open.
  • Expecting server-side / provider tools over realtime. Only function tools are supported; other tool types raise
    NotImplementedError
    .
  • Forgetting to keep the loop alive.
    agent.run()
    is a context manager — exit closes the session. Keep it open (
    await asyncio.Event().wait()
    or your own loop) for a continuous conversation.
  • 安装了错误的扩展包:OpenAI需安装
    pip install "ag2[openai-realtime]"
    ,Gemini需安装
    "ag2[gemini-realtime]"
    ,本地音频需安装
    "sounddevice[numpy]"
    。符号可正常导入,但首次使用时会触发错误。
  • 缺失numpy
    SoundDeviceRecorder
    /
    SoundDevicePlayer
    需要numpy——
    sounddevice[numpy]
    会自动安装该依赖。若无numpy,会触发额外依赖
    ImportError
  • 从错误模块导入配置类型
    AudioOutput
    /
    TextOutput
    /
    InputConfig
    是服务商专属的:
    ag2.live.gemini
    vs
    ag2.live.openai
    。它们不会从
    ag2.live
    重新导出。
  • 录制器、播放器和
    run()
    未共享同一上下文
    :需将
    agent.run()
    返回的
    ctx
    传入
    SoundDeviceRecorder(context=ctx)
    /
    SoundDevicePlayer(context=ctx)
    ,否则音频事件无法到达会话/扬声器。
  • 使用了服务商不支持的语音:Gemini和OpenAI的语音名称不同(见对比表)。虽然接受任意字符串,但未知语音会在会话开启时被服务商拒绝。
  • 期望实时会话支持服务商端工具:仅支持函数工具;其他工具类型会触发
    NotImplementedError
  • 忘记保持会话活跃
    agent.run()
    是上下文管理器——退出会关闭会话。需保持会话开启(
    await asyncio.Event().wait()
    或自定义循环)以实现连续对话。

Going deeper

深入了解

  • Source:
    ag2/live/{realtime.py,gemini.py,openai.py,protocols.py,observer.py,stt.py,sound_device.py}
    .
  • realtime.py
    LiveAgent
    + the
    RealtimeConfig
    protocol.
  • For sending a recorded audio file as a one-off input to a text agent, see
    ag2-multimodal-input
    (
    AudioInput
    ).
  • For observers in general (token monitors, loop detection), see
    ag2-observers-and-alerts
    .
  • 源码:
    ag2/live/{realtime.py,gemini.py,openai.py,protocols.py,observer.py,stt.py,sound_device.py}
  • realtime.py
    ——
    LiveAgent
    RealtimeConfig
    协议。
  • 如需将录制的音频文件作为单次输入发送给文本代理,请查看
    ag2-multimodal-input
    AudioInput
    )。
  • 如需了解通用观察者(令牌监控、循环检测),请查看
    ag2-observers-and-alerts