ag2-live
Compare original and translation side by side
🇺🇸
Original
English🇨🇳
Translation
ChineseRealtime voice / live audio agents
实时语音/直播音频代理
When to use
适用场景
- The user wants a talking agent: speak into a mic, the agent replies with voice, hands-free, low latency (Gemini Live or OpenAI Realtime).
- They want live transcription of the user's speech while the agent talks.
- They want a voice pipeline over an existing text : transcribe a recorded clip → run the agent → (optionally) speak the reply.
Agent - They want to add spoken output (TTS) to an otherwise text-only streaming .
Agent
If the user only needs to send a recorded audio file into an agent as one input (not a live session), use () instead.
ag2-multimodal-inputAudioInputHardware / keys caveat. A real session needs (a) a provider API key and (b) a working microphone + speaker. Neither is available headless. Everything in this skill constructs without hardware; the parts that actually open a socket or audio device are marked [needs keys + audio] below.
- 用户需要对话代理:对着麦克风说话,代理以语音回复,支持免提、低延迟(基于Gemini Live或OpenAI Realtime)。
- 用户需要在代理说话的同时实时转写自己的语音。
- 用户需要为现有文本搭建语音流水线:转写录制片段 → 运行代理 →(可选)语音回复。
Agent - 用户需要为纯文本流式添加语音输出(TTS)。
Agent
如果用户仅需将录制的音频文件作为单次输入发送给代理(而非实时会话),请使用()。
ag2-multimodal-inputAudioInput硬件/密钥注意事项。实时会话需要:(a) 服务商API密钥;(b) 可用的麦克风和扬声器。无外设环境下无法使用。本技能的所有功能均可离线构建;实际打开套接字或音频设备的部分标记为**[需要密钥+音频设备]**。
Installation
安装
The live module splits across optional extras — install the ones you need:
bash
undefined直播模块拆分为可选扩展包,请按需安装:
bash
undefinedOpenAI Realtime + OpenAI TTS/STT
OpenAI Realtime + OpenAI TTS/STT
pip install "ag2[openai-realtime]"
pip install "ag2[openai-realtime]"
Gemini Live
Gemini Live
pip install "ag2[gemini-realtime]"
pip install "ag2[gemini-realtime]"
Local microphone / speaker I/O (SoundDeviceRecorder / SoundDevicePlayer)
本地麦克风/扬声器输入输出(SoundDeviceRecorder / SoundDevicePlayer)
pip install "sounddevice[numpy]"
A typical OpenAI voice app on the local sound card: `pip install "ag2[openai-realtime]" "sounddevice[numpy]"`.
> Required. Run the relevant install before delivering code. Without the matching extra, the public symbol resolves to a placeholder that raises `ImportError` (`openai`/`gemini`) or the additional-dependency error (`sounddevice[numpy]`) the moment you use it.pip install "sounddevice[numpy]"
本地声卡上运行的典型OpenAI语音应用:`pip install "ag2[openai-realtime]" "sounddevice[numpy]"`。
> 必须操作。在运行代码前完成对应安装。若未安装匹配的扩展包,公共符号会解析为占位符,在首次使用时触发`ImportError`(针对`openai`/`gemini`)或额外依赖错误(针对`sounddevice[numpy]`)。Public API
公开API
All exported from :
ag2.live| Symbol | Role |
|---|---|
| Wraps a prompt + realtime config; |
| Gemini Live realtime provider config |
| OpenAI Realtime provider config |
| Text → speech (PCM bytes) |
| One-shot speech → text (transcription) |
| One-shot speech → English text (translation) |
| Mic capture → |
| Plays |
| Observer that speaks a text |
所有API均从导出:
ag2.live| 符号 | 作用 |
|---|---|
| 封装提示词与实时配置; |
| Gemini Live实时服务商配置 |
| OpenAI Realtime实时服务商配置 |
| 文本转语音(PCM字节) |
| 一次性语音转文本(转写) |
| 一次性语音转英文文本(翻译) |
| 麦克风捕获 → 流中生成 |
| 将 |
| 观察者,通过TTS配置将文本 |
How a live session works
实时会话工作原理
LiveAgentConversationContextmic ──SoundDeviceRecorder──▶ RecordedAudioEvent ──▶ provider session (Gemini/OpenAI)
│
SynthesizedAudioEvent (assistant audio)
▼
SoundDevicePlayer ──▶ speakerAlong the way the provider also emits / (your speech), (assistant text), /, and . is an async context manager that yields the shared so you can attach the recorder and player to it.
TranscriptionChunkEventTranscriptionCompletedEventModelMessageChunkToolCallEventToolResultEventUsageEventagent.run()ConversationContextLiveAgentConversationContext麦克风 ──SoundDeviceRecorder──▶ RecordedAudioEvent ──▶ 服务商会话(Gemini/OpenAI)
│
SynthesizedAudioEvent(助手音频)
▼
SoundDevicePlayer ──▶ 扬声器过程中服务商还会输出/(用户语音转写)、(助手文本)、/以及。是异步上下文管理器,会返回共享的,以便将录制器和播放器绑定到该上下文。
TranscriptionChunkEventTranscriptionCompletedEventModelMessageChunkToolCallEventToolResultEventUsageEventagent.run()ConversationContextMinimal recipe — full duplex voice loop [needs keys + audio]
最简示例——全双工语音循环 [需要密钥+音频设备]
python
import asyncio
from ag2.live import (
LiveAgent,
OpenAIRealTimeConfig,
SoundDevicePlayer,
SoundDeviceRecorder,
)
from ag2.live.openai import AudioOutput, InputConfig
agent = LiveAgent(
"voice_bot",
"You are a friendly realtime voice assistant. Keep replies short and conversational.",
config=OpenAIRealTimeConfig(
"gpt-realtime",
output=AudioOutput(voice="marin"),
# Transcribe the user's speech so we can see it too.
input=InputConfig(transcription={"model": "gpt-4o-mini-transcribe"}),
),
)
async def main() -> None:
# run() yields the shared ConversationContext.
async with agent.run() as ctx:
# Recorder and player bind to the SAME context so audio flows on one bus.
async with (
SoundDeviceRecorder(context=ctx), # mic -> RecordedAudioEvent
SoundDevicePlayer(context=ctx), # SynthesizedAudioEvent -> speaker
):
print("Talk to the agent. Ctrl-C to stop.")
await asyncio.Event().wait() # keep the session open
if __name__ == "__main__":
asyncio.run(main())Set (or pass ). For Gemini, set / (or pass ).
OPENAI_API_KEYclient=AsyncOpenAI(...)GEMINI_API_KEYGOOGLE_API_KEYclient=Swap to Gemini by changing only the config:
python
from ag2.live import GeminiRealTimeConfig
from ag2.live.gemini import AudioOutput, InputConfig
agent = LiveAgent(
"voice_bot",
"You are a friendly realtime voice assistant.",
config=GeminiRealTimeConfig(
"gemini-2.5-flash-native-audio-preview-12-2025",
output=AudioOutput(voice="Kore"),
input=InputConfig(transcribe=True), # user-speech transcription
),
)python
import asyncio
from ag2.live import (
LiveAgent,
OpenAIRealTimeConfig,
SoundDevicePlayer,
SoundDeviceRecorder,
)
from ag2.live.openai import AudioOutput, InputConfig
agent = LiveAgent(
"voice_bot",
"You are a friendly realtime voice assistant. Keep replies short and conversational.",
config=OpenAIRealTimeConfig(
"gpt-realtime",
output=AudioOutput(voice="marin"),
# Transcribe the user's speech so we can see it too.
input=InputConfig(transcription={"model": "gpt-4o-mini-transcribe"}),
),
)
async def main() -> None:
# run() yields the shared ConversationContext.
async with agent.run() as ctx:
# Recorder and player bind to the SAME context so audio flows on one bus.
async with (
SoundDeviceRecorder(context=ctx), # mic -> RecordedAudioEvent
SoundDevicePlayer(context=ctx), # SynthesizedAudioEvent -> speaker
):
print("Talk to the agent. Ctrl-C to stop.")
await asyncio.Event().wait() # keep the session open
if __name__ == "__main__":
asyncio.run(main())设置(或传入)。使用Gemini时,设置/(或传入)。
OPENAI_API_KEYclient=AsyncOpenAI(...)GEMINI_API_KEYGOOGLE_API_KEYclient=只需修改配置即可切换为Gemini:
python
from ag2.live import GeminiRealTimeConfig
from ag2.live.gemini import AudioOutput, InputConfig
agent = LiveAgent(
"voice_bot",
"You are a friendly realtime voice assistant.",
config=GeminiRealTimeConfig(
"gemini-2.5-flash-native-audio-preview-12-2025",
output=AudioOutput(voice="Kore"),
input=InputConfig(transcribe=True), # user-speech transcription
),
)Provider matrix — Gemini Live vs OpenAI Realtime
服务商对比——Gemini Live vs OpenAI Realtime
Gemini ( | OpenAI ( | |
|---|---|---|
| Import the knob types from | | |
| Output modes | | |
Voices ( | | |
| Example models (first positional arg) | | |
| User-speech transcription | | |
| Turn detection / VAD | | |
| Audio I/O sample rate | in 16 kHz / out 24 kHz (fixed by API) | configurable via |
| Generation knobs | | |
| Escape hatch for raw provider config | | |
| Bring your own client | | |
The model name is a positional arg; everything else is keyword-only. The model/voice lists above are accepted, but any string is allowed too (forward-compatible with new models).
LiteralAudioOutputlanguage_code=speed=format=TextOutput()ModelMessageChunkGemini( | OpenAI( | |
|---|---|---|
| 配置类型导入自 | | |
| 输出模式 | | |
语音选项( | | |
| 示例模型(第一个位置参数) | | |
| 用户语音转写 | | |
| 说话检测/VAD | | |
| 音频输入输出采样率 | 输入16 kHz / 输出24 kHz(API固定) | 可通过 |
| 生成控制参数 | | |
| 原始服务商配置入口 | | |
| 自定义客户端 | | |
模型名称为位置参数;其余均为关键字参数。上述类型的模型/语音列表均可直接使用,同时也支持任意字符串(兼容未来新增模型)。
LiteralAudioOutputlanguage_code=speed=format=TextOutput()ModelMessageChunkTools, HITL, observers on a LiveAgent
LiveAgentLiveAgent的工具、人工介入、观察者
LiveAgent(...)Agenttools=hitl_hook=middleware=observers=dependencies=variables=plugins=ToolCallEventrun()config=prompt=tools=observers=hitl_hook=Note:only supports function tools over realtime — provider server-side tool types raiseLiveAgentin both backends.NotImplementedError
LiveAgent(...)Agenttools=hitl_hook=middleware=observers=dependencies=variables=plugins=ToolCallEventrun()config=prompt=tools=observers=hitl_hook=注意:仅支持实时会话中的函数工具——服务商端工具类型在两个后端均会触发LiveAgent。NotImplementedError
Audio I/O — SoundDeviceRecorder
/ SoundDevicePlayer
SoundDeviceRecorderSoundDevicePlayer音频输入输出——SoundDeviceRecorder / SoundDevicePlayer
Both are async context managers () and bind to a via . The device opens on [needs audio]; construction alone touches no hardware.
async withConversationContextcontext=__aenter__python
from ag2.live import SoundDeviceRecorder, SoundDevicePlayer两者均为异步上下文管理器(),通过绑定到。设备会在时打开**[需要音频设备]**;仅构建实例不会操作硬件。
async withcontext=ConversationContext__aenter__python
from ag2.live import SoundDeviceRecorder, SoundDevicePlayerRecorder: mic -> RecordedAudioEvent on the stream.
录制器:麦克风 → 流中生成RecordedAudioEvent
recorder = SoundDeviceRecorder(context=ctx, sample_rate=16000, channels=1) # block_size optional
recorder = SoundDeviceRecorder(context=ctx, sample_rate=16000, channels=1) # block_size可选
Player: subscribes to SynthesizedAudioEvent, writes PCM to the speaker (24 kHz mono).
播放器:订阅SynthesizedAudioEvent,将PCM数据输出到扬声器(24 kHz单声道)
player = SoundDevicePlayer(context=ctx)
`SoundDeviceRecorder` also has a **one-shot blocking** `record(duration)` that returns a `VoiceInput` (16-bit PCM) — handy to feed the STT pipeline below without running a live session:
```python
voice = recorder.record(duration=4.0) # blocks 4s, returns VoiceInput [needs audio]player = SoundDevicePlayer(context=ctx)
`SoundDeviceRecorder`还提供**一次性阻塞式**`record(duration)`方法,返回`VoiceInput`(16位PCM)——无需运行实时会话即可为下文的STT流水线提供输入:
```python
voice = recorder.record(duration=4.0) # 阻塞4秒,返回VoiceInput [需要音频设备]One-shot speech-to-text + voice pipeline [needs keys; audio only if recording live]
一次性语音转文本+语音流水线 [需要密钥;录制实时音频时需音频设备]
OpenAITranscriberOpenAITranslationTranscriberVoiceInput.pipe(agent)VoicePipelineAgentpython
from ag2 import Agent
from ag2.config import OpenAIConfig
from ag2.live import OpenAITranscriber, SoundDeviceRecorder
from ag2.live.stt import VoiceInput
agent = Agent("assistant", "You answer questions.", config=OpenAIConfig(model="gpt-4o-mini"))
pipeline = OpenAITranscriber("gpt-4o-transcribe").pipe(agent)OpenAITranscriberOpenAITranslationTranscriberVoiceInput.pipe(agent)VoicePipelineAgentpython
from ag2 import Agent
from ag2.config import OpenAIConfig
from ag2.live import OpenAITranscriber, SoundDeviceRecorder
from ag2.live.stt import VoiceInput
agent = Agent("assistant", "You answer questions.", config=OpenAIConfig(model="gpt-4o-mini"))
pipeline = OpenAITranscriber("gpt-4o-transcribe").pipe(agent)Record a clip, transcribe it, and ask the agent — in one call. [needs keys + audio]
录制片段、转写并询问代理——一步完成 [需要密钥+音频设备]
voice: VoiceInput = SoundDeviceRecorder().record(4.0)
reply = await pipeline.ask(voice)
print(await reply.content())
voice: VoiceInput = SoundDeviceRecorder().record(4.0)
reply = await pipeline.ask(voice)
print(await reply.content())
Continue the conversation with another clip:
录制另一个片段继续对话:
followup = await reply.ask(SoundDeviceRecorder().record(4.0))
`VoiceInput(content: bytes, frame_rate: int, channels: int)` wraps 16-bit PCM; build it directly from any PCM source if you aren't using the recorder.
`OpenAITranslationTranscriber` is identical but always outputs English (useful for non-English speech in → English text out).followup = await reply.ask(SoundDeviceRecorder().record(4.0))
`VoiceInput(content: bytes, frame_rate: int, channels: int)`封装16位PCM数据;若不使用录制器,可直接从任意PCM源构建该对象。
`OpenAITranslationTranscriber`功能相同,但始终输出英文(适用于非英文语音输入转英文文本输出的场景)。Text-to-speech — OpenAITTSConfig
and TTSObserver
OpenAITTSConfigTTSObserver文本转语音——OpenAITTSConfig和TTSObserver
OpenAITTSConfigpython
from ag2.live import OpenAITTSConfig
tts = OpenAITTSConfig("gpt-4o-mini-tts", voice="alloy", speed=1.0)
pcm: bytes = await tts.synthesize("Hello there!") # [needs keys]TTSObserverAgentModelMessageChunkSynthesizedAudioEventSoundDevicePlayerpython
from ag2 import Agent, MemoryStream
from ag2.config import OpenAIConfig
from ag2.context import ConversationContext
from ag2.live import OpenAITTSConfig, SoundDevicePlayer, TTSObserver
agent = Agent(
"narrator",
"You are a helpful assistant.",
config=OpenAIConfig(model="gpt-4o-mini"),
observers=[TTSObserver(OpenAITTSConfig("gpt-4o-mini-tts", voice="nova"))],
)OpenAITTSConfigpython
from ag2.live import OpenAITTSConfig
tts = OpenAITTSConfig("gpt-4o-mini-tts", voice="alloy", speed=1.0)
pcm: bytes = await tts.synthesize("Hello there!") # [需要密钥]TTSObserverAgentModelMessageChunkSynthesizedAudioEventSoundDevicePlayerpython
from ag2 import Agent, MemoryStream
from ag2.config import OpenAIConfig
from ag2.context import ConversationContext
from ag2.live import OpenAITTSConfig, SoundDevicePlayer, TTSObserver
agent = Agent(
"narrator",
"You are a helpful assistant.",
config=OpenAIConfig(model="gpt-4o-mini"),
observers=[TTSObserver(OpenAITTSConfig("gpt-4o-mini-tts", voice="nova"))],
)Agent and Player must share ONE stream: agent.ask(stream=...) and
Agent和Player必须共享同一个流:agent.ask(stream=...)和
SoundDevicePlayer(context=...) whose .stream is that same stream.
SoundDevicePlayer(context=...)的.stream必须为同一个流
stream = MemoryStream()
ctx = ConversationContext(stream=stream)
async with SoundDevicePlayer(context=ctx): # plays the synthesized speech [needs audio]
reply = await agent.ask("Tell me a one-line joke.", stream=stream)
print(await reply.content())
`TTSObserver(config)` returns a `CompositeObserver`. It flushes any remaining buffered text on the final `ModelMessage`, so trailing partial sentences are still spoken.stream = MemoryStream()
ctx = ConversationContext(stream=stream)
async with SoundDevicePlayer(context=ctx): # 播放合成语音 [需要音频设备]
reply = await agent.ask("Tell me a one-line joke.", stream=stream)
print(await reply.content())
`TTSObserver(config)`返回`CompositeObserver`。它会在最终`ModelMessage`时刷新所有缓冲文本,确保末尾的不完整句子也能被播放。Usage reporting
使用报告
await LiveAgent.usage_report(ctx)UsageReportawait LiveAgent.usage_report(ctx)UsageReportWhat is construction-tested vs needs live keys + hardware
可离线构建测试 vs 需要实时密钥+硬件的功能
Run by this skill's (all green) — constructed / exercised offline:
references/test_samples.py- All 9 public symbols import as real classes (not missing-dependency placeholders).
- /
GeminiRealTimeConfigconstruction withOpenAIRealTimeConfig/AudioOutput/TextOutput, voices, temperature/max-tokens, andInputConfigmerge._build_session(instructions=...) - ,
OpenAITTSConfig,OpenAITranscriberconstruction.OpenAITranslationTranscriber - construction (string prompt, list prompt, with
LiveAgent/tools=).observers= - /
SoundDeviceRecorderconstruction (no device opened).SoundDevicePlayer - returns a
TTSObserver.CompositeObserver - →
OpenAITranscriber(...).pipe(agent);VoicePipelinedataclass.VoiceInput
Requires real API keys + microphone/speaker (NOT runnable headless):
- opening a live websocket to Gemini/OpenAI.
agent.run() - /
async with SoundDeviceRecorder(...)(opens the sound card onSoundDevicePlayer(...)).__aenter__ - (blocks on the mic).
SoundDeviceRecorder.record(duration) - ,
OpenAITTSConfig.synthesize(...), andOpenAITranscriber.transcribe(...)(hit the OpenAI API).VoicePipeline.ask(...)
本技能的可运行以下离线构建/测试功能(全部通过):
references/test_samples.py- 所有9个公开符号均可导入为真实类(而非缺失依赖的占位符)。
- /
GeminiRealTimeConfig的构建,包括OpenAIRealTimeConfig/AudioOutput/TextOutput、语音选项、温度/最大令牌数,以及InputConfig合并逻辑。_build_session(instructions=...) - 、
OpenAITTSConfig、OpenAITranscriber的构建。OpenAITranslationTranscriber - 的构建(字符串提示词、列表提示词,包含
LiveAgent/tools=)。observers= - /
SoundDeviceRecorder的构建(不打开设备)。SoundDevicePlayer - 返回
TTSObserver。CompositeObserver - 生成
OpenAITranscriber(...).pipe(agent);VoicePipeline数据类。VoiceInput
需要真实API密钥+麦克风/扬声器(无外设环境无法运行):
- 打开与Gemini/OpenAI的实时WebSocket连接。
agent.run() - /
async with SoundDeviceRecorder(...)(在SoundDevicePlayer(...)时打开声卡)。__aenter__ - (阻塞等待麦克风输入)。
SoundDeviceRecorder.record(duration) - 、
OpenAITTSConfig.synthesize(...)以及OpenAITranscriber.transcribe(...)(调用OpenAI API)。VoicePipeline.ask(...)
Common pitfalls
常见陷阱
- Wrong extra installed. for OpenAI,
pip install "ag2[openai-realtime]"for Gemini,"ag2[gemini-realtime]"for local audio. The symbol imports fine but raises on first use otherwise."sounddevice[numpy]" - missing.
numpy/SoundDeviceRecorderneed numpy —SoundDevicePlayerpulls it in. Without it you get an additional-dependencysounddevice[numpy].ImportError - Importing knob types from the wrong module. /
AudioOutput/TextOutputare provider-specific:InputConfigvsag2.live.gemini. They are not re-exported fromag2.live.openai.ag2.live - Recorder, player, and not sharing one context. Pass the
run()yielded byctxintoagent.run()/SoundDeviceRecorder(context=ctx), or audio events won't reach the session/speaker.SoundDevicePlayer(context=ctx) - Voice not in the provider's list. Gemini and OpenAI have different voice names (see matrix). A bare string is accepted but an unknown voice is rejected by the provider at session open.
- Expecting server-side / provider tools over realtime. Only function tools are supported; other tool types raise .
NotImplementedError - Forgetting to keep the loop alive. is a context manager — exit closes the session. Keep it open (
agent.run()or your own loop) for a continuous conversation.await asyncio.Event().wait()
- 安装了错误的扩展包:OpenAI需安装,Gemini需安装
pip install "ag2[openai-realtime]",本地音频需安装"ag2[gemini-realtime]"。符号可正常导入,但首次使用时会触发错误。"sounddevice[numpy]" - 缺失numpy:/
SoundDeviceRecorder需要numpy——SoundDevicePlayer会自动安装该依赖。若无numpy,会触发额外依赖sounddevice[numpy]。ImportError - 从错误模块导入配置类型:/
AudioOutput/TextOutput是服务商专属的:InputConfigvsag2.live.gemini。它们不会从ag2.live.openai重新导出。ag2.live - 录制器、播放器和未共享同一上下文:需将
run()返回的agent.run()传入ctx/SoundDeviceRecorder(context=ctx),否则音频事件无法到达会话/扬声器。SoundDevicePlayer(context=ctx) - 使用了服务商不支持的语音:Gemini和OpenAI的语音名称不同(见对比表)。虽然接受任意字符串,但未知语音会在会话开启时被服务商拒绝。
- 期望实时会话支持服务商端工具:仅支持函数工具;其他工具类型会触发。
NotImplementedError - 忘记保持会话活跃:是上下文管理器——退出会关闭会话。需保持会话开启(
agent.run()或自定义循环)以实现连续对话。await asyncio.Event().wait()
Going deeper
深入了解
- Source: .
ag2/live/{realtime.py,gemini.py,openai.py,protocols.py,observer.py,stt.py,sound_device.py} - —
realtime.py+ theLiveAgentprotocol.RealtimeConfig - For sending a recorded audio file as a one-off input to a text agent, see (
ag2-multimodal-input).AudioInput - For observers in general (token monitors, loop detection), see .
ag2-observers-and-alerts
- 源码:。
ag2/live/{realtime.py,gemini.py,openai.py,protocols.py,observer.py,stt.py,sound_device.py} - ——
realtime.py及LiveAgent协议。RealtimeConfig - 如需将录制的音频文件作为单次输入发送给文本代理,请查看(
ag2-multimodal-input)。AudioInput - 如需了解通用观察者(令牌监控、循环检测),请查看。
ag2-observers-and-alerts