Back to Details

writing-livekit-scenarios

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Writing simulation scenarios

编写模拟场景

A scenario is a simulated user's 
instructions
 (who they are, what they want) plus 
agent_expectations
 (what counts as success). A simulation plays the scenario against the real agent, and an LLM judge grades the transcript.
Runs are disposable. The scenario file is what lasts: it gets reviewed in diffs, re-run for years, and it's what catches a prompt edit that breaks something without anyone noticing.
Before writing a file, look up the exact schema and CLI flags with 
reading-livekit-docs
. Field names and commands change, so this skill doesn't restate them. Conceptually, a file is a named group of scenarios. Each scenario has the simulated user's instructions, the pass criteria, and optionally tags for grouping and per-scenario data the agent can read at runtime. If the CLI offers to add stable per-scenario ids to a file, accept and commit them.
一个场景是一个模拟用户的**
instructions
**（他们是谁，想要什么）加上 
agent_expectations
（什么算成功）。模拟会针对真实agent运行场景，并由LLM评判器对对话记录进行评分。
运行是可丢弃的。场景文件才是持久的：它会在diff中被审查，持续运行多年， 并且它能捕捉到某个提示词编辑在无人注意的情况下破坏功能的情况。
在编写文件之前，请使用
reading-livekit-docs
查看确切的模式和CLI标志。字段 名称和命令会变化，因此本skill不重复说明它们。从概念上讲，一个文件是一组命名的场景。 每个场景包含模拟用户的指令、通过标准，以及可选的用于分组的标签和agent在运行时可以读取的每个场景数据。如果CLI提供为文件添加稳定的每场景id，请接受并提交它们。

Start from a generated baseline

从生成的基线开始

Don't start from an empty file. LiveKit's generator reads the agent's source and produces a decent first pass much faster than you'd write ten scenarios by hand:
bash
lk agent simulate text -n 10    # confirm the exact flags with --help
Generating from source uploads the code to LiveKit Cloud so the generator can read it, which is why the CLI asks for confirmation first. Tell the user that before running it and let them decline. Some code can't leave the machine, and in that case you write the scenarios by hand. A flag skips the prompt for non-interactive runs.
When the run finishes, the CLI either tells you where it saved the generated scenarios or offers to save them into the project. Either way, get that file into the repo as the starting point.
You can't steer the generator. It infers intent from the code, so it writes plausible conversations instead of the ones this agent's users have, and it doesn't know what the user is worried about. Treat its output as a draft:
  1. Read every scenario and delete the ones that don't matter. A file you haven't read isn't a test suite.
  2. Sharpen vague expectations. If the judge can't decide an 
    agent_expectations
    , its verdict flips between runs. This is the biggest cause of flaky scenarios.
  3. Add what generation misses. It drifts toward happy paths. 
    references/risk-coverage.md
    explains how to turn the agent's constraints into a checklist and give each item a scenario.
  4. Add what the user is worried about. If they haven't said, ask what they want stress-tested. That input is why a suite a person steers is better than a generated one.
不要从空文件开始。LiveKit的生成器读取agent的源代码，并生成一个不错的 初稿，比手动编写十个场景快得多：
bash
lk agent simulate text -n 10    # 使用--help确认确切的标志
从源码生成会将代码上传到LiveKit Cloud，以便生成器读取它，这就是 CLI首先要求确认的原因。在运行之前告诉用户，并让他们可以选择拒绝。 有些代码不能离开机器，在这种情况下，你需要手动编写场景。有一个标志可以在非交互式运行时跳过提示。
当运行完成时，CLI要么告诉你它在哪里保存了生成的场景，要么提供 将它们保存到项目中。无论哪种方式，将该文件放入仓库作为起点。
你无法操控生成器。它从代码中推断意图，因此它编写的是合理的对话， 而不是这个agent用户实际会遇到的那些，并且它不知道用户担心什么。 将它的输出视为草稿：
  1. 阅读每个场景，删除不重要的场景。一个你没有读过的文件不是测试套件。
  2. 完善模糊的期望。 如果评判器无法决定
    agent_expectations
    ，它的判定 会在运行之间翻转。这是场景不稳定的最大原因。
  3. 补充生成遗漏的内容。 它倾向于快乐路径。
    references/risk-coverage.md
    解释了如何将agent的约束转化为检查清单，并为每个项目分配一个场景。
  4. 添加用户担心的内容。 如果他们没有说明，询问他们想要压力测试什么。 这种输入是为什么有人指导的套件优于生成的套件的原因。

Ask what to probe

询问要探测什么

The user knows which flow keeps breaking, which customer complained, and which change they're nervous about. Ask, and weight the suite toward that area with more and deeper scenarios.
Focus adds coverage. It never removes coverage of the agent's hard limits. If the user has no preference, generate broadly and tell them that's what you did.
用户知道哪个流程一直出问题，哪个客户抱怨过，哪个更改让他们紧张。询问，并将套件偏向该区域，增加更多更深入的场景。
专注会增加覆盖。它永远不会移除对agent硬性限制的覆盖。如果用户没有偏好，生成广泛的覆盖，并告诉他们你就是这样做的。

Ground scenarios in what the agent can do

将场景基于agent能做什么

Read the agent's code locally with your normal tools before writing anything. Look for what a user can ask for (capabilities), where requests get blocked (constraints: required steps, unavailable items by name, caps, eligibility), and what the agent must refuse.
Constraints matter most. A scenario that asks for something the agent can't do is only valid if the expectation is that the agent says so. Written the other way, the test fails when the agent behaves correctly.
Scope is the agent reachable from the session entrypoint, plus agents it hands off to and tasks it awaits. Ignore other classes in the directory, unused imports, and example files.
在编写任何内容之前，用你常用的工具在本地阅读agent的代码。查找用户能请求什么（capabilities），请求在哪些地方被阻止（constraints：必需步骤、按名称不可用的项目、上限、资格），以及agent必须拒绝什么。
约束最重要。一个要求agent做不到的事情的场景只有在期望agent这么说时才有效。以另一种方式编写，当agent行为正确时测试会失败。
范围是从会话入口点可达的agent，加上它移交给的agent以及它等待的任务。忽略目录中的其他类、未使用的导入和示例文件。

Write instructions the simulated user can follow

编写模拟用户能遵循的指令

There are two shapes, and which one you use depends on what the scenario is for. Details and examples are in 
references/scenario-craft.md
.
  • Point-form direction for deterministic flows you'll grade on end state. A structured template (persona, opening line, facts revealed only when asked, steps in order, conditional reactions) keeps the persona model on track and makes the run repeatable.
  • A persona paragraph plus goals for open-ended and adversarial scenarios graded only on the conversation, where you want the simulated user to improvise.
In both shapes, goals are requests to the agent, never the agent's own actions. Use real values from the agent's domain and assume no prior state. Vary persona, mood and difficulty across the suite so it isn't ten copies of the same cooperative caller.
有两种形式，使用哪一种取决于场景的用途。详细信息和示例在
references/scenario-craft.md
中。
  • 要点式指导用于你将在最终状态上评分的确定性流程。一个结构化的模板 （角色、开场白、只在被问到时揭示的事实、按顺序的步骤、条件反应） 保持角色模型在轨道上，并使运行可重复。
  • 一个角色段落加上目标用于开放式和对抗性场景，这些场景仅根据对话评分， 在这种情况下你希望模拟用户即兴发挥。
在两种形式中，目标都是向agent的请求，绝不是agent自己的行为。使用来自agent领域的真实值，并假设没有先前状态。在套件中变化角色、情绪和难度，这样它就不是十个相同协作来电者的副本。

Split scenarios into sets, one file per set

将场景拆分为组，每组一个文件

The run command takes one scenario file, so a file is a run. Split along the lines you want to run separately, which usually isn't topic.
The most important axis is how the set is graded:
  • Scenarios that drive concrete tool flows to a deterministic end state, graded on final state and the conversation (see "Make the agent consume the scenario" below).
  • Open-ended and adversarial scenarios graded on 
    agent_expectations
    alone.
These need different instructions shapes, different agent wiring, and often a different run cadence, so they go in different files. The second axis is cadence: a small set to run before a merge, and the full set before a release.
Give each file a 
name
that describes the set, since it labels the run. Inside a file, use 
tags
to slice. Tag 
feature
so a failure points at the part of the agent that owns it, and add whatever else you filter by (channel, difficulty). Put a header comment on the file recording the set's assumptions: pinned dates, required environment variables, and what its expectations depend on.
运行命令接受一个场景文件，因此一个文件就是一次运行。沿着你想单独运行的界线拆分，这通常不是主题。
最重要的轴是组的评分方式：
  • 将具体工具流程驱动到确定性最终状态的场景，根据最终状态以及对话评分（见下文“让agent消费场景”）。
  • 仅根据
    agent_expectations
    评分的开放式和对抗性场景。
这些需要不同的指令形式、不同的agent接线，并且通常需要不同的运行节奏，所以它们放在不同的文件中。第二个轴是节奏：合并前运行的小型集合，发布前运行的完整集合。
给每个文件一个描述集合的
name
，因为它标记了运行。在文件内部，使用
tags
进行切片。标记
feature
，这样失败时可以指向拥有它的agent部分，并添加你过滤所需的任何其他内容（渠道、难度）。在文件上放置一个头部注释，记录集合的假设：固定的日期、必需的环境变量以及其期望所依赖的内容。

Keep scenarios reproducible

保持场景可复现

A scenario should give the same result months from now as it does today.
  • Write absolute dates and pin the agent's clock (usually with an environment variable) so availability and expectations always line up. Relative dates rot.
  • Don't let scenarios hit real backends. Seed deterministic state from the scenario instead (see "Make the agent consume the scenario" below).
  • Keep ids stable where the CLI supports them. Labels and instructions change, and the id is what ties a scenario's runs together over time. A scenario rewritten from scratch is a new scenario and gets a new id.
一个场景应该在几个月后给出与今天相同的结果。
  • 编写绝对日期并固定agent的时钟（通常使用环境变量），这样 可用性和期望总是对齐的。相对日期会腐烂。
  • 不要让场景命中真实后端。 而是从场景中播种确定性状态 （见下文“让agent消费场景”）。
  • 在CLI支持的情况下保持id稳定。 标签和指令会变化，而id是 将场景的多次运行随时间联系起来的纽带。一个从头重写的场景是一个新场景，并获得一个新id。

Make the agent consume the scenario

让agent消费场景

A scenario file can't fix two problems by itself. A scenario that reaches a real backend grades differently every run, and a judge that only reads the transcript will pass a run that booked the wrong room. Both are fixed in the agent's code, in one branch at the top of the entrypoint:
  1. Detect a simulation from the job context. In production the check comes back empty and nothing else changes.
  2. Seed state from the scenario's per-scenario data (a fake calendar, an in-memory database) that the agent's real tools read through unchanged.
  3. Mock tools for the session's lifetime, so tool selection is still under test but execution is deterministic.
  4. Grade the final state in the end-of-simulation callback. Compare what the agent ended with against what the scenario expected, and fail the run if they differ. Your check can fail a run the judge passed. It can't rescue one the judge failed.
Only tool-flow scenarios need this. Open-ended and adversarial ones are graded on the conversation alone. 
references/connecting-the-agent.md
covers it in full, including how to confirm the wiring took effect. Look up the current API names with 
reading-livekit-docs
.
一个场景文件本身无法解决两个问题。一个触及真实后端的场景每次运行评分都不同，而一个只读取对话记录的评判器会让一个订错房间的运行通过。两者都在agent的代码中修复，在入口点顶部的一个分支中：
  1. 检测来自作业上下文的模拟。在生产环境中，检查结果为空，其他一切不变。
  2. 播种来自场景的每场景数据（一个假日历、一个内存数据库）的状态， agent的真实工具通过不变的方式读取这些数据。
  3. 在会话的生命周期内模拟（mock）工具，这样工具选择仍然在测试范围内，但执行是确定性的。
  4. 在模拟结束回调中评估最终状态。 将agent最终得到的结果与场景期望的结果进行比较，如果不同则使运行失败。你的检查可以失败评判器通过的运行。它无法挽救评判器失败的运行。
只有工具流程场景需要这个。开放式和对抗性场景仅根据对话评分。
references/connecting-the-agent.md
完整介绍了这一点，包括如何确认接线是否生效。使用
reading-livekit-docs
查找当前的API名称。

Grow the suite from real failures

从真实失败中扩展套件

The most valuable scenarios describe something that went wrong. Once the agent is live, derive them from recorded sessions instead of inventing more. Where the CLI supports it, a subcommand derives a scenario from a recorded session (check 
lk agent simulate --help
). A derived scenario describes one call, so widen it to the class of calls it represents and sharpen its expectation into the rule you want enforced.
Every production bug you fix should get a permanent scenario in the file.
最有价值的场景描述了某种出错的情况。一旦agent上线，从录制的会话中派生它们，而不是发明更多。在CLI支持的地方，一个子命令从录制的会话中派生场景（检查
lk agent simulate --help
）。一个派生的场景描述一个通话，因此将其扩展到它代表的通话类别，并将其期望磨砺为你想要执行的规则。
你修复的每个生产bug都应该在文件中获得一个永久场景。

Don't write bad tests

不要编写糟糕的测试

The judge grades the agent against 
agent_expectations
, so a careless expectation punishes correct behavior:
  • For guardrails and negative cases, the pass is that the agent refuses, escalates, or declines to invent data. Write the expectation that way.
  • Never write an expectation the agent can only meet by misbehaving, such as stating data it can't know or giving specific medical, legal or financial directives. If passing requires misbehavior, the scenario is wrong.
  • Judge by outcome, from the user's point of view. Don't prescribe wording.
评判器根据
agent_expectations
对agent进行评分，因此粗心的期望会惩罚正确行为：
  • 对于护栏和负面案例，通过条件是agent拒绝、升级或拒绝编造数据。以这种方式编写期望。
  • 永远不要编写agent只有通过不当行为才能满足的期望，例如陈述它无法知道的数据或给出具体的医疗、法律或财务指令。如果通过需要不当行为，那么场景就是错误的。
  • 从用户的角度按结果判断。不要规定措辞。

References

参考

  • references/risk-coverage.md
    — turning constraints into a checklist and guaranteeing coverage
  • references/scenario-craft.md
    — instructions shapes, persona variety, audio-specific scenarios
  • references/connecting-the-agent.md
    — the agent-side code: detect, seed, mock, grade final state
  • references/risk-coverage.md
    — 将约束转化为检查清单并保证覆盖
  • references/scenario-craft.md
    — 指令形式、角色多样性、特定于音频的场景
  • references/connecting-the-agent.md
    — agent端代码：检测、播种、模拟、评估最终状态

Related skills

相关skills

  • Running them: 
    running-livekit-simulations
  • Schema and CLI facts: 
    reading-livekit-docs
  • 运行它们：
    running-livekit-simulations
  • 模式和CLI事实：
    reading-livekit-docs