Back to Details

generate-synthetic-data

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Generate Synthetic Data

生成合成数据

Generate diverse, realistic test inputs that cover the failure space of an LLM pipeline.
生成多样化、贴近真实场景的测试输入，覆盖LLM管道的故障场景。

Prerequisites

前提条件

Before generating synthetic data, identify where the pipeline is likely to fail. Ask the user about known failure-prone areas, review existing user feedback, or form hypotheses from available traces. Dimensions (Step 1) must target anticipated failures, not arbitrary variation.
在生成合成数据之前，需确定管道可能出现故障的位置。询问用户已知的易故障区域、查看现有用户反馈，或从可用轨迹中形成假设。维度（步骤1）必须针对预期的故障点，而非随意的变量。

Core Process

核心流程

Step 1: Define Dimensions

步骤1：定义维度

Dimensions are axes of variation specific to your application. Choose dimensions based on where you expect failures.
Dimension 1: [Name] — [What it captures]
  Values: [value_a, value_b, value_c, ...]

Dimension 2: [Name] — [What it captures]
  Values: [value_a, value_b, value_c, ...]

Dimension 3: [Name] — [What it captures]
  Values: [value_a, value_b, value_c, ...]
Example for a real estate assistant:
Feature: what task the user wants
  Values: [property search, scheduling, email drafting]

Client Persona: who the user serves
  Values: [first-time buyer, investor, luxury buyer]

Scenario Type: query clarity
  Values: [well-specified, ambiguous, out-of-scope]
Start with 3 dimensions. Add more only if initial traces reveal failure patterns along new axes.
维度是特定于你的应用的变化轴。根据你预期的故障点选择维度。
Dimension 1: [Name] — [What it captures]
  Values: [value_a, value_b, value_c, ...]

Dimension 2: [Name] — [What it captures]
  Values: [value_a, value_b, value_c, ...]

Dimension 3: [Name] — [What it captures]
  Values: [value_a, value_b, value_c, ...]
房地产助手示例：
Feature: what task the user wants
  Values: [property search, scheduling, email drafting]

Client Persona: who the user serves
  Values: [first-time buyer, investor, luxury buyer]

Scenario Type: query clarity
  Values: [well-specified, ambiguous, out-of-scope]
从3个维度开始。只有当初始轨迹揭示了新轴上的故障模式时，再添加更多维度。

Step 2: Draft 20 Tuples with the User

步骤2：与用户共同草拟20个元组

A tuple is one combination of dimension values defining a specific test case. Present 20 draft tuples to the user and iterate until they confirm the tuples reflect realistic scenarios. The user's domain knowledge is essential here — they know which combinations actually occur and which are unrealistic.
(Feature: Property Search, Persona: Investor, Scenario: Ambiguous)
(Feature: Scheduling, Persona: First-time Buyer, Scenario: Well-specified)
(Feature: Email Drafting, Persona: Luxury Buyer, Scenario: Out-of-scope)
元组是维度值的一种组合，定义了一个特定的测试用例。向用户展示20个草拟的元组，反复迭代直到用户确认这些元组反映了真实场景。用户的领域知识至关重要——他们知道哪些组合实际存在，哪些不切实际。
(Feature: Property Search, Persona: Investor, Scenario: Ambiguous)
(Feature: Scheduling, Persona: First-time Buyer, Scenario: Well-specified)
(Feature: Email Drafting, Persona: Luxury Buyer, Scenario: Out-of-scope)

Step 3: Generate More Tuples with an LLM

步骤3：使用LLM生成更多元组

Generate 10 random combinations of ({dim1}, {dim2}, {dim3})
for a {your application description}.

The dimensions are:
{dim1}: {description}. Possible values: {values}
{dim2}: {description}. Possible values: {values}
{dim3}: {description}. Possible values: {values}

Output each tuple in the format: ({dim1}, {dim2}, {dim3})
Avoid duplicates. Vary values across dimensions.
Generate 10 random combinations of ({dim1}, {dim2}, {dim3})
for a {your application description}.

The dimensions are:
{dim1}: {description}. Possible values: {values}
{dim2}: {description}. Possible values: {values}
{dim3}: {description}. Possible values: {values}

Output each tuple in the format: ({dim1}, {dim2}, {dim3})
Avoid duplicates. Vary values across dimensions.

Step 4: Convert Each Tuple to a Natural Language Query

步骤4：将每个元组转换为自然语言查询

Use a separate prompt for this step. Single-step generation (tuples + queries together) produces repetitive phrasing.
We are generating synthetic user queries for a {your application}.
{Brief description of what it does.}

Given:
{dim1}: {value}
{dim2}: {value}
{dim3}: {value}

Write a realistic query that a user might enter. The query should
reflect the specified persona and scenario characteristics.

Example: "{one of your hand-written examples}"

Now generate a new query.
此步骤使用单独的提示。单步生成（元组+查询一起生成）会导致措辞重复。
We are generating synthetic user queries for a {your application}.
{Brief description of what it does.}

Given:
{dim1}: {value}
{dim2}: {value}
{dim3}: {value}

Write a realistic query that a user might enter. The query should
reflect the specified persona and scenario characteristics.

Example: "{one of your hand-written examples}"

Now generate a new query.

Step 5: Filter for Quality

步骤5：质量过滤

Review generated queries. Discard and regenerate when:
  • Phrasing is awkward or unrealistic
  • Content doesn't match the tuple's intent
  • Queries are too similar to each other
Optional: use an LLM to rate realism on a 1-5 scale, discard below 3.
审核生成的查询。出现以下情况时，丢弃并重新生成：
  • 措辞生硬或不真实
  • 内容与元组的意图不符
  • 查询之间过于相似
可选：使用LLM在1-5分范围内对真实性评分，丢弃3分以下的查询。

Step 6: Run Queries Through the Pipeline

步骤6：在管道中运行查询

Execute all queries through the full LLM pipeline. Capture complete traces: input, all intermediate steps, tool calls, retrieved docs, final output.
Target: ~100 high-quality, diverse traces. This is a rough heuristic for reaching saturation (where new traces stop revealing new failure categories). The number depends on system complexity.
在完整的LLM管道中执行所有查询。捕获完整轨迹：输入、所有中间步骤、工具调用、检索到的文档、最终输出。
目标：约100条高质量、多样化的轨迹。 这是达到饱和状态（新轨迹不再揭示新的故障类别）的大致经验值。具体数量取决于系统复杂度。

Sampling Real User Data

真实用户数据抽样

When you have real queries available, don't sample randomly. Use stratified sampling:
  1. Identify high-variance dimensions — read through queries and find ways they differ (length, topic, complexity, presence of constraints).
  2. Assign labels — for small sets, with the user; for large sets, use K-means clustering on query embeddings.
  3. Sample from each group — ensures coverage across query types, not just the most common ones.
When both real and synthetic data are available, use synthetic data to fill gaps in underrepresented query types.
当有真实查询可用时，不要随机抽样。使用分层抽样：
  1. 识别高方差维度 —— 通读查询，找出它们的差异点（长度、主题、复杂度、是否包含约束条件）。
  2. 分配标签 —— 对于小数据集，与用户一起完成；对于大数据集，使用查询嵌入的K-means聚类。
  3. 从每个组中抽样 —— 确保覆盖所有查询类型，而不仅仅是最常见的类型。
当同时拥有真实数据和合成数据时，使用合成数据填补代表性不足的查询类型的空白。

Anti-Patterns

反模式

  • Unstructured generation. Prompting "give me test queries" without the dimension/tuple structure produces generic, repetitive, happy-path examples.
  • Single-step generation. Generating tuples and queries in one prompt produces less diverse results than the two-step separation.
  • Arbitrary dimensions. Dimensions that don't target failure-prone regions waste test budget.
  • Skipping user review of tuples. Without the user validating tuples first, you can't judge whether LLM-generated tuples are realistic.
  • Synthetic data when no one can judge realism. If no one can judge whether a synthetic trace is realistic, use real data instead.
  • Synthetic data for complex domain-specific content (legal filings, medical records) where LLMs miss structural nuance.
  • Synthetic data for low-resource languages or dialects where LLM-generated samples are unrealistic.
  • 无结构化生成：仅提示“给我测试查询”而不使用维度/元组结构，会生成通用、重复的常规路径示例。
  • 单步生成：将元组和查询在一个提示中生成，结果的多样性不如两步分离生成。
  • 随意定义维度：未针对易故障区域的维度会浪费测试资源。
  • 跳过用户对元组的审核：如果没有用户先验证元组，你无法判断LLM生成的元组是否真实。
  • 在无人能判断真实性时使用合成数据：如果没人能判断合成轨迹是否真实，请改用真实数据。
  • 针对复杂领域特定内容使用合成数据（如法律文件、医疗记录），此时LLM会遗漏结构细节。
  • 针对低资源语言或方言使用合成数据，此时LLM生成的样本不真实。