launchdarkly-experiment-setup

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

LaunchDarkly Experiment Setup

LaunchDarkly 实验设置

You're using a skill that guides you through setting up and running experiments in LaunchDarkly. Your job is to design the experiment, create it with the right metrics, treatments, and flag config, start data collection, evolve the design between iterations when needed, and stop with a winner. Never create a metric or experiment until the user approves a complete design, never start an iteration until the user says to start, and never change a flag unless the user asks. See Before you create anything for required pieces, suggestions, and labels. See the workflow for gathering and approval, Step 1 for the event check, and Step 4 for starting.

你正在使用一项 skill，该 skill 会指导你在 LaunchDarkly 中设置并运行实验。你的工作是设计实验，使用正确的指标、Treatment 和 Flag 配置创建实验，启动数据收集，在需要时跨迭代演进设计，并以胜出方案结束。除非用户批准完整设计，否则绝不创建指标或实验；除非用户明确要求开始，否则绝不启动迭代；除非用户要求，否则绝不更改 Flag。参见“创建任何内容之前”了解必需项、建议项和标签。参见“工作流”了解信息收集和批准流程，参见“步骤 1”了解事件检查，参见“步骤 4”了解如何启动。

Prerequisites

前置条件

This skill requires the remotely hosted LaunchDarkly MCP server to be configured in your environment.

Required MCP tools:

```
create-experiment
```
— create a new experiment with its initial iteration (hypothesis, metrics, treatments, flag config).
```
start-experiment-iteration
```
— begin collecting data for an experiment's current draft iteration.
```
get-experiment
```
— check experiment status, treatments, metrics, and current iteration.

Optional MCP tools:

```
list-experiments
```
— browse existing experiments in the project.
```
update-experiment
```
— update fields on the experiment or its current iteration. Honours
```
mutableFieldsByStatus
```
, so what's editable depends on whether the iteration is
```
not_started
```
,
```
running
```
, or
```
stopped
```
. Returns rejected inputs under
```
skipped
```
.
```
save-and-start-experiment-iteration
```
— the API-recommended way to change locked fields on a running experiment. Stops the current iteration, creates a new draft with the supplied field updates, and starts it in one call.
```
stop-experiment-iteration
```
— stop the running iteration. You must declare a winner: pass the
```
winningTreatmentId
```
(and a
```
winningReason
```
). If no variation outperformed, pick the baseline/control as the winner.
```
list-metrics
```
,
```
create-metric
```
,
```
list-metric-events
```
— manage metrics referenced by the experiment.
```
get-flag
```
— read one flag's targeting in an environment: whether it is on, the variation a rule serves, its off variation, and whether that rule is a percentage rollout.
```
list-flags
```
— search flags in a project by name or key.
```
create-flag
```
— create a feature flag in a project. It is off in every environment until turned on.
```
get-flag-status-across-envs
```
— read a flag's status in each environment (
```
new
```
,
```
active
```
,
```
launched
```
, or
```
inactive
```
).
```
search-contexts
```
— find stored contexts in an environment by kind, to see which context kinds are in use.

此 skill 要求在环境中配置远程托管的 LaunchDarkly MCP 服务端（MCP server）。

必需的 MCP 工具：

```
create-experiment
```
— 创建新实验及其初始迭代（假设、指标、Treatment、Flag 配置）。
```
start-experiment-iteration
```
— 开始为实验当前的草稿迭代收集数据。
```
get-experiment
```
— 检查实验状态、Treatment、指标和当前迭代。

可选的 MCP 工具：

```
list-experiments
```
— 浏览项目中的现有实验。
```
update-experiment
```
— 更新实验或其当前迭代的字段。遵循
```
mutableFieldsByStatus
```
，因此可编辑内容取决于迭代是
```
not_started
```
、
```
running
```
还是
```
stopped
```
。被拒绝的输入会在
```
skipped
```
下返回。
```
save-and-start-experiment-iteration
```
— API 推荐的方式，用于更改运行中实验的锁定字段。通过一次调用停止当前迭代、创建应用所提供字段更新的新草稿并启动它。
```
stop-experiment-iteration
```
— 停止正在运行的迭代。必须声明胜出者：传入
```
winningTreatmentId
```
（以及
```
winningReason
```
）。如果没有哪个变体表现更优，则选择 baseline/control 作为胜出者。

list-metrics

、

create-metric

、

list-metric-events

— 管理实验引用的指标。

```
get-flag
```
— 读取某个环境中单个 Flag 的定位规则：Flag 是否开启、规则提供哪个变体、off variation 是哪个，以及该规则是否为百分比放量。
```
list-flags
```
— 按名称或 key 搜索项目中的 Flag。
```
create-flag
```
— 在项目中创建 Feature Flag。在开启之前，该 Flag 在所有环境中均处于关闭状态。
```
get-flag-status-across-envs
```
— 读取 Flag 在各个环境中的状态（
```
new
```
、
```
active
```
、
```
launched
```
或
```
inactive
```
）。
```
search-contexts
```
— 按 kind 查找某个环境中存储的 context，以查看正在使用哪些 context kind。

Core Concepts

核心概念

What Are Experiments?

实验是什么？

Experiments in LaunchDarkly measure the impact of feature flag variations on key metrics. An experiment consists of:

Treatments: the flag variations being compared (control vs. test). Each treatment has an
```
allocationPercent
```
; the values across treatments should sum to 100.
Metrics: what you're measuring (conversion rate, latency, revenue, etc.). One must be the primary metric.
Flag config: the
```
flagKey
```
,
```
ruleId
```
, and
```
flagConfigVersion
```
of the targeting rule that drives the experiment.
Iteration: a single data-collection window. Created in
```
not_started
```
status, becomes
```
running
```
when started, transitions to
```
stopped
```
when ended.
Holdout (optional): a project-level group of users excluded from the experiment for baseline measurement (
```
holdoutId
```
).

LaunchDarkly 中的实验用于衡量 Feature Flag 变体对关键指标的影响。一个实验包括：

Treatments（处理组）：正在比较的 Flag 变体（对照与测试）。每个 Treatment 都有一个
```
allocationPercent
```
；所有 Treatment 的值应合计为 100。
Metrics（指标）：要衡量的内容（转化率、延迟、收入等）。其中必须有一个主要指标。
Flag config（Flag 配置）：驱动实验的定位规则所对应的
```
flagKey
```
、
```
ruleId
```
和
```
flagConfigVersion
```
。
Iteration（迭代）：单个数据收集窗口。创建时状态为
```
not_started
```
，启动后变为
```
running
```
，结束后转为
```
stopped
```
。
Holdout（保留组，可选）：项目级用户组，被排除在实验之外以进行基线衡量（
```
holdoutId
```
）。

Experiment Lifecycle

实验生命周期

Create the experiment with its first iteration (
```
create-experiment
```
).
Start the iteration to begin data collection (
```
start-experiment-iteration
```
), only after the user explicitly says to start.
Monitor results as data accumulates (
```
get-experiment
```
).
Evolve the design mid-experiment if needed — change locked fields like
```
treatments
```
,
```
metrics
```
, or
```
methodology
```
by calling
```
save-and-start-experiment-iteration
```
, which stops the current iteration, creates a new draft with your changes, and starts it. That call starts data collection, so make it only after the user explicitly says to start.
Stop the iteration when you have a winner or a clear call (
```
stop-experiment-iteration
```
).
Ship the winning variation.

使用
```
create-experiment
```
创建实验及其第一个迭代。
启动迭代以开始收集数据（
```
start-experiment-iteration
```
），且只能在用户明确要求开始后执行。
随着数据积累监控结果（
```
get-experiment
```
）。
如有必要，在实验中途调整设计——调用
```
save-and-start-experiment-iteration
```
更改
```
treatments
```
、
```
metrics
```
或
```
methodology
```
等锁定字段。该调用会停止当前迭代、创建包含变更的新草稿并启动它。由于此调用会开始收集数据，因此只能在用户明确要求开始后执行。
获得胜出者或作出明确决定后，停止迭代（
```
stop-experiment-iteration
```
）。
发布胜出的变体。

Core Principles

核心原则

Metrics first: ensure the metrics you'll reference exist before creating the experiment.
Clear hypothesis: every iteration requires a
```
hypothesis
```
string from the user. See Before you create anything.
Proper controls: exactly one treatment must have
```
baseline: true
```
.
Sufficient sample size: let iterations run long enough to support a decision. See Step 7.
One change at a time: test one variable per experiment for clear attribution.

指标优先：创建实验前，确保将要引用的指标已经存在。
清晰的假设：每个迭代都需要用户提供一个
```
hypothesis
```
字符串。参见“创建任何内容之前”。
适当的对照组：必须恰好有一个 Treatment 设置
```
baseline: true
```
。
充足的样本量：让迭代运行足够长时间，以支持决策。参见“步骤 7”。
一次只更改一个变量：每个实验只测试一个变量，以确保归因清晰。

Before you create anything

创建任何内容之前

These three must come from the user. Never fill them in:

Flag and variations: which flag to test, and which of its variations are in the experiment.
Hypothesis: what they expect to improve, and by how much if they know.
Primary metric: the single metric or funnel that decides the result.

If any of these three are missing, leave them blank until the user supplies them. They go in the gather message. See the workflow. Once the user names the flag, see Step 2.

Control variation: which of those variations is the baseline (

baseline: true

). If the user named it, use that. If they did not:

Ask, and leave it blank, when the flag is brand new, the targeted rule serves a percentage rollout, or the flag is off and
```
get-flag-status-across-envs
```
reports that environment as
```
inactive
```
. A flag is brand new if you created it in this conversation, or
```
get-flag-status-across-envs
```
reports that environment as
```
new
```
: the flag was created fewer than seven days ago and has never been evaluated there. A percentage rollout is a targeted rule that splits traffic across variations instead of serving one.
```
get-flag
```
shows which.
```
inactive
```
means the flag has not been evaluated for at least seven days, so no one is currently being served the off variation.
Otherwise suggest one. If the flag is on, suggest the variation the targeted rule serves today. If the flag is off, suggest its off variation.

Randomization unit: the context kind the experiment randomizes on. If the user named it, use that. If they did not, call

search-contexts

for this project and environment. Suggest

user

only when a

kind

user

returns contexts and

filter

kind notEquals "user"

returns

totalCount

0, so

user

is the only context kind in use. If

search-contexts

is unavailable, returns no contexts, or any other kind is stored, ask and leave it blank.

Label every suggested value in the design as suggested, with a one-line reason, for example

Control (suggested, what users see today): Start free trial

. The user can change any suggestion before approving. Approving the design accepts the suggestions still shown in it.

Settings the user did not mention can stay on LaunchDarkly's defaults. List each default you are using in the design:

Traffic split: equal
```
allocationPercent
```
across the variations the user named (two variations is 50/50). The values must sum to 100.
Statistics: LaunchDarkly default (frequentist, sequential testing). Do not send
```
methodology
```
unless the user chooses one. If they choose one, send that value and show it here instead of this default. A Bayesian choice is
```
methodology
```
```
"bayesian"
```
.
Data source:
```
dataSource
```
```
"launchdarkly"
```
.
Reshuffle:
```
canReshuffleTraffic
```
```
true
```
.

Leave

analysisConfig

unset unless the user set it, and say that in the design. The randomization unit is never one of these defaults.

以下三项必须来自用户。绝不能自行填写：

Flag 与变体：要测试哪个 Flag，以及实验包含该 Flag 的哪些变体。
假设：用户预期改善什么，以及如果知道的话，改善幅度是多少。
主要指标：决定实验结果的单一指标或漏斗。

如果这三项中任何一项缺失，应保持空白，直到用户提供后再填写。这些内容应包含在信息收集消息中。参见“工作流”。用户指定 Flag 后，参见“步骤 2”。

Control variation：哪个变体是 baseline（

baseline: true

）。如果用户已指定，则直接使用。如果没有指定：

当 Flag 是全新创建的、目标规则提供百分比放量，或者 Flag 处于关闭状态且
```
get-flag-status-across-envs
```
报告该环境为
```
inactive
```
时，应询问用户并保持空白。如果 Flag 是在本次对话中创建的，或者
```
get-flag-status-across-envs
```
报告该环境为
```
new
```
，则该 Flag 属于全新 Flag：它创建于不到七天前，并且从未在该环境中被评估过。百分比放量是指将流量拆分到不同变体、而不是只提供一个变体的目标规则，可通过
```
get-flag
```
查看。
```
inactive
```
表示 Flag 至少七天未经过评估，因此当前没有用户被提供 off variation。
在其他情况下，建议一个变体。如果 Flag 已开启，建议目标规则当前提供的变体；如果 Flag 已关闭，建议其 off variation。

Randomization unit（随机化单元）：实验进行随机化所使用的 context kind。如果用户已指定，则直接使用。如果没有指定，则针对该项目和环境调用

search-contexts

。仅当 kind 为

user

的查询返回了 context，且筛选

kind notEquals "user"

返回的

totalCount

为 0，从而表明

user

是唯一正在使用的 context kind 时，才建议使用

user

。如果

search-contexts

不可用、没有返回 context，或者存储了任何其他 kind，则询问用户并保持空白。

将设计中的每个建议值标记为“建议”，并附上一行原因，例如

Control（建议，用户当前看到的版本）：Start free trial

。用户可以在批准前更改任何建议。批准设计即表示接受其中仍然显示的建议。

用户未提及的设置可以保留 LaunchDarkly 默认值。在设计中列出正在使用的每个默认值：

流量分配：在用户指定的各变体之间平均分配
```
allocationPercent
```
（两个变体时为 50/50）。这些值必须合计为 100。
统计方法：LaunchDarkly 默认值（频率主义、序贯检验）。除非用户选择其他方法，否则不要发送
```
methodology
```
。如果用户选择了其他方法，则改为发送该值，并在此显示，而不是显示默认值。Bayesian 选择为
```
methodology
```
```
"bayesian"
```
。
数据源：
```
dataSource
```
```
"launchdarkly"
```
。
重新分流：
```
canReshuffleTraffic
```
```
true
```
。

除非用户已设置

analysisConfig

，否则保持其未设置状态，并在设计中说明这一点。Randomization unit 绝不属于这些默认值。

Workflow

工作流

If nothing is missing, skip the gather message and go straight to the design. Otherwise gather every missing piece in one message before you show any design, then wait:

the flag and variations, the hypothesis, or the primary metric, when one is missing
the named flag, when
```
get-flag
```
cannot find it: say so and offer three choices in one message: correct the key (list similar flags from
```
list-flags
```
); create the flag themselves, then tell you to continue; or have you create it. For that last choice, propose the key, name, type, and variations from their design, say it will be created off, and wait for an explicit yes before calling
```
create-flag
```
. If
```
launchdarkly-flag-create
```
is installed, follow it for that step. After the flag exists, continue setup. Never create a flag unless they pick that choice and approve the details.
the control or the randomization unit, when the rules above say to ask instead of suggest
for a new primary metric, its kind (
```
custom
```
,
```
pageview
```
, or
```
click
```
), what it is based on (event key, URL match, or click selector), and how it is measured (share of units who converted, a total count, or an average), when the user has not already made that clear

list-metrics

and

get-flag

are fine while you gather this.

create-metric

and

create-experiment

are not.

If the user's wording already makes one of these clear, do not ask about it. "The share of users who sent the

cta-clicked

event" already names the measure, the kind, and the event key.

Show the design only when nothing is blank. Never show a design with a missing value. Include every piece gathered above, the other treatments, flag config (

flagKey

ruleId

flagConfigVersion

), and every default you are applying. Statistics reads

LaunchDarkly default (frequentist, sequential testing)

unless the user chose a methodology, in which case show their choice. Label each suggested value. See Before you create anything.

STOP HERE until the user approves the design.

Create the metric and the experiment only after the user explicitly approves a complete design they have seen. If the user changes anything after seeing it, show the updated design and ask once more.

如果没有任何缺失项，则跳过信息收集消息，直接进入设计。否则，在展示任何设计之前，应在一条消息中收集所有缺失内容，然后等待：

Flag 与变体、假设或主要指标缺失时，收集相应内容。
当
```
get-flag
```
无法找到用户指定的 Flag 时，应说明情况，并在一条消息中提供三个选择：更正 key（通过
```
list-flags
```
列出相似的 Flag）；由用户自行创建 Flag，然后通知你继续；或者由你创建该 Flag。对于最后一种选择，应根据用户的设计提出 key、name、type 和 variations，并说明 Flag 创建后处于关闭状态，在调用
```
create-flag
```
前等待用户明确同意。如果已安装
```
launchdarkly-flag-create
```
，则该步骤遵循该 skill。Flag 存在后继续设置。除非用户选择由你创建并批准具体详情，否则绝不创建 Flag。
当上述规则要求询问而不是建议时，收集 control 或 randomization unit。
对于新的主要指标，如果用户尚未明确说明，则收集其 kind（
```
custom
```
、
```
pageview
```
或
```
click
```
）、依据什么构建（event key、URL match 或 click selector），以及如何衡量（转化单元占比、总计数或平均值）。

信息收集期间可以使用

list-metrics

和

get-flag

，但不能使用

create-metric

和

create-experiment

。

如果用户的表述已经使其中某项内容足够明确，则不要询问。例如，“发送

cta-clicked

事件的用户占比”已经明确了衡量方式、kind 和 event key。

仅当没有任何空白项时才展示设计。绝不展示存在缺失值的设计。设计中必须包含上面收集到的全部内容、其他 Treatments、Flag 配置（

flagKey

、

ruleId

、

flagConfigVersion

），以及正在应用的每一个默认值。除非用户选择了某种统计方法，否则统计方法显示为

LaunchDarkly 默认值（频率主义、序贯检验）

；如果用户选择了统计方法，则显示其选择。标记每一个建议值。参见“创建任何内容之前”。

在此停止，直到用户批准设计。

仅当用户明确批准其看过的完整设计后，才创建指标和实验。如果用户在看过设计后更改了任何内容，应展示更新后的设计，并再次请求批准。

Step 1: Prepare Metrics

步骤 1：准备指标

Use
```
list-metrics
```
to find existing metrics. The user names the primary metric.
If it already exists, keep its name and key for the design review.
If it does not exist, a miss only means the metric is new. The user has often named the metric and not its kind or event key. Gather those before the design. Until the user has said the kind and, for a custom event, the event key:
- Do not call
```
list-metric-events
```
  .
- Do not check the metric name, or any other key the user did not give, as the event key.
- Do not skip the flow check by treating the metric as
```
pageview
```
  or
```
click
```
  .
- Do not block creation for an event you have not identified.
See the gather rule for a new metric's kind, what it is based on, and how it is measured.
After the user has said the new metric is
```
custom
```
and named its event key, call
```
list-metric-events
```
for that event key. Either way, the metric goes in the design once nothing else is blank. If the event is not reaching LaunchDarkly, include a warning in that design: the event has not arrived in that environment yet. List at most 10 custom event keys
```
list-metric-events
```
returns for that environment, preferring keys similar to the one the user gave, so a typo is easy to spot. Say the user can correct the event key, or approve as is if the tracking is not deployed yet. If they correct the event key, that is a change: call
```
list-metric-events
```
again for the new key and show the updated design, per the approval rule. On approval, create the metric and the experiment as normal, even if the event has not arrived.
Pageview and click metrics have no custom event key. Skip
```
list-metric-events
```
only after the user has said the kind is
```
pageview
```
or
```
click
```
.
Pass the user-named primary metric as
```
primarySingleMetricKey
```
or
```
primaryFunnelKey
```
.

Goal	Metric type	Example key
Conversion	Custom conversion	`checkout-completed`
Performance	Custom numeric	`page-load-time-ms`
Engagement	Custom conversion	`feature-clicked`
Revenue	Custom numeric	`order-value`

使用
```
list-metrics
```
查找现有指标。用户会指定主要指标。
如果指标已经存在，在设计审查中保留其名称和 key。
如果指标不存在，查询未命中只表示该指标是新指标。用户通常会指定指标名称，但不会指定其 kind 或 event key。在展示设计前收集这些信息。在用户指定 kind，并且对于 custom 指标指定 event key 之前：
- 不要调用
```
list-metric-events
```
  。
- 不要将指标名称或用户未提供的任何其他 key 当作 event key 来检查。
- 不要跳过流程检查并将指标视为
```
pageview
```
  或
```
click
```
  。
- 不要因为尚未识别事件而阻止创建。
有关新指标的 kind、构建依据和衡量方式，参见收集规则。
用户指定新指标为
```
custom
```
并给出 event key 后，针对该 event key 调用
```
list-metric-events
```
。无论事件是否到达，只要其他内容没有空缺，该指标就会进入设计。如果事件尚未到达 LaunchDarkly，应在设计中加入警告：该事件尚未到达当前环境。列出
```
list-metric-events
```
针对该环境返回的最多 10 个 custom event key，并优先选择与用户提供的 key 相似的 key，以便发现拼写错误。告知用户可以更正 event key，也可以在埋点尚未部署时按当前内容批准。如果用户更正 event key，则属于设计变更：针对新 key 再次调用
```
list-metric-events
```
，并按照批准规则展示更新后的设计。获得批准后，即使事件尚未到达，也照常创建指标和实验。
Pageview 和 click 指标没有 custom event key。只有在用户明确指定 kind 为
```
pageview
```
或
```
click
```
后，才跳过
```
list-metric-events
```
。
将用户指定的主要指标作为
```
primarySingleMetricKey
```
或
```
primaryFunnelKey
```
传入。

目标	指标类型	示例 key
转化	自定义转化	`checkout-completed`
性能	自定义数值	`page-load-time-ms`
互动	自定义转化	`feature-clicked`
收入	自定义数值	`order-value`

Step 2: Identify the Targeting Rule

步骤 2：确定定位规则

You need the

ruleId

and current

flagConfigVersion

of the flag rule that will drive the experiment. Use

get-flag

on the flag (or its environment-scoped status) to find them. The fallthrough rule's id is the string

"fallthrough"

需要获取驱动实验的 Flag 规则所对应的

ruleId

和当前

flagConfigVersion

。对该 Flag（或其环境范围状态）使用

get-flag

来查找。fallthrough 规则的 id 是字符串

"fallthrough"

。

Step 3: Create the Experiment

步骤 3：创建实验

See the gather and approval rule above, then call

create-metric

for a new metric and

create-experiment

. See Step 1 for the event check.

Call

create-experiment

. The top-level fields describe the experiment; the nested

iteration

object describes the first data-collection window. The values in the example below are the shape of the call, not values to copy. See Before you create anything for the randomization unit.

json

{
  "projectKey": "my-project",
  "environmentKey": "production",
  "key": "checkout-flow-v2-experiment",
  "name": "Checkout Flow v2 Experiment",
  "description": "Compare the redesigned checkout against the current flow.",
  "tags": ["growth", "checkout"],
  "iteration": {
    "hypothesis": "The redesigned checkout will lift completion rate by 3%.",
    "primarySingleMetricKey": "checkout-completed",
    "metrics": [
      { "key": "checkout-completed" },
      { "key": "checkout-time-seconds" }
    ],
    "treatments": [
      {
        "name": "Control",
        "baseline": true,
        "allocationPercent": 50,
        "parameters": [
          { "flagKey": "checkout-flow-v2", "variationId": "variation-a-id" }
        ]
      },
      {
        "name": "New Checkout",
        "baseline": false,
        "allocationPercent": 50,
        "parameters": [
          { "flagKey": "checkout-flow-v2", "variationId": "variation-b-id" }
        ]
      }
    ],
    "flags": {
      "checkout-flow-v2": {
        "ruleId": "fallthrough",
        "flagConfigVersion": 7
      }
    },
    "randomizationUnit": "user"
  }
}

Omit

methodology

unless the user chose one. When they did, set it to that value (

"bayesian"

"frequentist"

, or

"export_only"

Useful optional top-level fields:

```
holdoutId
```
— attach an existing holdout.

dataSource

—

"launchdarkly"

(default),

"snowflake"

, or

"databricks"

```
methodology
```
— send only when the user chooses one:
```
"bayesian"
```
,
```
"frequentist"
```
, or
```
"export_only"
```
. Omit it otherwise. LaunchDarkly's default is frequentist with sequential testing.
```
analysisConfig
```
— set only the fields the user set. Leave it unset to keep sequential testing on.

Useful optional iteration fields:

```
attributes
```
— array of context attribute keys to slice results by (e.g.
```
["country", "device"]
```
).
```
covariateId
```
— covariate CSV id for stratified sampling.
```
canReshuffleTraffic
```
— defaults to
```
true
```
; set
```
false
```
to lock users to their initial variation when allocations change.

参见上述信息收集和批准规则，然后为新指标调用

create-metric

，并调用

create-experiment

。有关事件检查，参见“步骤 1”。

调用

create-experiment

。顶层字段描述实验，嵌套的

iteration

对象描述第一个数据收集窗口。下面示例中的值仅用于展示调用结构，不可直接复制。有关 randomization unit，参见“创建任何内容之前”。

json

{
  "projectKey": "my-project",
  "environmentKey": "production",
  "key": "checkout-flow-v2-experiment",
  "name": "Checkout Flow v2 Experiment",
  "description": "Compare the redesigned checkout against the current flow.",
  "tags": ["growth", "checkout"],
  "iteration": {
    "hypothesis": "The redesigned checkout will lift completion rate by 3%.",
    "primarySingleMetricKey": "checkout-completed",
    "metrics": [
      { "key": "checkout-completed" },
      { "key": "checkout-time-seconds" }
    ],
    "treatments": [
      {
        "name": "Control",
        "baseline": true,
        "allocationPercent": 50,
        "parameters": [
          { "flagKey": "checkout-flow-v2", "variationId": "variation-a-id" }
        ]
      },
      {
        "name": "New Checkout",
        "baseline": false,
        "allocationPercent": 50,
        "parameters": [
          { "flagKey": "checkout-flow-v2", "variationId": "variation-b-id" }
        ]
      }
    ],
    "flags": {
      "checkout-flow-v2": {
        "ruleId": "fallthrough",
        "flagConfigVersion": 7
      }
    },
    "randomizationUnit": "user"
  }
}

除非用户选择了统计方法，否则省略

methodology

。如果用户已选择，则将其设为该值（

"bayesian"

、

"frequentist"

或

"export_only"

）。

常用的可选顶层字段：

```
holdoutId
```
— 关联现有 Holdout。

dataSource

—

"launchdarkly"

（默认值）、

"snowflake"

或

"databricks"

。

```
methodology
```
— 仅当用户选择一种方法时发送：
```
"bayesian"
```
、
```
"frequentist"
```
或
```
"export_only"
```
，否则省略。LaunchDarkly 默认使用频率主义和序贯检验。
```
analysisConfig
```
— 仅设置用户指定的字段。保持未设置状态可继续启用序贯检验。

常用的可选迭代字段：

```
attributes
```
— 用于按维度拆分结果的 context attribute key 数组，例如
```
["country", "device"]
```
。
```
covariateId
```
— 用于分层抽样的 covariate CSV id。
```
canReshuffleTraffic
```
— 默认为
```
true
```
；设为
```
false
```
时，在分配比例发生变化后将用户锁定在其初始变体中。

Step 4: Start Data Collection

步骤 4：启动数据收集

Creating the experiment leaves the iteration in

not_started

. Call

start-experiment-iteration

only after the user explicitly says to start. That same wait applies to any other call that starts an iteration.

json

{
  "projectKey": "my-project",
  "environmentKey": "production",
  "experimentKey": "checkout-flow-v2-experiment"
}

Before starting, the API requires that:

the flag is toggled on,
the iteration has a
```
randomizationUnit
```
, and
at least one treatment has a non-zero
```
allocationPercent
```
.

If the primary metric is a custom event, call

list-metric-events

again right before starting. If the event is arriving, start as normal once the flag is on. If it still is not arriving, the experiment can start but results will stay empty until the event arrives, so ask whether to start anyway.

If the flag is off in that environment when the user says to start, don't change the flag. Tell the user the flag must be on in that environment, and that turning it on starts serving its variations to users there. Ask whether they want to turn it on themselves or have you do it, and wait for an explicit answer before calling any tool that changes the flag. Once the flag is on, call

get-flag

again to get the current

flagConfigVersion

, update the draft to use it, and tell the user you did, before starting.

If the flag is off and the custom event still is not arriving, raise both in one message, the flag question and whether to start anyway, and wait for that one answer. Do not ask them separately.

Pass

changeJustification

if you're restarting after a prior iteration was stopped.

创建实验后，迭代会处于

not_started

状态。只有在用户明确要求开始后，才调用

start-experiment-iteration

。同样的等待规则也适用于任何其他会启动迭代的调用。

json

{
  "projectKey": "my-project",
  "environmentKey": "production",
  "experimentKey": "checkout-flow-v2-experiment"
}

启动前，API 要求：

Flag 已切换为开启状态；
迭代具有
```
randomizationUnit
```
；
至少一个 Treatment 的
```
allocationPercent
```
非零。

如果主要指标是 custom 事件，请在启动前再次调用

list-metric-events

。如果事件正在到达，并且 Flag 已开启，则照常启动。如果事件仍未到达，实验仍可启动，但在事件到达前结果会保持为空，因此应询问用户是否仍要启动。

如果用户要求启动时 Flag 在该环境中处于关闭状态，不要更改 Flag。告知用户必须开启该环境中的 Flag，开启后会开始向该环境中的用户提供其变体。询问用户希望自行开启，还是由你开启，并在调用任何更改 Flag 的工具前等待明确答复。Flag 开启后，再次调用

get-flag

获取当前的

flagConfigVersion

，更新草稿以使用该版本，并告知用户已经完成更新，然后再启动。

如果 Flag 处于关闭状态，且 custom 事件仍未到达，应在一条消息中同时提出这两个问题：Flag 的处理方式，以及是否仍要启动。等待用户对这一个问题作出答复。不要分别询问。

如果在此前已停止的迭代之后重新启动，请传入

changeJustification

。

Step 5: Verify

步骤 5：验证

After the user has asked you to start, call
```
get-experiment
```
and confirm
```
currentIteration.status === "running"
```
.
Check that treatments are present with the expected allocations.
Check the metric list and the primary metric.

用户要求开始后，调用

get-experiment

并确认

currentIteration.status === "running"

。

检查 Treatments 是否存在，并且流量分配符合预期。
检查指标列表和主要指标。

Step 6: Evolve the Design Mid-Experiment (when needed)

步骤 6：在实验中途调整设计（如有需要）

Most structural fields (treatments, metrics, methodology, hypothesis, …) are locked while an iteration is

running

. Two ways to change them:

Light edits while running —
```
update-experiment
```
will let through anything
```
mutableFieldsByStatus
```
permits in the
```
running
```
state (typically just metadata like
```
name
```
,
```
description
```
,
```
maintainerId
```
,
```
tags
```
, plus appending
```
metrics
```
/
```
attributes
```
). It surfaces rejected fields under
```
skipped
```
with a reason.
Real design changes — call
```
save-and-start-experiment-iteration
```
. It stops the current iteration, creates a new draft with the supplied field updates applied, and starts it in one call. Inputs match
```
update-experiment
```
, plus
```
changeJustification
```
. Mutability is checked against
```
not_started
```
since updates land on the new draft.

save-and-start-experiment-iteration

starts the new iteration. Show the change, then wait until the user explicitly says to start before calling it.

Example: swap the treatment allocation and add a metric in a single call.

json

{
  "projectKey": "my-project",
  "environmentKey": "production",
  "experimentKey": "checkout-flow-v2-experiment",
  "changeJustification": "Lowering control allocation now that variant looks safe.",
  "treatments": [
    {
      "name": "Control",
      "baseline": true,
      "allocationPercent": 30,
      "parameters": [{ "flagKey": "checkout-flow-v2", "variationId": "variation-a-id" }]
    },
    {
      "name": "New Checkout",
      "baseline": false,
      "allocationPercent": 70,
      "parameters": [{ "flagKey": "checkout-flow-v2", "variationId": "variation-b-id" }]
    }
  ],
  "metrics": [
    { "key": "checkout-completed" },
    { "key": "checkout-time-seconds" },
    { "key": "checkout-error-rate" }
  ]
}

当迭代处于

running

状态时，大多数结构字段（

treatments

、

metrics

、

methodology

、

hypothesis

等）均被锁定。有两种更改方式：

运行期间的轻量编辑 —
```
update-experiment
```
仅允许
```
mutableFieldsByStatus
```
在
```
running
```
状态下允许修改的内容（通常包括
```
name
```
、
```
description
```
、
```
maintainerId
```
、
```
tags
```
等元数据，以及追加
```
metrics
```
/
```
attributes
```
）。被拒绝的字段会在
```
skipped
```
下返回，并附带原因。
实际设计变更 — 调用
```
save-and-start-experiment-iteration
```
。它会停止当前迭代、创建应用所提供字段更新的新草稿，并一次调用启动新迭代。输入参数与
```
update-experiment
```
相同，另需提供
```
changeJustification
```
。由于更新会落到新草稿上，因此可变性针对
```
not_started
```
状态进行检查。

save-and-start-experiment-iteration

会启动新迭代。展示变更内容，然后等待用户明确要求开始，之后才能调用它。

示例：一次调用中交换 Treatment 的流量分配并添加一个指标。

json

{
  "projectKey": "my-project",
  "environmentKey": "production",
  "experimentKey": "checkout-flow-v2-experiment",
  "changeJustification": "Lowering control allocation now that variant looks safe.",
  "treatments": [
    {
      "name": "Control",
      "baseline": true,
      "allocationPercent": 30,
      "parameters": [{ "flagKey": "checkout-flow-v2", "variationId": "variation-a-id" }]
    },
    {
      "name": "New Checkout",
      "baseline": false,
      "allocationPercent": 70,
      "parameters": [{ "flagKey": "checkout-flow-v2", "variationId": "variation-b-id" }]
    }
  ],
  "metrics": [
    { "key": "checkout-completed" },
    { "key": "checkout-time-seconds" },
    { "key": "checkout-error-rate" }
  ]
}

Step 7: Stop the Iteration

步骤 7：停止迭代

When the results support a decision, or the user has made a call, stop the iteration. For a frequentist experiment that decision is statistical significance. For a Bayesian experiment it is the probability to beat baseline or the probability to be best. Quote only the statistic for the methodology in use. A winning treatment is required to stop — LaunchDarkly does not let you end an iteration without declaring a winner. Pass the winning treatment's id (returned in

get-experiment

_id

on each treatment) plus a

winningReason

If the experiment was inconclusive or no variation beat the control, declare the baseline/control treatment as the winner and say so in

winningReason

(e.g. "Inconclusive — keeping control"). There is no "stop without a winner" path.

json

{
  "projectKey": "my-project",
  "environmentKey": "production",
  "experimentKey": "checkout-flow-v2-experiment",
  "winningTreatmentId": "treat-002",
  "winningReason": "Two weeks of data, +4.1% lift on the primary metric."
}

Name the decision statistic in

winningReason

. Frequentist: significance, or that it was not significant. Bayesian: probability to beat baseline or probability to be best. Do not quote a Bayesian probability, or a prior, on a frequentist experiment.

Report results:

Iteration stopped with the declared
```
winningTreatmentId
```
(the control/baseline if inconclusive).
Lift on the primary metric, plus the decision statistic for the methodology in use.
Next steps (ship the winner, roll back, or start a follow-up iteration).

当结果支持作出决定，或用户已经作出决定时，停止迭代。对于频率主义实验，该决定依据是统计显著性；对于 Bayesian 实验，该决定依据是胜出 baseline 的概率或成为最佳方案的概率。只能引用当前统计方法对应的统计量。停止时必须指定一个胜出的 Treatment — LaunchDarkly 不允许在不声明胜出者的情况下结束迭代。传入胜出 Treatment 的 id（

get-experiment

会通过每个 Treatment 上的

_id

返回）以及

winningReason

。

如果实验结果不确定，或没有任何变体优于 control，则将 baseline/control Treatment 声明为胜出者，并在

winningReason

中说明（例如“Inconclusive — keeping control”）。不存在“停止但不选择胜出者”的方式。

json

{
  "projectKey": "my-project",
  "environmentKey": "production",
  "experimentKey": "checkout-flow-v2-experiment",
  "winningTreatmentId": "treat-002",
  "winningReason": "Two weeks of data, +4.1% lift on the primary metric."
}

在

winningReason

中注明决策统计量。对于频率主义实验，注明显著性，或说明结果不显著；对于 Bayesian 实验，注明胜出 baseline 的概率或成为最佳方案的概率。不要在频率主义实验中引用 Bayesian 概率或先验概率。

报告结果：

迭代已停止，并声明了
```
winningTreatmentId
```
（结果不确定时为 control/baseline）。
主要指标的提升幅度，以及当前统计方法对应的决策统计量。
后续步骤（发布胜出方案、回滚或启动后续迭代）。

Edge Cases

边界情况

Situation	Action
A new primary metric's kind or event key is unknown	Gather them before the event check. See Step 1.
The new metric is a custom event and the user named its event key	Warn in the design if it has not arrived; still create on approval. See Step 1.
Flag, variations, hypothesis, or primary metric is missing	Ask, and leave them blank. See Before you create anything.
The named flag does not exist in the project	Offer to correct the key, have them create it, or create it after they approve the details. Never create a flag otherwise. See the gather rule.
User did not name the control, and the rules say to ask	Ask for the control; don't suggest one. See Before you create anything.
User did not name the control, and the rules say to suggest	Suggest the control and label it. See Before you create anything.
User did not name the randomization unit	Suggest or ask for the randomization unit. See Before you create anything.
Design not approved	Leave `create-metric` and `create-experiment` uncalled. See the gather and approval rule above.
User has not said to start	Leave the iteration in `not_started` . `start-experiment-iteration` and `save-and-start-experiment-iteration` stay uncalled.
Flag is off when the user says to start	Don't change the flag; ask first. See Step 4.
Flag has no variations to compare	Add variations only when the user explicitly asks. Otherwise say which variations exist and ask.
You don't know the flag's `ruleId` / `flagConfigVersion`	Use `get-flag` or `get-flag-status-across-envs` . The fallthrough rule's id is the string `"fallthrough"` .
Experiment already exists	Use `list-experiments` to find it; `get-experiment` for details.
Need to change locked fields mid-experiment	Use `save-and-start-experiment-iteration` (single call) rather than stopping and recreating by hand.
`update-experiment` returns `skipped` for a field	Inspect the `currentStatus` and `allowedFields` in the response — that field isn't mutable in the current iteration status. Either stop the iteration first or use `save-and-start-experiment-iteration` .

情况	处理方式
新主要指标的 kind 或 event key 未知	在事件检查前收集这些信息。参见“步骤 1”。
新指标是 custom 事件，且用户已指定 event key	如果事件尚未到达，在设计中加入警告；获得批准后仍可创建。参见“步骤 1”。
缺少 Flag、变体、假设或主要指标	询问用户并保持空白。参见“创建任何内容之前”。
指定的 Flag 不存在于项目中	提供更正 key、由用户创建，或在用户批准详情后由你创建的选择。否则绝不创建 Flag。参见收集规则。
用户未指定 control，且规则要求询问	询问 control，不要建议。参见“创建任何内容之前”。
用户未指定 control，且规则要求建议	建议 control 并标记为建议。参见“创建任何内容之前”。
用户未指定 randomization unit	建议或询问 randomization unit。参见“创建任何内容之前”。
设计未获批准	不调用 `create-metric` 和 `create-experiment` 。参见上述信息收集和批准规则。
用户尚未要求开始	让迭代保持 `not_started` 状态。不调用 `start-experiment-iteration` 和 `save-and-start-experiment-iteration` 。
用户要求启动时 Flag 处于关闭状态	不要更改 Flag；先询问用户。参见“步骤 4”。
Flag 没有可比较的变体	仅在用户明确要求时添加变体。否则说明现有变体并询问用户。
不知道 Flag 的 `ruleId` / `flagConfigVersion`	使用 `get-flag` 或 `get-flag-status-across-envs` 。Fallthrough 规则的 id 是字符串 `"fallthrough"` 。
实验已经存在	使用 `list-experiments` 查找；使用 `get-experiment` 获取详情。
需要在实验中途更改锁定字段	使用 `save-and-start-experiment-iteration` （单次调用），不要手动停止并重新创建。
`update-experiment` 对某个字段返回 `skipped`	检查响应中的 `currentStatus` 和 `allowedFields` — 该字段在当前迭代状态下不可修改。可以先停止迭代，或使用 `save-and-start-experiment-iteration` 。

What NOT to Do

禁止事项

Don't fill in a required piece yourself, or present a suggestion without labeling it as suggested. See Before you create anything.
Don't call
```
list-metric-events
```
before Step 1 says to, or block creation because an event has not arrived. See Step 1.
Don't call
```
create-metric
```
or
```
create-experiment
```
before the approval rule above is satisfied.

Don't call

start-experiment-iteration

save-and-start-experiment-iteration

until the user explicitly says to start.

Don't turn a flag on or off, or change its targeting, unless the user explicitly asks you to.
Don't create a flag unless the user asks you to and approves the proposed key, name, type, and variations. See the gather rule.
Don't omit
```
iteration
```
on
```
create-experiment
```
— it's required.
Don't set
```
baseline: true
```
on more than one treatment.
Don't let
```
allocationPercent
```
values fail to sum to 100 across treatments.
Don't try to change locked iteration fields with
```
update-experiment
```
while the iteration is
```
running
```
— reach for
```
save-and-start-experiment-iteration
```
instead.
Don't stop iterations early — wait for a decision the methodology supports. See Step 7.
Don't run multiple experiments on the same flag at the same time without a careful holdout design.

不要自行填写必需项，也不要展示未标记为建议的建议值。参见“创建任何内容之前”。
不要在“步骤 1”允许之前调用
```
list-metric-events
```
，也不要因为事件尚未到达而阻止创建。参见“步骤 1”。
不要在满足上述批准规则之前调用
```
create-metric
```
或
```
create-experiment
```
。

不要在用户明确要求开始之前调用

start-experiment-iteration

或

save-and-start-experiment-iteration

。

除非用户明确要求，否则不要开启或关闭 Flag，也不要更改其定位规则。
除非用户要求并批准提出的 key、name、type 和 variations，否则不要创建 Flag。参见收集规则。
不要在
```
create-experiment
```
中省略
```
iteration
```
— 它是必需字段。
不要在多个 Treatment 上设置
```
baseline: true
```
。
不要让各 Treatment 的
```
allocationPercent
```
之和不等于 100。
当迭代处于
```
running
```
状态时，不要尝试使用
```
update-experiment
```
更改锁定字段 — 应改用
```
save-and-start-experiment-iteration
```
。
不要过早停止迭代 — 应等待统计方法所支持的决策条件。参见“步骤 7”。
如果没有经过周密的 Holdout 设计，不要在同一 Flag 上同时运行多个实验。