Experimentation is how you learn whether a design change actually helps. But poorly designed experiments produce false confidence.
Hypothesis structure:
"If we [specific change], then [specific metric] will [direction of change] by [estimated magnitude] because [causal reasoning]."
Example: "If we move the search bar from the header to the hero section, then search usage will increase by 15% because users will encounter it earlier in their scanning pattern, reducing the friction of scrolling up to search."
Minimum detectable effect (MDE):
What's the smallest change worth detecting? A 0.1% improvement in conversion may not be worth the engineering effort. A 5% improvement would be. Set the MDE before the test, not after. This determines your required sample size.
Sample size calculation:
Depends on: baseline conversion rate, MDE, statistical power (typically 80%), significance level (typically 95% / alpha = 0.05). Don't guess — use the formula or a calculator.
Quick reference for common scenarios (two-sided test, 80% power, 95% significance, two variants):
| Baseline rate | MDE (relative) | Sample size per variant |
|---|
| 5% | 20% (5% → 6%) | ~25,000 |
| 10% | 10% (10% → 11%) | ~14,500 |
| 10% | 20% (10% → 12%) | ~3,800 |
| 25% | 10% (25% → 27.5%) | ~4,800 |
| 50% | 5% (50% → 52.5%) | ~6,000 |
Lower baseline rates and smaller MDEs require dramatically more traffic. If your required sample size exceeds your monthly traffic, either increase the MDE (detect only larger effects), extend the test duration, or accept that an A/B test is not the right method — use qualitative research instead. Underpowered tests produce inconclusive results that waste time.
Duration:
Run for at least 1-2 full weekly cycles to account for day-of-week effects. Longer for seasonal businesses. Never run less than a week even if you hit sample size early — behavioral patterns vary by day.
Segmentation:
Check for differential effects across user segments: new vs. returning users, mobile vs. desktop, geography, plan type. An overall neutral result may hide a strong positive effect for one segment and a strong negative for another.
Guardrail metrics:
Define what must NOT get worse. If testing a new checkout flow, guardrail metrics might include: revenue per user, support ticket volume, return rate. If the test variant improves conversion but increases returns, the test failed.
Common mistakes:
- Peeking at results before the test reaches statistical significance (inflates false positive rate)
- Running too many variants without adjusting for multiple comparisons
- Ignoring novelty effects (new things get clicked more just because they're new — wait for the effect to stabilize)
- Stopping tests too early because early results "look decisive"
- Not accounting for interaction effects when multiple tests run simultaneously
- Testing cosmetic changes when the real problem is structural
实验是了解设计变更是否真正有效的方法。但设计糟糕的实验会产生虚假的信心。
假设结构：
“如果我们做出[具体变更]，那么[具体指标]将[变化方向] [预估幅度]，因为[因果推理]。”
示例：“如果我们将搜索栏从页眉移至Hero区域，那么搜索使用率将提升15%，因为用户在浏览时会更早看到它，减少了向上滚动搜索的摩擦。”
最小可检测效果（MDE）：
值得检测的最小变化是什么？转化率提升0.1%可能不值得投入工程资源。提升5%则值得。在测试前设定MDE，而非测试后。这决定了你所需的样本量。
样本量计算：
取决于：基准转化率、MDE、统计功效（通常为80%）、显著性水平（通常为95% / α=0.05）。不要猜测——使用公式或计算器。
常见场景快速参考（双侧检验，80%功效，95%显著性，两个变体）：
| 基准转化率 | 相对最小可检测效果（MDE） | 每个变体的样本量 |
|---|
| 5% | 20%（5% → 6%） | ~25,000 |
| 10% | 10%（10% → 11%） | ~14,500 |
| 10% | 20%（10% → 12%） | ~3,800 |
| 25% | 10%（25% → 27.5%） | ~4,800 |
| 50% | 5%（50% → 52.5%） | ~6,000 |
基准转化率越低、MDE越小，所需流量就越大。如果所需样本量超过月度流量，要么增大MDE（仅检测更大的效果）、延长测试时长，要么接受A/B测试不是合适的方法——改用定性研究。功效不足的测试会产生无结论的结果，浪费时间。
测试时长：
至少运行1-2个完整的周周期，以消除周内效应。季节性业务需更长时间。即使提前达到样本量，也不要运行少于一周——用户行为模式会随日期变化。
细分分析：
检查不同用户群体的差异化效果：新用户vs老用户、移动端vs桌面端、地域、套餐类型。整体中性的结果可能掩盖某一群体的显著正向效果和另一群体的显著负向效果。
约束指标：
定义哪些指标绝对不能变差。如果测试新的结账流程，约束指标可能包括：每用户收入、支持工单量、退货率。如果测试变体提升了转化率但增加了退货率，测试就是失败的。
常见错误：
- 在测试达到统计显著性之前查看结果（会提高假阳性率）
- 运行过多变体却不调整多重比较的阈值
- 忽略新奇效应（新事物只是因为新颖而获得更多点击——等待效果稳定）
- 因为早期结果“看起来决定性”而过早停止测试
- 同时运行多个测试时不考虑交互效应
- 当真正的问题是结构性问题时测试 cosmetic 变更