A/B Test Setup

1️⃣ Purpose & Scope

• Prevents "peeking"
• Enforces statistical power
• Blocks invalid hypotheses

2️⃣ Pre-Requisites

• A clear user problem
• Access to an analytics source
• Roughly estimated traffic volume

Hypothesis Quality Checklist

• Observation or evidence
• Single, specific change
• Directional expectation
• Defined audience
• Measurable success criteria

3️⃣ Hypothesis Lock (Hard Gate)

• Present the final hypothesis
• Specify:
  • Target audience
  • Primary metric
  • Expected direction of effect
  • Minimum Detectable Effect (MDE)

“Is this the final hypothesis we are committing to for this test?”

4️⃣ Assumptions & Validity Check (Mandatory)

• Traffic stability
• User independence
• Metric reliability
• Randomization quality
• External factors (seasonality, campaigns, releases)

• Warn the user
• Recommend delaying or redesigning the test

5️⃣ Test Type Selection

• A/B Test – single change, two variants
• A/B/n Test – multiple variants, higher traffic required
• Multivariate Test (MVT) – interaction effects, very high traffic
• Split URL Test – major structural changes

6️⃣ Metrics Definition

Primary Metric (Mandatory)

• Single metric used to evaluate success
• Directly tied to the hypothesis
• Pre-defined and frozen before launch

Secondary Metrics

• Provide context
• Explain why results occurred
• Must not override the primary metric

Guardrail Metrics

• Metrics that must not degrade
• Used to prevent harmful wins
• Trigger test stop if significantly negative

7️⃣ Sample Size & Duration

• Baseline rate
• MDE
• Significance level (typically 95%)
• Statistical power (typically 80%)

• Required sample size per variant
• Expected test duration

8️⃣ Execution Readiness Gate (Hard Stop)

• Hypothesis is locked
• Primary metric is frozen
• Sample size is calculated
• Test duration is defined
• Guardrails are set
• Tracking is verified

Running the Test

During the Test

• Monitor technical health
• Document external factors

• Stop early due to “good-looking” results
• Change variants mid-test
• Add new traffic sources
• Redefine success criteria

Analyzing Results

Analysis Discipline

• Do NOT generalize beyond the tested population
• Do NOT claim causality beyond the tested change
• Do NOT override guardrail failures
• Separate statistical significance from business judgment

Interpretation Outcomes

Documentation & Learning

Test Record (Mandatory)

• Hypothesis
• Variants
• Metrics
• Sample size vs achieved
• Results
• Decision
• Learnings
• Follow-up ideas

Refusal Conditions (Safety)

• Baseline rate is unknown and cannot be estimated
• Traffic is insufficient to detect the MDE
• Primary metric is undefined
• Multiple variables are changed without proper design
• Hypothesis cannot be clearly stated

Key Principles (Non-Negotiable)

• One hypothesis per test
• One primary metric
• Commit before launch
• No peeking
• Learning over winning
• Statistical rigor first

Final Reminder