Product Analytics

Quick Reference

A/B Test Evaluation

rules/ab-test-evaluation.md

## Experiment: [Name]

Hypothesis: If we [change], then [primary metric] will [direction] by [amount]
  because [evidence or reasoning].

Sample size: [N per variant] — calculated for MDE=[X%], power=80%, alpha=0.05
Duration: [Minimum weeks] — never stop early (peeking bias)

Results:
  Control:   [metric value]  n=[count]
  Treatment: [metric value]  n=[count]
  Lift:      [+/- X%]        p=[value]  95% CI: [lower, upper]

Decision: SHIP / EXTEND / KILL
  Rationale: [One sentence grounded in numbers, not gut feel]

• SHIP — p < 0.05, CI excludes zero, no guardrail regressions
• EXTEND — trending positive but underpowered (add runtime, not reanalysis)
• KILL — null result or guardrail degradation

rules/ab-test-evaluation.md

Cohort Retention

rules/cohort-retention.md

-- Day-N retention cohort query
SELECT
  DATE_TRUNC('week', first_seen)  AS cohort_week,
  COUNT(DISTINCT user_id)         AS cohort_size,
  COUNT(DISTINCT CASE
    WHEN activity_date = first_seen + INTERVAL '7 days'
    THEN user_id END) * 100.0
    / COUNT(DISTINCT user_id)     AS day_7_retention
FROM user_activity
GROUP BY 1
ORDER BY 1;

• Day 1: 40–60% is healthy
• Day 7: 20–35% is healthy
• Day 30: 10–20% is healthy
• Flat curve after day 30 = product-market fit signal

rules/cohort-retention.md

Funnel Analysis

rules/funnel-analysis.md

## Funnel: [Name] — [Date Range]

Stage 1: [Aware / Land]     → [N] users    (entry)
Stage 2: [Activate / Sign]  → [N] users    ([X]% from stage 1)
Stage 3: [Engage / Use]     → [N] users    ([X]% from stage 2)  ← biggest drop
Stage 4: [Convert / Pay]    → [N] users    ([X]% from stage 3)

Overall conversion: [X]%
Biggest drop-off:  Stage 2→3 ([X]% loss) — investigate first

rules/funnel-analysis.md

Statistical Foundations

references/stats-cheat-sheet.md

Common Pitfalls

1. Peeking — stopping an experiment early because results look good inflates false-positive rate. Commit to a runtime before launch.
2. Multiple comparisons — testing 10 metrics at p < 0.05 means ~1 false positive by chance. Apply Bonferroni correction or pre-register your primary metric.
3. Sample Ratio Mismatch (SRM) — if variant group sizes differ from expected split by > 1%, your experiment is broken. Fix before analyzing results.
4. Novelty effect — new features get inflated engagement in week 1. Run experiments long enough to see settled behavior (minimum 2 full business cycles).
5. Simpson's paradox — aggregate results can reverse when segmented. Always check results by key segments (device, plan tier, geography).

Ship / Extend / Kill Framework

Related Skills

• ork:product-frameworks

  — OKRs, KPI trees, RICE prioritization, PRD templates

• ork:metrics-instrumentation

  — Event naming, metric definition, alerting setup

• ork:brainstorm

  — Generate hypotheses and experiment ideas

• ork:assess

  — Evaluate product quality and risks

References

• rules/ab-test-evaluation.md

  — Hypothesis, sample size, significance, decision matrix

• rules/cohort-retention.md

  — Cohort types, retention curves, SQL patterns

• rules/funnel-analysis.md

  — Stage mapping, drop-off identification, optimization

• references/stats-cheat-sheet.md

  — Formulas, test selection, power analysis