ab-test-calculator

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

A/B Test Calculator

A/B测试计算器

Statistical significance testing for A/B experiments with power analysis and sample size estimation.

为A/B实验提供统计显著性检验，包含功效分析与样本量估计功能。

Features

功能特性

Significance Testing: Chi-square, Z-test, T-test for conversions
Sample Size Estimation: Calculate required samples for desired power
Power Analysis: Determine test power given sample size
Confidence Intervals: Calculate CIs for conversion rates
Multiple Variants: Support A/B/n testing
Bayesian Analysis: Probability to beat baseline

显著性检验：用于转化率分析的Chi-square、Z-test、T-test
样本量估计：计算达到预期功效所需的样本量
功效分析：根据给定样本量确定测试功效
置信区间：计算转化率的置信区间（CIs）
多变体支持：支持A/B/n测试
贝叶斯分析：变体击败基准版本的概率

Quick Start

快速开始

python

from ab_test_calc import ABTestCalculator

calc = ABTestCalculator()

python

from ab_test_calc import ABTestCalculator

calc = ABTestCalculator()

Test significance

检验显著性

result = calc.test_significance( control_visitors=10000, control_conversions=500, variant_visitors=10000, variant_conversions=550 )

print(f"Significant: {result['significant']}") print(f"P-value: {result['p_value']:.4f}") print(f"Lift: {result['lift']:.2%}")

undefined

result = calc.test_significance( control_visitors=10000, control_conversions=500, variant_visitors=10000, variant_conversions=550 )

print(f"Significant: {result['significant']}") print(f"P-value: {result['p_value']:.4f}") print(f"Lift: {result['lift']:.2%}")

undefined

CLI Usage

CLI使用方法

bash

undefined

bash

undefined

Test significance

检验显著性

python ab_test_calc.py --test 10000 500 10000 550

Calculate sample size

计算样本量

python ab_test_calc.py --sample-size --baseline 0.05 --mde 0.10 --power 0.8

Power analysis

功效分析

python ab_test_calc.py --power-analysis --baseline 0.05 --mde 0.10 --samples 5000

Bayesian analysis

贝叶斯分析

python ab_test_calc.py --bayesian 10000 500 10000 550

Multiple variants

多变体测试

python ab_test_calc.py --test-multi 10000 500 10000 550 10000 520

undefined

python ab_test_calc.py --test-multi 10000 500 10000 550 10000 520

undefined

API Reference

API参考

ABTestCalculator Class

ABTestCalculator类

python

class ABTestCalculator:
    def __init__(self, alpha: float = 0.05)

    # Significance testing
    def test_significance(self, control_visitors: int, control_conversions: int,
                         variant_visitors: int, variant_conversions: int,
                         test: str = "chi_square") -> dict

    # Sample size calculation
    def calculate_sample_size(self, baseline_rate: float,
                             minimum_detectable_effect: float,
                             power: float = 0.8,
                             alpha: float = 0.05) -> dict

    # Power analysis
    def calculate_power(self, baseline_rate: float,
                       minimum_detectable_effect: float,
                       sample_size: int,
                       alpha: float = 0.05) -> dict

    # Confidence interval
    def confidence_interval(self, visitors: int, conversions: int,
                           confidence: float = 0.95) -> dict

    # Bayesian analysis
    def bayesian_analysis(self, control_visitors: int, control_conversions: int,
                         variant_visitors: int, variant_conversions: int,
                         simulations: int = 100000) -> dict

    # Multiple variants
    def test_multiple_variants(self, control: tuple, variants: list,
                              correction: str = "bonferroni") -> dict

    # Duration estimation
    def estimate_duration(self, daily_visitors: int, baseline_rate: float,
                         minimum_detectable_effect: float,
                         power: float = 0.8) -> dict

python

class ABTestCalculator:
    def __init__(self, alpha: float = 0.05)

    # 显著性检验
    def test_significance(self, control_visitors: int, control_conversions: int,
                         variant_visitors: int, variant_conversions: int,
                         test: str = "chi_square") -> dict

    # 样本量计算
    def calculate_sample_size(self, baseline_rate: float,
                             minimum_detectable_effect: float,
                             power: float = 0.8,
                             alpha: float = 0.05) -> dict

    # 功效分析
    def calculate_power(self, baseline_rate: float,
                       minimum_detectable_effect: float,
                       sample_size: int,
                       alpha: float = 0.05) -> dict

    # 置信区间计算
    def confidence_interval(self, visitors: int, conversions: int,
                           confidence: float = 0.95) -> dict

    # 贝叶斯分析
    def bayesian_analysis(self, control_visitors: int, control_conversions: int,
                         variant_visitors: int, variant_conversions: int,
                         simulations: int = 100000) -> dict

    # 多变体测试
    def test_multiple_variants(self, control: tuple, variants: list,
                              correction: str = "bonferroni") -> dict

    # 测试时长估计
    def estimate_duration(self, daily_visitors: int, baseline_rate: float,
                         minimum_detectable_effect: float,
                         power: float = 0.8) -> dict

Test Methods

测试方法

Chi-Square Test (Default)

Chi-Square检验（默认）

Best for comparing conversion rates between groups.

python

result = calc.test_significance(
    control_visitors=10000,
    control_conversions=500,
    variant_visitors=10000,
    variant_conversions=550,
    test="chi_square"
)

最适合用于对比不同组之间的转化率。

python

result = calc.test_significance(
    control_visitors=10000,
    control_conversions=500,
    variant_visitors=10000,
    variant_conversions=550,
    test="chi_square"
)

Z-Test for Proportions

比例Z检验

Good for large sample sizes.

python

result = calc.test_significance(
    control_visitors=10000,
    control_conversions=500,
    variant_visitors=10000,
    variant_conversions=550,
    test="z_test"
)

适用于大样本量场景。

python

result = calc.test_significance(
    control_visitors=10000,
    control_conversions=500,
    variant_visitors=10000,
    variant_conversions=550,
    test="z_test"
)

Sample Size Estimation

样本量估计

Calculate the number of visitors needed per variant:

python

result = calc.calculate_sample_size(
    baseline_rate=0.05,          # Current conversion rate (5%)
    minimum_detectable_effect=0.10,  # 10% relative improvement
    power=0.8,                   # 80% power
    alpha=0.05                   # 5% significance level
)

计算每个变体所需的访客数量：

python

result = calc.calculate_sample_size(
    baseline_rate=0.05,          # 当前转化率（5%）
    minimum_detectable_effect=0.10,  # 10%的相对提升
    power=0.8,                   # 80%的功效
    alpha=0.05                   # 5%的显著性水平
)

Returns:

返回结果：

{ "sample_size_per_variant": 31234, "total_sample_size": 62468, "baseline_rate": 0.05, "expected_variant_rate": 0.055, "minimum_detectable_effect": 0.10, "power": 0.8, "alpha": 0.05 }

undefined

{ "sample_size_per_variant": 31234, "total_sample_size": 62468, "baseline_rate": 0.05, "expected_variant_rate": 0.055, "minimum_detectable_effect": 0.10, "power": 0.8, "alpha": 0.05 }

undefined

Power Analysis

功效分析

Calculate the probability of detecting an effect:

python

result = calc.calculate_power(
    baseline_rate=0.05,
    minimum_detectable_effect=0.10,
    sample_size=25000,
    alpha=0.05
)

计算检测到效应的概率：

python

result = calc.calculate_power(
    baseline_rate=0.05,
    minimum_detectable_effect=0.10,
    sample_size=25000,
    alpha=0.05
)

Returns:

返回结果：

{ "power": 0.72, "interpretation": "72% chance of detecting the effect if it exists" }

undefined

{ "power": 0.72, "interpretation": "72% chance of detecting the effect if it exists" }

undefined

Bayesian Analysis

贝叶斯分析

Get probability that variant beats control:

python

result = calc.bayesian_analysis(
    control_visitors=10000,
    control_conversions=500,
    variant_visitors=10000,
    variant_conversions=550
)

获取变体击败对照组的概率：

python

result = calc.bayesian_analysis(
    control_visitors=10000,
    control_conversions=500,
    variant_visitors=10000,
    variant_conversions=550
)

Returns:

返回结果：

{ "prob_variant_better": 0.9523, "prob_control_better": 0.0477, "expected_lift": 0.098, "credible_interval_95": [0.02, 0.18] }

undefined

{ "prob_variant_better": 0.9523, "prob_control_better": 0.0477, "expected_lift": 0.098, "credible_interval_95": [0.02, 0.18] }

undefined

Multiple Variant Testing

多变体测试

Test multiple variants with correction for multiple comparisons:

python

result = calc.test_multiple_variants(
    control=(10000, 500),          # (visitors, conversions)
    variants=[
        (10000, 550),              # Variant A
        (10000, 520),              # Variant B
        (10000, 480)               # Variant C
    ],
    correction="bonferroni"        # or "holm", "none"
)

通过多重比较校正来测试多个变体：

python

result = calc.test_multiple_variants(
    control=(10000, 500),          # （访客数，转化数）
    variants=[
        (10000, 550),              # 变体A
        (10000, 520),              # 变体B
        (10000, 480)               # 变体C
    ],
    correction="bonferroni"        # 可选值："holm"、"none"
)

Returns:

返回结果：

{ "control": {"visitors": 10000, "conversions": 500, "rate": 0.05}, "variants": [ {"visitors": 10000, "conversions": 550, "rate": 0.055, "lift": 0.10, "p_value": 0.012, "significant": True}, ... ], "winner": "Variant A", "correction_method": "bonferroni" }

undefined

undefined

Output Format

输出格式

Significance Test Result

显著性检验结果

python

{
    "significant": True,
    "p_value": 0.0234,
    "control_rate": 0.05,
    "variant_rate": 0.055,
    "lift": 0.10,
    "lift_absolute": 0.005,
    "confidence_interval": {
        "lower": 0.02,
        "upper": 0.18
    },
    "test_method": "chi_square",
    "alpha": 0.05,
    "recommendation": "Variant shows significant improvement"
}

python

{
    "significant": True,
    "p_value": 0.0234,
    "control_rate": 0.05,
    "variant_rate": 0.055,
    "lift": 0.10,
    "lift_absolute": 0.005,
    "confidence_interval": {
        "lower": 0.02,
        "upper": 0.18
    },
    "test_method": "chi_square",
    "alpha": 0.05,
    "recommendation": "Variant shows significant improvement"
}

Example Workflows

示例工作流

Pre-Test Planning

测试前规划

python

calc = ABTestCalculator()

python

calc = ABTestCalculator()

1. Estimate required sample size

1. 估计所需样本量

sample = calc.calculate_sample_size( baseline_rate=0.03, # Current 3% conversion minimum_detectable_effect=0.15, # Want to detect 15% lift power=0.8 ) print(f"Need {sample['sample_size_per_variant']} visitors per variant")

sample = calc.calculate_sample_size( baseline_rate=0.03, # 当前转化率3% minimum_detectable_effect=0.15, # 希望检测到15%的提升 power=0.8 ) print(f"每个变体需要{sample['sample_size_per_variant']}位访客")

2. Estimate test duration

2. 估计测试时长

duration = calc.estimate_duration( daily_visitors=5000, baseline_rate=0.03, minimum_detectable_effect=0.15 ) print(f"Test will take ~{duration['days']} days")

undefined

duration = calc.estimate_duration( daily_visitors=5000, baseline_rate=0.03, minimum_detectable_effect=0.15 ) print(f"测试大约需要{duration['days']}天")

undefined

Post-Test Analysis

测试后分析

python

calc = ABTestCalculator()

python

calc = ABTestCalculator()

1. Test significance

1. 检验显著性

result = calc.test_significance( control_visitors=15000, control_conversions=450, variant_visitors=15000, variant_conversions=525 )

2. Get Bayesian probability

2. 获取贝叶斯概率

bayes = calc.bayesian_analysis(15000, 450, 15000, 525)

print(f"P-value: {result['p_value']:.4f}") print(f"Lift: {result['lift']:.2%}") print(f"Probability variant wins: {bayes['prob_variant_better']:.1%}")

undefined

bayes = calc.bayesian_analysis(15000, 450, 15000, 525)

print(f"P值: {result['p_value']:.4f}") print(f"提升率: {result['lift']:.2%}") print(f"变体获胜概率: {bayes['prob_variant_better']:.1%}")

undefined

Dependencies

依赖项

scipy>=1.10.0
numpy>=1.24.0
statsmodels>=0.14.0

scipy>=1.10.0
numpy>=1.24.0
statsmodels>=0.14.0