AB Testing Analyzer

A fully-featured intelligent AB testing analysis tool developed based on the AB testing module of "Data Analyst Brother's Ten Talks".

🎯 Skill Overview

This skill provides a complete AB testing solution from experimental design to result analysis, supporting multiple statistical testing methods, user segmentation analysis, and visual report generation.

✨ Core Features

🧪 Complete AB Testing Workflow
- Experimental design and sample size calculation
- Random group verification
- Conversion rate and retention rate analysis
- Statistical significance testing
📊 Comprehensive Statistical Methods
- t-test (independent samples, paired samples)
- Chi-square test (goodness of fit, independence)
- Confidence interval estimation
- Effect size calculation
- Multiple comparison correction
👥 Intelligent User Segmentation
- Value segmentation (high/low value customers)
- Demographic segmentation
- Behavioral pattern segmentation
- Custom segmentation strategies
- Interaction effect analysis
📈 Rich Visualization Features
- Conversion rate comparison chart
- Retention rate curve chart
- User segmentation heatmap
- Interaction effect visualization
- Statistical test result chart
🔧 Advanced Analysis Features
- Multivariate AB testing
- Bayesian AB testing
- Time series analysis
- Robustness check
- Causal inference support

🚀 Quick Start

1. Environment Requirements

bash

# Dependencies
pip install pandas numpy scipy matplotlib seaborn statsmodels

2. Basic Usage

python

from scripts.ab_test_analyzer import ABTestAnalyzer
from scripts.statistical_tests import StatisticalTests
from scripts.visualizer import ABTestVisualizer

# Initialize analyzer
analyzer = ABTestAnalyzer()
stats_tests = StatisticalTests()
visualizer = ABTestVisualizer()

# Load AB test data
data = analyzer.load_data('ab_test_data.csv')

# Basic conversion rate analysis
conversion_results = analyzer.analyze_conversion(
    data,
    group_col='Page Version',
    conversion_col='Purchased'
)

# Statistical significance testing
t_test_result = stats_tests.t_test(
    data,
    group_col='Page Version',
    metric_col='Purchased'
)

# Generate visualization report
fig = visualizer.plot_conversion_comparison(conversion_results)

3. Run Examples

bash

# Quick test
python quick_test.py

# Basic AB test example
python examples/basic_ab_test_example.py

# Advanced segmentation analysis example
python examples/advanced_segmentation_example.py

# Comprehensive analysis example
python examples/comprehensive_analysis_example.py

📁 Project Structure

ab-testing-analyzer/
├── SKILL.md                     # Detailed skill documentation
├── README.md                    # User guide (this file)
├── quick_test.py               # Quick function test
├── test_skill.py               # Complete test suite
│
├── scripts/                    # Core function modules
│   ├── __init__.py
│   ├── ab_test_analyzer.py         # Core AB test analysis
│   ├── statistical_tests.py        # Statistical testing module
│   ├── segment_analyzer.py         # User segmentation analysis
│   └── visualizer.py              # Visualization generator
│
└── examples/                   # Examples and data
    ├── sample_data/            # Sample data
    │   ├── sample_ab_test_data.csv
    │   └── sample_user_segments.csv
    ├── basic_ab_test_example.py     # Basic AB test example
    ├── advanced_segmentation_example.py # Advanced segmentation analysis example
    └── comprehensive_analysis_example.py # Comprehensive analysis example

💡 Key Features

1. Basic AB Testing Analysis

Conversion Rate Analysis

python

# Calculate conversion rates for each group
conversion_rates = analyzer.calculate_conversion_rates(
    data,
    group_col='Test Group',
    conversion_col='Conversion Status'
)

# Calculate lift and confidence interval
lift_analysis = analyzer.calculate_lift(
    conversion_rates,
    control_group='Control Group',
    test_group='Test Group'
)

Retention Rate Analysis

python

# Calculate retention rates
retention_rates = analyzer.calculate_retention_rates(
    data,
    group_col='Test Group',
    retention_col='retention_7'
)

# Visualize retention rate curves
fig = visualizer.plot_retention_curves(retention_rates)

2. Statistical Significance Testing

t-test

python

# Independent samples t-test
t_result = stats_tests.t_test(
    data,
    group_col='Page Version',
    metric_col='Purchase Amount',
    test_type='independent'
)

# Paired samples t-test
paired_t_result = stats_tests.t_test(
    before_after_data,
    group_col='User ID',
    metric_col='Behavioral Metric',
    test_type='paired'
)

Chi-square Test

python

# Goodness of fit test
chi2_goodness = stats_tests.chi_square_test(
    observed_data,
    expected_data,
    test_type='goodness_of_fit'
)

# Independence test
chi2_independence = stats_tests.chi_square_test(
    data,
    group_col='Test Group',
    outcome_col='Conversion Status',
    test_type='independence'
)

Effect Size Calculation

python

# Calculate Cohen's d
cohens_d = stats_tests.cohens_d(
    data,
    group_col='Test Group',
    metric_col='Conversion Status'
)

# Calculate Cramer's V
cramers_v = stats_tests.cramers_v(data, group_col, outcome_col)

3. User Segmentation Analysis

Value-based Segmentation

python

from scripts.segment_analyzer import SegmentAnalyzer

segment_analyzer = SegmentAnalyzer()

# Value-based user segmentation
value_segments = segment_analyzer.value_based_segmentation(
    data,
    value_col='Total Consumption Amount',
    n_tiers=3
)

# Segmentation conversion analysis
segment_conversion = segment_analyzer.segment_conversion_analysis(
    data,
    segment_col='Value Tier',
    group_col='Test Group',
    conversion_col='Conversion Status'
)

Interaction Effect Analysis

python

# Interaction effect between page version and user characteristics
interaction_analysis = segment_analyzer.interaction_analysis(
    data,
    group_col='Page Version',
    segment_col='Value Tier',
    outcome_col='Conversion Status'
)

# Visualize interaction effects
fig = visualizer.plot_interaction_effects(interaction_analysis)

4. Advanced Statistical Analysis

Bayesian AB Testing

python

# Bayesian AB test analysis
bayesian_result = analyzer.bayesian_ab_test(
    data,
    group_col='Test Group',
    conversion_col='Conversion Status',
    prior='jeffreys'
)

# Calculate win probability
win_probability = analyzer.calculate_win_probability(bayesian_result)

Multivariate Testing

python

# Multivariate AB testing (MVT)
mvt_result = analyzer.multivariate_test(
    data,
    group_cols=['Page Version', 'Button Color', 'Title Copy'],
    conversion_col='Conversion Status'
)

5. Visualization Reports

Basic Charts

python

# Conversion rate comparison chart
fig = visualizer.plot_conversion_comparison(
    conversion_data,
    title='AB Test Conversion Rate Comparison'
)

# Confidence interval chart
fig = visualizer.plot_confidence_intervals(
    statistical_results,
    metric='Conversion Rate'
)

# User segmentation heatmap
fig = visualizer.plot_segment_heatmap(
    segment_data,
    title='User Segmentation Conversion Rate Heatmap'
)

Interactive Dashboard

python

# Generate interactive dashboard
dashboard = visualizer.create_interactive_dashboard(
    analysis_results,
    output_file='ab_test_dashboard.html'
)

📊 Data Format

AB Testing Data Format (ab_test_data.csv)

csv

User ID,Test Group,Conversion Status,Retention Status,Total Consumption Amount,Gender,Age,Value Tier,Device Type
U001,Test Group,Yes,TRUE,299.99,Male,25,High Value,Mobile
U002,Control Group,No,FALSE,59.99,Female,32,Low Value,PC
U003,Test Group,Yes,TRUE,599.99,Male,28,High Value,Mobile
U004,Control Group,No,FALSE,199.99,Female,35,Medium Value,PC

User Segmentation Data Format (user_segments.csv)

csv

User ID,RFM Segment,Behavioral Segment,Demographic Segment,Comprehensive Segment
U001,High Value,Active User,Young Male,High Value Young User
U002,Low Value,Churned User,Mature Female,Need Re-engagement User

🎯 Application Scenarios

Product Optimization

Website revision effect evaluation
Function launch impact analysis
UI optimization testing
Performance improvement verification

Marketing Campaigns

Ad creative testing
Promotion strategy evaluation
Email marketing optimization
Social media campaign analysis

Operational Strategies

Pricing strategy testing
Recommendation algorithm optimization
User registration process improvement
Customer service strategy evaluation

⚙️ Advanced Configuration

Statistical Parameter Settings

python

# Set significance level
analyzer.set_significance_level(alpha=0.05)

# Set statistical power
analyzer.set_statistical_power(power=0.8)

# Set multiple comparison correction method
analyzer.set_multiple_comparison_correction(method='bonferroni')

Custom Segmentation Strategies

python

# Define custom segmentation rules
custom_segments = {
    'high_value': {'Total Consumption Amount': (500, float('inf'))},
    'medium_value': {'Total Consumption Amount': (100, 500)},
    'low_value': {'Total Consumption Amount': (0, 100)}
}

# Apply custom segmentation
segmented_data = segment_analyzer.apply_custom_segments(
    data,
    segment_rules=custom_segments
)

Advanced Visualization Configuration

python

# Set chart style
visualizer.set_style(style='seaborn', palette='viridis')

# Custom chart configuration
chart_config = {
    'figsize': (12, 8),
    'dpi': 300,
    'format': 'png',
    'style': 'professional'
}

fig = visualizer.plot_with_config(data, config=chart_config)

🐛 Frequently Asked Questions

Q: How to determine the appropriate sample size?

A: Use the sample size calculation function, considering effect size, significance level, and statistical power to calculate the minimum sample size.

Q: Does a p-value less than 0.05 mean the result is significant?

A: A p-value less than 0.05 indicates that the probability of observing the current result or a more extreme one is less than 5% assuming the null hypothesis is true. You need to combine effect size and practical significance to interpret the result.

Q: How to handle multiple comparison issues?

A: Use methods like Bonferroni correction or FDR correction to adjust p-values and avoid false positives.

Q: When to use Bayesian AB testing?

A: Consider using Bayesian methods when you need prior information, have a small sample size, or want probabilistic conclusions.

📈 Performance Optimization

Use vectorized operations to accelerate calculations
Implement incremental statistical updates
Adopt parallel computing for big data processing
Cache calculation results to avoid redundant computations
Optimize memory usage patterns

📚 Technical Principles

Basics of Statistical Testing

Based on hypothesis testing theory, judge the statistical significance of experimental results by calculating test statistics and p-values.

Central Limit Theorem

In large samples, the distribution of sample means approaches a normal distribution, providing a theoretical basis for many statistical methods.

Bayesian Inference

Combine prior information and observed data to perform parameter estimation and hypothesis testing through posterior distributions.

Multiple Comparison Correction

When conducting multiple hypothesis tests simultaneously, control the overall error rate to avoid an increase in false positive results.

🤝 Contribution Guidelines

Welcome to submit Issues and Pull Requests to improve this skill.

📄 License

MIT License

🎉 Get Started

Now that you have learned all the features of the AB Testing Analyzer skill, you can start using it:

bash

# Quick function verification
python quick_test.py

# Run examples
python examples/basic_ab_test_example.py

Enjoy your AB testing analysis journey! 🚀

ab-testing-analyzer

NPX Install

Tags

SKILL.md Content (Chinese)

AB Testing Analyzer

🎯 Skill Overview

✨ Core Features

🚀 Quick Start

1. Environment Requirements

2. Basic Usage

3. Run Examples

📁 Project Structure

💡 Key Features

1. Basic AB Testing Analysis

Conversion Rate Analysis

Retention Rate Analysis

2. Statistical Significance Testing

t-test

Chi-square Test

Effect Size Calculation

3. User Segmentation Analysis

Value-based Segmentation

Interaction Effect Analysis

4. Advanced Statistical Analysis

Bayesian AB Testing

Multivariate Testing

5. Visualization Reports

Basic Charts

Interactive Dashboard

📊 Data Format

AB Testing Data Format (ab_test_data.csv)

User Segmentation Data Format (user_segments.csv)

🎯 Application Scenarios

Product Optimization

Marketing Campaigns

Operational Strategies

⚙️ Advanced Configuration

Statistical Parameter Settings

Custom Segmentation Strategies

Advanced Visualization Configuration

🐛 Frequently Asked Questions

Q: How to determine the appropriate sample size?

Q: Does a p-value less than 0.05 mean the result is significant?

Q: How to handle multiple comparison issues?

Q: When to use Bayesian AB testing?

📈 Performance Optimization

📚 Technical Principles

Basics of Statistical Testing

Central Limit Theorem

Bayesian Inference

Multiple Comparison Correction

🤝 Contribution Guidelines

📄 License

🎉 Get Started