For a fintech SaaS company, does moving premium tier benefits above the fold on the pricing page increase the conversion rate to a paid plan, compared to the current page layout? And does this change come at the cost of lower revenue per converting user?
If premium tier benefits are moved above the fold (Variant B), then conversion rate to a paid plan will increase by at least 2 percentage points compared to the current layout (Control), because the value proposition becomes immediately visible without scrolling.
- Primary metric: Conversion rate to a paid plan.
- Guardrail metric: Average revenue per converting user — should not meaningfully decrease.
Small changes to a pricing page can meaningfully affect both signups and revenue per customer. Testing this systematically, rather than guessing, avoids shipping a change that looks good on one metric while quietly damaging another.
This project uses a simulated dataset of 11,000 users randomly and deterministically assigned to Control or Treatment groups, with conversion and revenue outcomes generated to reflect a designed experiment: a real effect on conversion rate and no designed effect on revenue per converter. Real A/B test data is proprietary to individual companies and not publicly available, so this project simulates a realistic, statistically grounded scenario instead — enabling verification that the analysis correctly detects a known, designed effect (conversion rate) while correctly failing to detect a known non-effect (revenue per converter).
The hypothesis — that moving premium tier benefits above the fold would increase conversion rate by at least 2 percentage points — is confirmed. The observed lift (2.85 points) exceeds the stated Minimum Detectable Effect, and the result is statistically significant.
This experiment used 11,000 total users (5,481 Control, 5,519 Treatment), exceeding the 10,540 required by the power analysis — confirming the experiment was adequately powered to detect the hypothesized effect before it was run, not just in hindsight.
Conversion rate increased from 14.85% (Control) to 17.70% (Treatment) — a lift of 2.85 percentage points (95% CI: 1.47 to 4.23 points), statistically significant (χ² = 16.19, p = 0.0001).
Average revenue per converting user was nearly identical between groups ($119.90 Control vs. $119.99 Treatment), not statistically significant (p = 0.93), with a negligible effect size (Cohen's d = 0.004).
Before trusting any result, the random group split was validated: observed split (5,481 / 5,519) closely matched the expected 50/50 split (p = 0.72) — confirming randomization was healthy.
Both conditions for a clean win are met: the primary metric improved by a statistically and practically meaningful margin, and the guardrail metric showed no evidence of harm.
Per 10,000 pricing page visitors:
- Conversions: approximately 285 additional conversions (95% CI: 147 to 423).
- Revenue per converter: no meaningful change — the additional conversions represent genuinely incremental revenue, not conversions "bought" by discounting toward cheaper plans.
- Combined effect: approximately $34,200 in additional revenue per 10,000 visitors.
- Novelty effect: a short-term lift doesn't guarantee the effect persists once the novelty of a new layout fades.
- Single time-boxed test: seasonality and day-of-week effects aren't modeled.
- No early-peeking simulation: reflects one complete dataset collected to the full required sample size, not a simulation of premature stopping.
- Two-sided guardrail test: stricter industry practice sometimes uses a one-sided non-inferiority test for guardrail metrics; this project uses a standard two-sided test as a simplification.
- Simulated data: doesn't account for traffic source, device type, or new vs. returning visitor status.
- Testing too many changes at once: this experiment isolates a single change (premium tier placement) to ensure any observed effect can be attributed to that change specifically — a real test that simultaneously changes copy, layout, and color would make it impossible to isolate what actually drove the result.
- Verifying tracking/instrumentation: in a real production experiment, results are only as trustworthy as the underlying event tracking. This project uses simulated data, so this risk doesn't apply here, but it's a critical first check before trusting any real experiment's results.
- Power analysis (
src/power.py) — required sample size calculated usingNormalIndPowerand Cohen's h (the correct effect-size measure for comparing two proportions). - Simulation (
src/simulate.py) — users deterministically assigned to Control/Treatment via MD5 hash of user ID (ensuring reproducible, consistent assignment); conversion and revenue outcomes simulated per group. - Sample Ratio Mismatch check (
src/simulate.py) — chi-square goodness-of-fit test confirming the actual split matched the intended 50/50 ratio before trusting downstream results. - Statistical testing (
src/analysis.py) — chi-square test on conversion rate (primary metric) with 95% confidence interval on the lift; t-test with Cohen's d effect size on revenue per converter (guardrail metric).
ab-testing-pricing-page/ ├── notebooks/ │ └── 01_ab_test_analysis.ipynb # full narrated walkthrough ├── src/ │ ├── power.py # sample size / power analysis │ ├── simulate.py # experiment simulation + SRM check │ └── analysis.py # statistical tests + charts └── outputs/ └── figures/ ├── conversion_rate_comparison.png └── revenue_distribution_comparison.png
# 1. Set up environment
python -m venv venv
source venv/Scripts/activate # Windows Git Bash
pip install -r requirements.txt
# 2. Run the full pipeline
python src/power.py
python src/simulate.py
python src/analysis.pyAlternatively, open notebooks/01_ab_test_analysis.ipynb for a
narrated, cell-by-cell walkthrough with inline charts and explanations.
Python, pandas, NumPy, Matplotlib, Seaborn, SciPy, statsmodels