A/B Testing & Causal Inference

🧭 New here? Start with OVERVIEW.md — a one-page tour.

Designing and analysing experiments the way a data/analytics team actually does: from power analysis before launch, through the statistical test, to the harder question of causal inference when a clean randomized test isn't available.

📄 Full write-up: REPORT.md · 🧭 One-page tour: OVERVIEW.md · 📐 Methods explained (plain English): METHODOLOGY.md

What's inside: experiment design (power), frequentist + Bayesian tests, bootstrap intervals, subgroup analysis, a full causal-inference section, an interactive Streamlit calculator, and a pytest suite.

Why this project

Anyone can compute a p-value. The valuable skills are: sizing an experiment before running it, checking its validity (sample-ratio mismatch), quantifying an effect with a confidence interval (not just "significant / not"), and — crucially — knowing what to do when assignment wasn't random, where a naive comparison gives the wrong answer. This project demonstrates all of that.

Because the data is synthetic with a documented true effect, every method can be judged on whether it actually recovers the truth.

The experiment

An e-commerce store tests a new checkout call-to-action. 40,000 visitors are randomized 50/50 to control (old CTA) or treatment (new CTA). We record conversion and revenue, plus two covariates (device, prior purchases). The data-generating process (in generate_data.py) sets the ground truth: a +0.15 log-odds treatment effect (~+2pp lift at this base rate) and a +$3 lift in average order value.

Results so far

Conversion (two-proportion z-test):

Group	Conversion
Control	12.76%
Treatment	14.72%

• Absolute lift +1.97pp, 95% CI [+1.29pp, +2.64pp] — the interval excludes 0, and it contains the true ~+2pp effect. ✅
• Relative lift +15.4%, z = 5.72, p ≈ 1e-8 (highly significant).
• Revenue per visitor +$1.59 (p ≈ 9e-12), and a sample-ratio-mismatch check confirms the split is a valid 50/50.

Power analysis shows, for a 12.8% baseline at 80% power, you'd need ~8,200 visitors/arm to detect a +1.5pp lift (and ~18,000/arm for +1.0pp) — so this test was comfortably powered.

Bayesian view (the business decision)

A frequentist p-value says "significant"; a Bayesian analysis answers what stakeholders actually ask — how likely is the new version better, and what's the risk? Using a Beta-Binomial model:

• P(treatment > control) = 100%, posterior lift +1.97pp, 95% credible interval [+1.29pp, +2.64pp] (it matches the frequentist interval).
• Expected loss of shipping treatment ≈ 0% vs. ~1.97pp of lost conversion if we wrongly keep control → clear decision: ship it.

Bootstrap confidence intervals

The bootstrap makes no distributional assumptions — useful for the skewed revenue metric. Resampling 10,000 times gives a 95% CI of [+1.29pp, +2.63pp] for conversion lift and [+$1.15, +$2.04] for revenue per visitor — both excluding zero, and both agreeing with the analytic intervals.

Does it work for everyone? (segment analysis)

Splitting by device, desktop shows a +2.42pp lift and mobile +1.57pp — but a proper interaction test (treatment × device in a logistic regression) gives p = 0.50, so that gap is not real: the treatment works similarly on both. This guards against a common mistake — reading two separate significant results as a difference between them.

Causal inference — when there's no clean A/B test

The hardest, most valuable case: a new loyalty feature that was not randomly assigned — high-value customers (more prior purchases) adopted it far more, and they already convert better. So a naive "adopters vs non-adopters" comparison is confounded and overstates the effect.

Using a confounded dataset with a known true effect of +7.25pp, three standard corrections all recover it — while the naive number is badly wrong:

Method	Estimate	Error vs. true
Naive difference (biased)	+12.69pp	+5.44pp
Regression adjustment (g-computation)	+5.98pp	−1.27pp
Propensity-score matching	+4.62pp	−2.63pp
Difference-in-differences	+7.97pp	+0.72pp

Propensity-score matching is only credible when the groups actually overlap — they do here:

Takeaway: the naive comparison would have oversold the feature by ~75%. Regression adjustment, matching, and difference-in-differences each strip out the confounding and land near the truth — with difference-in-differences (which uses the pre/post periods) closest of all.

Run it

pip install -r requirements.txt
python generate_data.py     # create the synthetic experiment -> data/
python ab_analysis.py       # frequentist analysis + charts
python power_analysis.py    # sample-size & power curve
python bayesian_ab.py       # Bayesian (Beta-Binomial) decision analysis
python bootstrap_analysis.py# bootstrap confidence intervals
python segment_analysis.py  # subgroup lift + interaction test
python generate_confounded.py  # confounded observational data + known truth
python causal_analysis.py   # regression adj., PSM, difference-in-differences
python cuped_analysis.py    # CUPED variance reduction
pytest                      # run the test suite
streamlit run app.py        # interactive A/B calculator + sample-size planner

Tech

Python · pandas · NumPy · SciPy · statsmodels · matplotlib

---

Author: Muhammad Nasiruddin. Part of a data-science portfolio spanning classification, clustering, forecasting, and experimentation.