Upload your experiment results, get a defensible ship / no-ship decision. A web app that turns a raw CSV of A/B test data into a recommendation backed by a fixed-horizon test, an always-valid sequential test (mSPRT), and a power analysis — plus a pre-experiment sample-size planner.
🔗 Live demo: https://ab-framework.streamlit.app — opens on a populated decision dashboard (no upload needed); switch between a clear winner, a null, and a borderline result, or drop in your own CSV.
Product managers, growth analysts, and data scientists who run experiments and need to answer three questions without re-deriving the statistics each time:
- Did it work? — lift, significance, and a confidence interval you can quote.
- Can I stop early? — an always-valid test that makes peeking safe.
- Should I ship it? — a transparent verdict tied to your effect-size bar.
- Upload a CSV with columns
variant(control/treatment),user_id,conversion(0/1), and optionalrevenue. - Headline stats — sample size and conversion rate per variant, absolute and relative lift, two-proportion z-test p-value, and 95% CIs on both lifts.
- Power — achieved power at the observed effect, and (more usefully) power to detect your minimum detectable effect at the sample size you reached.
- Always-valid sequential panel — an mSPRT confidence sequence plotted over the life of the experiment, telling you whether and when you crossed a valid stopping boundary.
- Decision — a
Ship/Do Not Ship/Keep Runningverdict with a full audit trail of every criterion. - Planner tab — enter baseline rate, MDE, alpha, and power; get the required sample size per variant before you launch.
Three example datasets in examples/ land instantly on a clear
winner, a (well-powered) null, and a borderline result.
Fixed-horizon test. Conversion rates are means of Bernoulli outcomes, so a
two-proportion z-test applies. The hypothesis test uses the pooled-variance
statistic (correct under H0: p_t = p_c); the confidence interval on the
difference uses the unpooled variance. Cross-checked against statsmodels in the
test suite.
Always-valid inference (the part most A/B tools get wrong). Classical p-values
are only valid if you fix the sample size in advance — "peeking" and stopping when
you see significance inflates the false-positive rate badly. This app implements
the mixture Sequential Probability Ratio Test (mSPRT) with a zero-mean normal
mixing distribution over the effect (Johari et al. 2017/2022; Lindon & Malek
2022). The mixture likelihood ratio Λₙ is a test martingale, so by Ville's
inequality the anytime-valid p-value min(1, 1/Λₙ) can be monitored continuously
with type-I error controlled at any stopping time. Inverting the test yields a
confidence sequence — an interval that covers the true lift at all sample
sizes simultaneously. The test suite verifies the type-I guarantee by simulation.
Decision logic. Ship requires a positive effect that is both statistically
resolved (fixed-horizon significance or a crossed always-valid boundary) and
practically meaningful (observed lift ≥ your MDE). A significant-but-tiny effect,
or a well-powered null, returns Do Not Ship. Everything else is Keep Running.
pip install -r requirements.txt
streamlit run streamlit_app.py
# regenerate the example datasets (optional):
python examples/generate_examples.py
# run the test suite:
pytest -qThe repo is deploy-ready: streamlit_app.py and requirements.txt live at the
root. Go to https://share.streamlit.io, connect this repository, pick the main
branch and streamlit_app.py, and claim the subdomain ab-framework. Every push
to main auto-redeploys.
streamlit_app.py # the app (UI only; stats live in src/)
src/
ab_stats.py # two-proportion test, CIs, power, sample size
sequential.py # mSPRT: anytime-valid p-value + confidence sequence
decision.py # ship / no-ship / keep-running logic
examples/ # winner / null / borderline CSVs + generator
tests/ # pytest: stats vs statsmodels, mSPRT type-I control
.github/workflows/ci.yml
Python · Streamlit · scipy.stats · statsmodels · pandas · plotly · pytest