Anytime-valid A/B analyzer that holds the false-positive rate under 1.5% while you peek at the dashboard continuously, where naive fixed-horizon testing leaks to 23% at 15 looks, and cuts required sample size with CUPED variance reduction.
- Peeking at a dashboard and stopping early inflates false positives. Measured on A/A data with no true effect, naive fixed-horizon testing checked at 15 looks reaches a 23% false-positive rate against a nominal 5%.
- An anytime-valid test fixes it. The mSPRT holds the false-positive rate under 1.5% across up to 15 looks, so you can monitor continuously and stop the moment evidence accumulates.
- CUPED gets to significance with fewer users. Using a pre-period covariate removed 50% of the metric variance on the demo experiment, cutting the standard error from 0.223 to 0.135.
A false experiment win is expensive. Someone ships a change that does nothing, other teams build on it, and when it is eventually caught the whole experiment program loses credibility. The most common way a false win is manufactured is also the most human: watching a live dashboard and stopping the test the first time p drops below 0.05. A fixed-horizon t-test is only valid for one analysis at a pre-committed sample size, and checking it repeatedly is not that. On A/A data generated here with no true effect, the naive stop-when-significant rule produced a false positive 23% of the time across 15 looks, against the 5% the team believed they were running at.
ab-test-guard analyzes an experiment through three lenses at once: the fixed-horizon Welch test a team already uses, a CUPED-adjusted version that removes pre-experiment variance, and an anytime-valid mSPRT that can be checked as often as you like. The mSPRT statistic is a non-negative martingale under the null, so by Ville's inequality the chance it ever crosses the decision boundary is bounded by alpha regardless of how many times you look. The guard compares the fixed-horizon and anytime-valid verdicts and raises a warning when they disagree below the pre-registered sample size, which is the structural fingerprint of a peeked-into-existence result.
Every number in this README comes from running bench/run_benchmark.py in the
build container: the false-positive curve, the power comparison, and the CUPED
variance reduction. The Welch implementation is checked against scipy to nine
decimals in the test suite, and the CUPED reduction is verified to equal the
squared metric-covariate correlation. All data is synthetic and labeled as such.
Naive fixed-horizon testing (blue) climbs steadily as you look more often. The anytime-valid mSPRT (orange) stays flat near the floor, well under the nominal 5% line, no matter how many times you peek.
flowchart LR
A[Per-unit control and treatment data] --> B[Welch t-test]
A --> C[CUPED adjust]
C --> D[Welch on adjusted metric]
A --> E[mSPRT anytime-valid]
B --> F[Guard]
D --> F
E --> F
F -->|fixed sig but anytime not,<br/>below target N| W[Peeking warning]
F --> G[(Analysis audit log)]
subgraph cov["Pre-period covariate"]
X[Prior-period metric]
end
X -.feeds.-> C
The guard is the integration point: it collects all three results, decides whether they agree, and writes the verdict plus any warnings to the audit log.
| Technology | Role in this project | Why chosen here |
|---|---|---|
| numpy | Array math for all estimators | Explicit, visible formulas; see ADR-0002. |
| scipy.stats | t and normal reference distributions | Also the oracle the tests check Welch against. |
| pandas | CSV experiment loading in the CLI | Simple arm-and-metric table ingestion. |
| matplotlib | The false-positive-rate chart | Real PNG from the same arrays as the results JSON. |
| PyYAML | Experiment definitions and guardrails | Experiments are config, not code. |
| SQLite (stdlib) | Analysis audit log | Zero-ops; DDL kept in sql/schema.sql for review. |
Prerequisites: Python 3.12.
git clone https://github.com/NavyasriAmand/ab-test-guard.git
cd ab-test-guard
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
# run the tests (Welch is checked against scipy)
pytest --cov=ab_test_guard --cov-report=term
# measure the peeking effect yourself
abguard simulate --n 2000 --looks 10 --trials 2000
# analyze a synthetic experiment end to end
python bench/generate_experiment.py --n 4000 --effect 0.8 --rho 0.7 --out bench/experiment.csv --seed 11
abguard analyze --experiment checkout_button --data bench/experiment.csv
# reproduce the full benchmark and regenerate the chart
python bench/run_benchmark.py --n 2000 --max-looks 15 --trials 1500abguard analyze exits non-zero when the fixed-horizon and anytime-valid
verdicts disagree, so it can gate an early-stop decision in a notebook or CI
step rather than leaving it to eyeballing.
Methodology: all figures are from Monte Carlo simulations run in the build container with fixed seeds. The false-positive rates come from A/A experiments (both arms drawn from the same distribution, so any rejection is a false positive), with 1,500 trials per look. Power comes from A/B experiments with a true standardized effect of 0.1 over 1,500 trials. n is 2,000 units per arm.
False-positive rate by number of looks (nominal alpha = 5%):
| Looks | Naive fixed-horizon | Anytime-valid mSPRT |
|---|---|---|
| 1 | 5.1% | 0.1% |
| 5 | 13.3% | 0.4% |
| 10 | 18.2% | 1.1% |
| 15 | 23.3% | 1.5% |
Power at the full sample (true effect present):
| Method | Power |
|---|---|
| Fixed-horizon, single look | 87.7% |
| Anytime-valid, single look | 54.3% |
| Anytime-valid, allowed to peek across 15 looks | 60.6% |
CUPED variance reduction on the demo experiment (covariate correlation ~0.7): 50.2%, which cut the effect standard error from 0.223 to 0.135.
Where the trade sits: the anytime-valid method has less power per single look, which is the price of its peeking robustness. It buys the power back by letting you keep looking and stop as soon as the evidence is there, which a fixed-horizon test cannot do without breaking its own guarantee. Naming this trade instead of hiding it is the point.
- ADR-0001: Anytime-valid inference over fixed-horizon testing
- ADR-0002: numpy and scipy for the statistics, not statsmodels (the boring choice)
Multiple-comparison correction across many simultaneous metrics is not handled. The guard analyzes one primary metric plus named guardrails; it does not adjust for testing dozens of metrics at once. Add a Benjamini-Hochberg or Bonferroni layer over the per-metric anytime-valid p-values only when an experiment routinely reports more than a handful of metrics, since that is when the family-wise error across metrics starts to dominate the per-metric error this project already controls.
- All configuration (alpha, tau, cost model, database and experiment paths) is
read from environment variables through a frozen
Settingsobject; nothing is hardcoded. - The audit log stores effect estimates and verdicts, not raw user records. If per-unit data is ever persisted it should inherit the same access controls and retention as other experiment data; this is stated rather than assumed.
- No secrets are read or logged. Structured JSON logs carry the experiment name and verdict flags only.
- CI runs the test suite with a coverage floor and a separate simulation gate that fails if the anytime-valid false-positive control regresses.
- Metric has no variance. The Welch and proportion tests raise a clear error rather than dividing by zero, so a degenerate metric fails fast.
- Covariate is post-treatment or constant. A constant covariate yields theta 0 and no adjustment (safe). A post-treatment covariate would bias the estimate; the CUPED module documents that the covariate must be pre-treatment, and the experiment definition names a prior-period metric to enforce this by convention.
- Sample below the pre-registered size. The guard warns and, when the fixed-horizon test is significant but the anytime-valid one is not, flags a likely false win rather than letting the early result stand unqualified.
- Audit database unavailable. Analysis does not depend on the store; a store outage degrades the record, not the verdict.
While reviewing the benchmark I noticed the naive and anytime-valid simulations built their look schedules in two different places. For equal arms they happened to agree, which is why the A/A false-positive numbers were correct, but the two implementations diverged under arm imbalance and the final look under-sampled the larger arm (1,496 of 1,500 units with eight looks). That meant a like-for-like comparison of the two methods was not guaranteed once arm sizes differed, and a future reader running an imbalanced experiment could have been comparing the methods at different sample sizes without realizing it.
The fix was a single shared peek_schedule helper used by both the naive
simulation and the sequential decision, always pinning the final look to the
full sample. I verified the headline A/A result was unchanged afterward (naive
23.3% versus anytime-valid 1.5% at 15 looks), which confirmed the equal-arm
numbers never depended on the defect. The fix is commit 72f1b9b. There is a
second, honest footnote in the history: fixing this is what briefly broke the
build, because the edit deleted a helper's definition; the test suite caught it
immediately, which is exactly what the suite is for.
A related judgment call worth noting: my first power comparison reported only the single-look anytime-valid power (54%), which made the method look strictly worse than fixed-horizon. That framing is misleading, because it denies the sequential method its defining ability to peek. I added a sequential power measurement (61%) so the trade is presented fairly rather than in the light most flattering to the fixed-horizon baseline.
- Add group-sequential boundaries (O'Brien-Fleming) as an alternative to the mSPRT for teams that prefer a fixed number of pre-planned looks.
- Add a multiple-metric correction layer once experiments report many metrics.
- Persist per-look trajectories so the audit dashboard can show when a naive rule would have stopped versus when the anytime-valid rule did.
- First metric to watch after deploying: the rate of fixed-versus-anytime disagreements in the audit log, which measures how often the team was about to ship a false win.