In 2000 committed null simulations, a naive experimenter peeking daily shipped a false winner 27.5% of the time at a nominal alpha of 5%; this engine's always-valid sequential test held it to 1.7%.
- Teams that peek at running A/B tests inflate false positives severalfold without knowing it; measured here at 27.5% vs the 5% they think they are running.
- Teams that refuse to peek block ship decisions on fixed horizons; the sequential test decided with 42% fewer samples at equal power in simulation.
- Every statistical guarantee in this repo is verified by committed Monte Carlo simulation with fixed seeds, not cited from a paper.
An A/B test is a purchase of certainty, and both failure modes are expensive. Ship a false winner and you pay in reverted launches and polluted metrics baselines; hold a real winner for a fixed four-week horizon and you pay in delayed revenue for every experiment in the queue. The most common failure is quieter: someone opens the dashboard on day 6, sees p = 0.04, and ships. Under a daily-peeking schedule with a stop-at-significance rule, this repo measures the real false positive rate at 27.5% (N = 2000 null simulations, 30 daily looks, 500 units per arm per day, 5% baseline conversion, seed 20260731). The team believes it is running a 5% error rate. It is running a coin flip against a fivefold-inflated one.
The mechanism, not the pitch: the engine implements the mixture sequential probability ratio test (mSPRT), which replaces the fixed-horizon p-value with an always-valid p-value process built from a likelihood ratio martingale. By Ville's inequality, the probability that the process ever crosses the rejection boundary under the null is at most alpha, for any stopping rule, so peeking is safe by construction rather than by policy. On top of that, CUPED variance reduction uses a pre-experiment covariate to shrink metric variance before the sequential test sees it; the two compose because CUPED is a pre-processing step that leaves the effect estimate unbiased. ADR-001 documents why mSPRT beat group-sequential boundaries and Bayesian decision rules for this design; ADR-002 documents why the whole engine stays frequentist.
Measured results, all reproducible from benchmark/run_full_simulations.py on the committed seeds (2 vCPU, 4 GB shared Linux container): the mSPRT held the false positive rate at 1.65% under the same daily peeking that broke the naive test (27.5%). With a real effect at 1.5x the minimum detectable effect, it decided at an average of 10,100 units against a 16,316-unit fixed-horizon plan, a 38.1% sample saving at ~100% observed power. At exactly the MDE it used 18.7% more samples than the 80%-power fixed plan but delivered 98% observed power; a fixed test with that power needs 33,486 units, so the like-for-like saving is 42.2%. CUPED removed 9.0%, 25.0%, and 49.0% of metric variance at covariate correlations of 0.3, 0.5, and 0.7, matching the rho-squared theory to within 0.03 percentage points; variance removed converts one-for-one into sample size and calendar time not spent.
flowchart LR
subgraph plan_stage [Planning]
A[plan: power analysis<br/>min sample size]
end
subgraph run_stage [Experiment]
B[assign: 50/50 split] --> C[observe: daily<br/>metric + covariate]
end
subgraph decide_stage [Decision]
D[SRM gate:<br/>chi-square on counts]
E[CUPED adjust<br/>leakage check]
F[sequential monitor:<br/>mSPRT, confidence sequence]
G[decision report:<br/>markdown + PNG for PMs]
end
H[peeking harness:<br/>adversarial Monte Carlo<br/>naive vs mSPRT]
A --> B
C --> D
D -- counts trusted --> E
D -- SRM detected: HALT,<br/>exit code 2 --> X[do not read metrics]
E -- covariate clean --> F
E -- leakage suspected:<br/>fall back to raw metric --> F
F --> G
H -. attacks the alpha claim of .-> F
style X fill:#c0392b,color:#fff
style H fill:#2c3e50,color:#fff
Failure boundaries: the SRM gate halts before any metric is read (broken randomization invalidates everything downstream); the CUPED leakage check degrades to the unadjusted metric rather than passing a biased one; the harness is not in the serving path at all, it exists to attack the monitor's statistical claims.
| Component | Choice | Why here specifically |
|---|---|---|
| Numerics | numpy | The harness runs 2000+ simulations as whole-matrix operations; per-sim Python loops would move full runs from seconds to minutes |
| Distributions | scipy.stats | Normal quantiles and the chi-square test only; all sequential logic is author-built and unit tested |
| Data interchange | pandas | CSV in, day-partitioned replay; the monitor consumes plain arrays so pandas stays at the edge |
| Config validation | pydantic v2 | Experiment configs are reviewed by non-statisticians; field-level errors beat a stack trace |
| CLI | click | Subcommands with typed options; exit code 2 reserved for SRM so pipelines can gate on it |
| Charts | matplotlib | Decision reports are static PNGs a PM can paste into a launch doc |
| Tests | pytest + pytest-cov | Statistical tests use reduced-N Monte Carlo with fixed seeds so CI is deterministic and fast |
Author-built vs library-provided: scipy provides distribution functions. The mSPRT likelihood ratio, always-valid p-value, confidence sequences, CUPED adjustment, leakage check, SRM gate, non-inferiority guardrails, and the peeking harness are implemented and tested in this repo.
git clone https://github.com/vedantpanchal/ab-decision-engine.git
cd ab-decision-engine
python3 -m venv venv && source venv/bin/activate
pip install -e ".[dev]"
# 1. Plan: minimum sample size for the experiment in examples/experiment.yaml
abengine plan --config examples/experiment.yaml
# 2. Simulate an experiment (synthetic data) and run it through the monitor
abengine simulate --metric-type proportion --n-per-arm 6000 \
--baseline 0.05 --effect 0.025 --seed 3 --out /tmp/exp.csv
abengine monitor --data /tmp/exp.csv --tau 0.01
# 3. Generate the PM-facing decision report (markdown + PNG in reports/)
# including a latency-like guardrail with a non-inferiority margin
abengine report --data /tmp/exp.csv --tau 0.01 --name demo \
--planned-n-total 16316 --out-dir reports \
--guardrail-col guardrail --guardrail-margin 5 --guardrail-lower-is-better
# 4. Run the peeking demo: naive daily peeking vs mSPRT on null data
abengine peek-demo --sims 500 --days 30 --daily 500
# Verify the engine itself
pytest --cov=abengine
python benchmark/run_full_simulations.py # full Monte Carlo, ~5 s
python examples/full_experiment_walkthrough.pyExpected: step 2 prints "decision": "ship" with an always-valid p-value; step 4 prints a naive false positive rate far above alpha while mSPRT stays under it.
Load for a simulation engine is simulation scale. The harness is fully vectorized across simulations, so one benchmark run covers naive + mSPRT over 30 looks each. Measured on a 2 vCPU, 4 GB shared Linux container (another build was running concurrently; treat times as upper bounds), seed 20260734, raw output in benchmark/results/harness_scaling.json:
| simulations | looks each | wall time (naive + mSPRT) | throughput |
|---|---|---|---|
| 500 | 30 | 0.016 s | 63,047 sims/s |
| 2,000 | 30 | 0.051 s | 77,925 sims/s |
| 8,000 | 30 | 0.227 s | 70,341 sims/s |
Honest degradation note: throughput is not monotone; fixed overhead dominates the 500-sim run and throughput drops about 10% from 2,000 to 8,000 simulations as working-set size grows past cache, so headline sims/s from the mid-size run should not be extrapolated to much larger sweeps. The full benchmark suite (all four claims) completes in under 5 seconds, which is the point: verifying your testing procedure by simulation is cheap enough to run in CI on every change.
Full records in docs/adr/:
- ADR-001: mSPRT over Pocock/O'Brien-Fleming group-sequential boundaries and Bayesian rules. Group-sequential boundaries are valid only at pre-scheduled looks; real consumption is a dashboard loaded at arbitrary times. mSPRT is about 60 lines of reviewable closed-form code and its guarantee holds under every stopping rule.
- ADR-002: the boring choice of frequentist sequential + CUPED over a Bayesian decision framework, for an org without a statistician to own priors. Includes the concrete trigger for revisiting (50+ experiments per quarter or genuinely loss-based ship criteria).
- Multi-armed bandits: adaptive allocation only pays for itself at high experiment volume and it complicates inference on every metric. Trigger to revisit: sustained backlog of more than roughly 20 concurrent experiments competing for the same traffic.
- Interference / network effects modeling: the engine assumes SUTVA (one unit's assignment does not affect another's outcome). Trigger: documented unit-level spillovers, for example marketplace cannibalization or social features, at which point cluster randomization belongs in the assignment layer, not here.
- Multiple-metric family-wise error control: one primary metric per experiment is a discipline worth keeping; guardrails are non-inferiority checks, not additional winners. Trigger: an org decision to formally ship on composite metrics.
- Every dataset in this repository is synthetic, generated by
abengine/simulate.pyfrom committed seeds. No user data, no PII, nothing scraped. - For real deployments the intended pattern is environment-based connection config (for example
AB_METRICS_DSN) injected at runtime; nothing in this codebase reads or stores credentials, and the CSV interface keeps the engine decoupled from any warehouse. - The CLI writes structured JSON logs to stderr with no metric values in log lines beyond what the invoking analyst explicitly requests on stdout.
| Failure | Detection | Behavior | Recovery |
|---|---|---|---|
| Sample ratio mismatch (broken randomization, asymmetric bot filtering, logging loss) | Built-in chi-square SRM gate on arm counts, threshold p < 0.001 | monitor prints the SRM block and exits with code 2; report stamps "do not trust this experiment" before any metric is shown |
Fix assignment or logging, restart the experiment; SRM-contaminated data is not salvageable by reweighting |
| Metric pipeline lag (conversions arrive late, recent days undercounted) | Effect estimate at the newest looks biased toward control; visible as a tail dip in the report chart | Always-valid inference makes re-running safe: a look at incomplete data is just another look | Re-run monitor after the pipeline catches up; no alpha is spent by reprocessing |
| Covariate leakage into CUPED (covariate measured post-assignment) | covariate_leakage_check: z-test of covariate balance across arms, flags at p < 0.001 |
Leakage warning; analysis should fall back to the unadjusted metric (report notes the reduction only when the covariate is clean) | Rebuild the covariate strictly from pre-experiment windows; rerun |
| Seed misuse (reusing one seed across supposedly independent runs) | Identical outputs across runs that should differ; benchmark suite derives distinct sub-seeds per claim | Committed seeds make published numbers reproducible; new studies must change the master seed | Bump the seed, rerun, compare against the committed results for drift |
The vectorized harness computes each simulation's stopping day as crossed.argmax(axis=1) over a boolean look matrix. argmax returns the 0-indexed look, but days are 1-indexed. Every stopping simulation therefore reported its decision one day early, and average sample size at decision came out 1,000 units too low, which overstated the sequential method's sample savings (44.2% instead of the correct 38.1% at 1.5x MDE). Nothing crashed; the numbers were merely flattering, which is the worst kind of bug in a repo whose whole claim is honest measurement. It surfaced while cross-checking avg_units_at_decision against 2 * daily * avg_days and then replaying 200 simulations through a deliberately slow per-simulation reference loop, which disagreed with the vectorized path by exactly one day on every stopping run. The reference loop is now a committed regression test (test_decision_day_matches_slow_reference). Fix commit: 91d196f.
- Streaming ingestion: consume an append-only event log instead of CSV snapshots, with watermark-aware looks.
- Quantile metrics: confidence sequences for medians and p95s via the Howard-Ramdas empirical Bernstein bounds.
- Stratified CUPED: multiple covariates via regression adjustment, with the same leakage gate per covariate.
- Experiment registry: persist plans and decisions so realized MDEs and durations feed back into future planning defaults.
- Optional group-sequential mode for orgs with genuinely fixed review cadences, where Pocock boundaries buy back some power.