A statistical A/B test on 90,000 mobile-game players — with the discipline to tell a significant result from an important one.
Cookie Cats is a mobile puzzle game that uses "gates" to pause player progression. This project analyzes a real A/B test the developers ran: the first gate was placed at level 30 (control) for half the new players and at level 40 (treatment) for the other half. The question: does moving the gate later change how many players stick around? The analysis is done in Python (pandas, scipy) and covers the full A/B toolkit — hypothesis testing, confidence intervals, bootstrapping, effect size, and sample-ratio validation.
A game studio wants more players to keep playing. Gates are a lever they can move — but moving a lever blindly can backfire. Before shipping a change to millions of players, they ran a controlled experiment and asked:
Does moving the first gate from level 30 to level 40 improve or hurt player retention?
Retention is measured two ways: whether a player came back 1 day after install (retention_1) and 7 days after (retention_7).
| Metric | gate_30 (control) | gate_40 (treatment) | p-value | Significant? |
|---|---|---|---|---|
| Day-1 retention | 44.8% | 44.2% | 0.074 | No |
| Day-7 retention | 19.0% | 18.2% | 0.0016 | Yes |
Moving the gate later is associated with a statistically significant drop in Day-7 retention — a relative decline of about 4.5%. Day-1 retention shows no significant difference, which itself is informative: on day 1 most players haven't reached the gate yet, so the manipulation hasn't had a chance to matter. By day 7, more players have hit the gate, and the effect emerges.
Three independent methods — a two-proportion z-test, a 95% confidence interval, and a 1,000-iteration bootstrap — all agree: the Day-7 interval excludes zero, and 99.8% of bootstrap samples show control ahead of treatment.
The core theme of the analysis is that statistical significance and practical importance are not the same thing — and with ~90,000 players per group, this matters constantly:
- The significant Day-7 effect is real but small (Cohen's h is tiny).
- The sample-ratio (SRM) check came back "significant" (p = 0.0085) even though the actual group split is a trivial 49.6% vs 50.4% — the test is simply so sensitive at this sample size that a sub-1% imbalance registers.
Throughout, effect size and business context are treated as companions to the p-value, not afterthoughts.
Keep the gate at level 30. The evidence indicates that delaying it to level 40 lowers long-term retention with no offsetting short-term gain. As with any single historical A/B split, this is a correlational result worth re-confirming before a permanent product change — but the direction is clear, and there is no reason to ship a change that appears to hurt retention.
A single, documented Jupyter notebook walks through the full A/B workflow:
- Data loading and quality checks (missing values, duplicates)
- Outlier cleaning (one impossible value of ~50,000 game rounds in 14 days)
- Group balance + a formal Sample Ratio Mismatch (SRM) chi-square check
- Hypothesis formulation (H0/H1, α = 0.05, two-tailed)
- Retention rates per group
- Two-proportion z-test (implemented from scratch, not a black-box call)
- 95% confidence intervals for the difference
- Bootstrap distribution of the difference (1,000 iterations, fixed seed)
- Visualizations (retention by group, bootstrap distributions)
- Effect size (relative lift + Cohen's h)
- Multiple-testing note (Bonferroni)
- Summary, limitations, and business recommendation
(Screenshots below.)
- Python — pandas (data handling), scipy.stats (chi-square, z-test, normal distribution), numpy (bootstrap), matplotlib (visualization)
- Statistical techniques — two-proportion z-test, confidence intervals, bootstrap resampling, Cohen's h effect size, Bonferroni correction, chi-square goodness-of-fit (SRM)
- Reproducibility — fixed random seed; notebook runs top-to-bottom without errors
The Cookie Cats A/B test dataset: 90,189 players, one row each, with the test group (gate_30 / gate_40), total game rounds played, and 1-day and 7-day retention flags. It is a well-known public dataset used to teach A/B testing, available on Kaggle: Mobile Games A/B Testing — Cookie Cats.
I came to data analytics from business and operations — I co-founded and helped run an online grocery business through to its sale. That background shapes the way I interpret analytical results. A p-value of 0.0016 may be statistically significant, but for me, that is only the starting point. The more important questions are: How large is the effect? Does it have a meaningful business impact? And is it strong enough to justify changing a decision, process, or investment? Managing a real P&L taught me that a statistically significant result and a result worth acting on are not always the same. This analysis is built around that distinction — connecting statistical evidence with business relevance and practical decision-making.