A garden of forking paths simulator. You get a dataset with no treatment effect in it and six ordinary analytic choices. You fiddle until p < 0.05, which you will find, and then the tool shows you every other analysis you could have run and what fraction of them also cleared the bar.
Catalog task EDU-038. Part of 722 things to build.
The page hands you a fictional randomised trial of a study-skills app on 240 students. The outcomes are generated before anybody is assigned to an arm, so the app cannot have done anything. There is no effect to find.
Then it gives you six decisions to make. Which of the three outcomes counts. Whether to drop students who barely attended, or whose score looks like a data-entry error. Which baseline covariates to adjust for. Whether to enter baseline study time as a continuous variable, a median split, or tertiles. Which subgroup to look at. Classical or robust standard errors. None of those is cheating. Every one of them is a call a careful analyst has to make, and most of them can be defended out loud.
Multiply them out and there are 1,920 answers to "did the app work?", all computed from the same 240 students. Roughly 6% of them are significant at 0.05 in any given cohort, so if you keep clicking you will find one. When you press the reveal button the page draws the distribution of all 1,920 p-values with yours marked inside it, a specification curve in the Simonsohn, Simmons and Nelson layout, and a table showing which forks the significant results are hiding behind.
The point is Gelman and Loken's, and it is about honest researchers rather than fraud. A researcher who runs exactly one analysis can still be caught by this, because the analysis they ran was chosen after seeing the data. Had the numbers come out differently they would have reasonably chosen differently. The whole set of would-have-beens is what sets the real error rate, and the single path actually walked says very little about it. Nobody has to know they are doing this. No data is fabricated.
| measured | |
|---|---|
| P(the single pre-specified test reaches p < 0.05) | 0.0600 (9 of 150 null cohorts) |
| P(at least one of 1,920 specifications reaches p < 0.05) | 0.8733 (131 of 150 null cohorts) |
| P(at least one reaches p < 0.05 favouring the app) | 0.5467 (82 of 150) |
| mean share of specifications significant in a cohort | 0.0418 |
Every row comes from the same generator, the same 150 cohorts, and the same arithmetic. The first row is the deal you signed up for when you chose alpha = 0.05. The second row is what searching the garden costs on top of it.
A tool that teaches people about false positives while computing p-values wrongly would be its own punchline, so nothing here is taken on trust.
The null is a property of the construction. src/data.js draws every
student trait and every outcome first, with no reference to any arm, and only then assigns
arms by shuffling an exactly balanced vector of zeros and ones using a separate RNG stream.
Step two cannot see step one, so the treatment indicator is independent of everything
measured by construction. The effect parameter exists only so the test suite can inject a
real effect and confirm the machinery notices; it defaults to zero and the page always
passes zero.
Calibration is the test that decides whether any of this teaches anything. Under the null, a single fixed pre-specified analysis must produce p-values uniform on [0, 1]. The suite runs 5,000 independent null datasets through four separate analysis paths and checks both the empirical rejection rate against a binomial band and the Kolmogorov-Smirnov distance to Uniform(0, 1). If the generator leaks an effect, or the incomplete beta function is wrong, or the tail is off by a factor of two, this notices and essentially nothing else would.
The uniformity checker shares no code with the thing it checks. src/uniformity.js
writes its own Kolmogorov series and its own normal tail from scratch. A validator that
reuses the implementation it is validating inherits that implementation's bugs and reports
clean.
The t distribution is checked four independent ways. Against published two-tailed
t-table critical values typed in from the standard table. Against exact elementary closed
forms for df = 1, 2, 3 and 4. Against test/reference/t_reference.py, which computes the
same tail by a completely different route (Gauss-Legendre quadrature on a substitution that
removes the endpoint singularity) and is executed during the test run rather than read from
a stale checked-in blob. And against two algebraic identities linking separately written
code paths: the pooled two-sample t-test equals OLS on a single 0/1 dummy, and the Pearson
correlation t equals the OLS slope t.
scipy is not installed on this machine, so the Python reference is the cross-language check in its place. It self-checks its own quadrature branch against two exact closed forms before emitting anything, and it integrates at 400 and 1,600 nodes and refuses to return a value if they disagree.
The factor-of-two bug has its own assertions. At the alpha = 0.05 critical value a two-sided tail must read exactly 0.05. A one-tailed implementation would read 0.025 and a doubled one 0.10, and both are asserted against explicitly at thirteen degrees of freedom from 1 to 120.
Both negative controls are in the suite. A deliberately broken p-value that always returns 0.01 must fail the uniformity check, and so must one that always returns 0.5, which a one-sided band would wave through as merely conservative. A real 0.6 SD effect injected into the generator must be detected by the pre-specified test with power above 0.9; measured power is 0.969. Without that last one, a tool that reported "null, nothing here" for every input would pass everything else.
npm test # the full suite, about 33 seconds
bash scripts/verify.sh # the suite, plus the page, plus a real browser
node scripts/build-page.mjs # regenerate docs/index.html from src/docs/index.html is generated, never hand-edited. scripts/build-page.mjs inlines the same
src/ modules the tests exercise into the template, so the page cannot drift away from the
tested code, and verify.sh runs it with --check to prove the committed page is exactly
what the current source produces.
Open docs/index.html in a browser directly, or serve the directory. It is a single
self-contained file with no network requests of any kind.
Simulation counts can be turned down for local iteration, at the cost of a wider band:
FORKING_SIMS=500 FORKING_SWEEPS=20 FORKING_POWER_SIMS=200 npm testsrc/stats.js RNG, log-gamma, incomplete beta, t distribution, t-tests, OLS with HC3
src/data.js the null generator and the pre-registered analysis
src/forking.js the six choice points, one specification, the whole multiverse
src/uniformity.js independent KS and binomial-band checks, sharing no code with stats.js
test/ calibration, statistics, multiverse, negative controls
test/reference/ the independent Python t-distribution reference
docs/ the page template and the generated single-file page
scripts/ build-page.mjs, verify.sh
bash scripts/verify.sh exits 0. Real output, pasted:
== Toolchain
node v24.13.0
python3 Python 3.12.3
browser $HOME/.cache/ms-playwright/chromium_headless_shell-1208/chrome-headless-shell-linux64/chrome-headless-shell
== The committed page matches the tested source
docs/index.html is up to date (84045 bytes)
== The page is self-contained and carries no secrets
no remote references, no credential-shaped strings
== Node test suite (calibration, statistics, multiverse, negative controls)
uniformity checker, synthetic streams:
genuinely uniform m=20000 rate(p<0.05)=0.04840 band=[0.04493, 0.05507] mean(p)=0.49962 KS=0.00713 sqrt(m)*KS=1.0086 KS_p=0.2609 UNIFORM
always 0.01 (broken) m=20000 rate(p<0.05)=1.00000 band=[0.04493, 0.05507] mean(p)=0.01000 KS=0.99000 sqrt(m)*KS=140.0071 KS_p=0.0000 NOT UNIFORM
always 0.5 (broken) m=20000 rate(p<0.05)=0.00000 band=[0.04493, 0.05507] mean(p)=0.50000 KS=0.50000 sqrt(m)*KS=70.7107 KS_p=0.0000 NOT UNIFORM
p^1.15 (anti-conservative) m=20000 rate(p<0.05)=0.07360 band=[0.04493, 0.05507] mean(p)=0.46541 KS=0.04903 sqrt(m)*KS=6.9335 KS_p=0.0000 NOT UNIFORM
CALIBRATION under the null, pre-specified analyses:
Welch t-test on examScore m=5000 rate(p<0.05)=0.05540 band=[0.03986, 0.06014] mean(p)=0.49365 KS=0.01362 sqrt(m)*KS=0.9629 KS_p=0.3119 UNIFORM
pooled t-test on examScore m=5000 rate(p<0.05)=0.05540 band=[0.03986, 0.06014] mean(p)=0.49365 KS=0.01362 sqrt(m)*KS=0.9629 KS_p=0.3118 UNIFORM
runSpec, pre-specified path m=5000 rate(p<0.05)=0.05540 band=[0.03986, 0.06014] mean(p)=0.49365 KS=0.01362 sqrt(m)*KS=0.9629 KS_p=0.3118 UNIFORM
Pearson screenTime vs examScore m=5000 rate(p<0.05)=0.05360 band=[0.03986, 0.06014] mean(p)=0.49685 KS=0.01270 sqrt(m)*KS=0.8982 KS_p=0.3953 UNIFORM
arm vs baseline priorScore m=5000 rate(p<0.05)=0.05040 band=[0.03986, 0.06014] mean(p)=0.49398 KS=0.01624 sqrt(m)*KS=1.1482 KS_p=0.1432 UNIFORM
covariate balance, mean standardised arm difference over 400 datasets:
priorScore -0.00229
attendance 0.00242
sleepHours -0.00275
studyMinutes -0.00473
screenTime 0.00183
gradeLevel -0.00171
firstGen -0.01262
✔ the uniformity checker can tell uniform from broken (14.477287ms)
✔ the KS machinery agrees with hand-computable cases (4.301985ms)
✔ Welch t-test p-values are uniform under the null (5000 datasets) (0.637324ms)
✔ pooled t-test p-values are uniform under the null (5000 datasets) (0.529707ms)
✔ the pre-specified runSpec path is uniform under the null (5000 datasets) (0.685095ms)
✔ correlation p-values are uniform under the null (5000 datasets) (0.681007ms)
✔ baseline covariates are balanced across arms (5000 datasets) (0.568142ms)
✔ the pooled t-test and the OLS path agree exactly on real generated data (6.584407ms)
✔ the treatment arm is exactly balanced and independent of every covariate (135.13082ms)
✔ the outcomes really are correlated with each other, so the null is not trivial (7.162682ms)
✔ the same seed gives the same dataset and different seeds do not (8.389693ms)
garden shape: 4 x 4 x 3 x 4 x 5 x 2 = 1920 specifications
seed 42: 1920 fitted, 1776 distinct p-values (144 no-op exclusion coincidences), min p = 1.618e-3, median p = 0.330, p<0.05 in 311/1920
HEADLINE, null data, no effect whatsoever:
datasets swept 150
specifications per dataset 1920
specifications that failed to fit 0
P(at least one specification p<0.05) 0.8733 (131/150)
P(at least one p<0.05 favouring treatment) 0.5467 (82/150)
mean fraction of specifications p<0.05 0.0418
median smallest p across the multiverse 1.450e-2
P(pre-specified single test p<0.05) 0.0600 (the honest 5%)
marginal p<0.05 rate by option, seed 8:
outcome examScore=0.000 homeworkRate=0.015 confidence=0.000 composite=0.000
exclusions none=0.000 attendance=0.000 outliers=0.000 both=0.015
covariates none=0.011 prior=0.000 priorSleep=0.000
studyTime omit=0.006 linear=0.004 median=0.000 tertile=0.004
subgroup all=0.000 firstGen=0.016 continuingGen=0.000 lower=0.003 upper=0.000
se classical=0.004 hc3=0.003
NEGATIVE CONTROL, a real effect injected:
true effect 0.6 ability SD = 5.40 exam points
mean estimated effect 5.367 exam points
power of the pre-specified test 0.9690 (1938/2000)
same via the runSpec path 0.9690
sign recovered 1.0000
Monte Carlo se of the mean 0.0311 exam points, 4-sigma band 0.125
with a real 0.6 SD effect, seed 4: p<0.05 in 1538/1920 specifications (80.1%), median estimate 0.518 SD
✔ the number of specifications equals the product of the choice cardinalities (6.989871ms)
✔ the multiverse actually explores: different specifications give different answers (397.14961ms)
✔ specification results are internally consistent (225.889195ms)
✔ runSpec refuses an unknown specification instead of guessing (0.500529ms)
✔ at least one specification reaches p<0.05 far more often than 5% (150 null datasets) (30087.178949ms)
✔ significance is scattered across forks rather than driven by one of them (169.292334ms)
✔ the specification curve is sorted and complete (184.429982ms)
✔ injecting a real effect of 0.6 SD is detected with high power (2000 datasets) (565.11396ms)
✔ a real effect also shows up almost everywhere in the multiverse (195.333692ms)
✔ the null multiverse and the effect multiverse are clearly different (386.26786ms)
✔ the Python reference produced a usable grid (0.728513ms)
✔ tTwoSidedP matches the independent Python reference everywhere (0.358904ms)
✔ tCritical reproduces published two-tailed t-table values (1.61618ms)
✔ tTwoSidedP at a published critical value returns exactly the alpha (0.138793ms)
✔ the large-df limit is the standard normal (0.119427ms)
✔ tTwoSidedP matches exact closed forms for df = 1, 2, 3 and 4 (0.112201ms)
✔ boundary behaviour of the t tail (0.164121ms)
✔ tCdf is the one-sided companion and is consistent with the two-sided tail (0.105147ms)
✔ logGamma matches known factorials and Gamma(1/2) (0.079794ms)
✔ incompleteBeta satisfies its defining identities (0.329823ms)
✔ descriptives on a hand-checked vector (0.154333ms)
✔ pooled two-sample t-test on a hand-computed dataset (0.150501ms)
✔ Welch t-test with unequal group sizes and variances, against Python (0.111691ms)
✔ the two-sample tests fail loudly rather than returning a fake result (0.071452ms)
✔ Pearson correlation on a hand-computed dataset, against Python (0.413234ms)
✔ Pearson correlation edge cases (0.083746ms)
✔ OLS on a single 0/1 dummy is exactly the pooled two-sample t-test (0.530412ms)
✔ the OLS slope t on one continuous predictor is exactly the correlation t (0.14101ms)
✔ OLS recovers an exactly determined linear relation (0.126786ms)
✔ OLS reports a singular design instead of inventing numbers (0.084351ms)
✔ HC3 standard errors differ from classical ones but stay in the same ballpark (0.182118ms)
✔ invert is a real matrix inverse and returns null when singular (0.066281ms)
ℹ tests 43
ℹ suites 0
ℹ pass 43
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 32278.557803
== Measured numbers
CALIBRATION, p-value uniformity under the null:
Welch t-test on examScore m=5000 rate(p<0.05)=0.05540 band=[0.03986, 0.06014] mean(p)=0.49365 KS=0.01362 sqrt(m)*KS=0.9629 KS_p=0.3119 UNIFORM
pooled t-test on examScore m=5000 rate(p<0.05)=0.05540 band=[0.03986, 0.06014] mean(p)=0.49365 KS=0.01362 sqrt(m)*KS=0.9629 KS_p=0.3118 UNIFORM
runSpec, pre-specified path m=5000 rate(p<0.05)=0.05540 band=[0.03986, 0.06014] mean(p)=0.49365 KS=0.01362 sqrt(m)*KS=0.9629 KS_p=0.3118 UNIFORM
Pearson screenTime vs examScore m=5000 rate(p<0.05)=0.05360 band=[0.03986, 0.06014] mean(p)=0.49685 KS=0.01270 sqrt(m)*KS=0.8982 KS_p=0.3953 UNIFORM
arm vs baseline priorScore m=5000 rate(p<0.05)=0.05040 band=[0.03986, 0.06014] mean(p)=0.49398 KS=0.01624 sqrt(m)*KS=1.1482 KS_p=0.1432 UNIFORM
HEADLINE, at least one specification significant on null data:
P(at least one specification p<0.05) 0.8733 (131/150)
P(at least one p<0.05 favouring treatment) 0.5467 (82/150)
mean fraction of specifications p<0.05 0.0418
P(pre-specified single test p<0.05) 0.0600 (the honest 5%)
NEGATIVE CONTROL, a real effect must be detected:
power of the pre-specified test 0.9690 (1938/2000)
uniformity verdicts: 6 uniform, 3 deliberately broken streams caught
== Real browser, docs/index.html at 390px and 1280px
node says: prespecP=0.6553990324 specs=1920 failed=0 sig=119 minP=0.0050276413 medianP=0.4407064268
browser 390,844: SELFTEST:OK viewport=375 controlsFlipped=6 seed=1 prespecP=0.6553990324 specs=1920 failed=0 sig=119 minP=0.0050276413 medianP=0.4407064268 curveMarks=1920 curveRows=22 paths=26 histBars=41 tableRows=22 labelTexts=32
console messages: 0
browser 1280,900: SELFTEST:OK viewport=1265 controlsFlipped=6 seed=1 prespecP=0.6553990324 specs=1920 failed=0 sig=119 minP=0.0050276413 medianP=0.4407064268 curveMarks=1920 curveRows=22 paths=26 histBars=41 tableRows=22 labelTexts=32
console messages: 0
== PASS
The page, the statistics, and the claims all check out.
The verify was attacked twice to confirm it can fail.
| attack | exit code |
|---|---|
| the null generator leaks a real 0.22 SD effect into every treated student | 1, calibration caught it: rate 0.284 outside [0.039859538965116036, 0.06014046103488397], 5 tests failed |
tTwoSidedP always returns 0.5 |
1, 20 tests failed, starting with rate 0 outside [0.039859538965116036, 0.06014046103488397] |
In both cases the page was rebuilt before running verify, so the failure came from the statistics rather than from the staleness check noticing an edited source file.
Browser results, Chrome via Playwright against docs/index.html served over HTTP on a port
bound to 0 and confirmed by comparing the served bytes to the file on disk:
- zero console messages at 390px and at 1280px, in light and in dark, including
pageerrorandrequestfailed - no horizontal body scroll at either width, and no element in
mainextending past the viewport except inside the two containers that scroll on purpose - the choice controls move the p-value: 0.6554 on the pre-specified path, 0.3985 after switching the sample to grades 9 and 10
data-themeoverrides the OS setting in both directions: a light stamp under OS dark gives--plane: #f9f9f7, a dark stamp under OS light gives--plane: #0d0d0d
Finding that took a real browser. The page had an unbalanced parenthesis in the histogram code, so the entire inline script failed to parse and the page rendered as static HTML with no numbers in it. Every node test passed the whole time.
- The multiverse is 1,920 specifications, which is far short of all of them. Sample size, the alpha level, one-sided versus two-sided, transformations of the outcome, interaction terms, and measurement decisions made before the data reached the analyst are all fixed here. A real garden is larger. 87.3% is a lower bound on the problem.
- Cluster and multiplicity corrections are not offered. There is no Bonferroni, no false discovery rate control, and no bootstrap-based joint test of the specification curve as Simonsohn and colleagues describe. The page shows the distribution; it does not offer to fix it.
- The data is Gaussian and well behaved. No missingness, no attrition, no measurement error, no floor or ceiling effects, no non-normal outcomes. That makes the p-values exactly uniform under the null and it makes the demonstration cleaner than reality. Real data would give the garden more branches than this.
- The headline is measured on 150 cohorts. The binomial standard error on 0.8733 is about 0.027, so read it as "roughly 87%" rather than as three significant figures. Those 150 cohorts are seeds 1 through 150, so the figure is byte-reproducible across runs but it is still one Monte Carlo sample. The suite asserts only that it exceeds 0.5 and is more than ten times the nominal 0.05.
- The composite outcome is standardised within whatever sample survives the exclusions, which is one specific defensible convention among several. A different convention would give slightly different numbers for the four composite-outcome branches.
prefers-reduced-motionis respected but there is little motion to reduce. The page has no animation beyond a tooltip fade.- The specification curve is a 1,920-column plot. It scrolls inside its own container. On a phone the plot window is about 200px wide, which is enough to see the red band at the end but not enough to read the middle without scrolling.
- Gelman, A. and Loken, E. (2013). The garden of forking paths: Why multiple comparisons can be a problem, even when there is no "fishing expedition" or "p-hacking" and the research hypothesis was posited ahead of time. Department of Statistics, Columbia University.
- Simmons, J. P., Nelson, L. D. and Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science 22(11), 1359-1366.
- Steegen, S., Tuerlinckx, F., Gelman, A. and Vanpaemel, W. (2016). Increasing transparency through a multiverse analysis. Perspectives on Psychological Science 11(5), 702-712.
- Simonsohn, U., Simmons, J. P. and Nelson, L. D. (2020). Specification curve analysis. Nature Human Behaviour 4, 1208-1214.
MIT. All data on the page is synthetic. No real students, no real school, no real app.