Measure what checking an eval mid-run does to your false-positive rate, then compute the paired sample size and the group-sequential boundaries that fix it. Monte Carlo plus recursive numerical integration, in one HTML file, with no libraries and no network.
Live demo: https://0xelitesystem.github.io/eval-peeking-planner/
Open the page. There are three tools and they all compute on the spot.
-
Peeking simulator. Give it a total number of paired eval cases, a discordance rate, and a list of look counts. It simulates the whole run under a true null and reports how often the test fires at least once. It runs four statistics side by side: McNemar uncorrected, McNemar with the continuity correction, an exact mid-p binomial, and a continuous z on accumulating normal information. Every run also produces the control table, which repeats the identical simulation but tests only at the final look. If the control does not land on the nominal alpha, the inflation above it means nothing, so it is printed with a 95 percent Monte Carlo interval and a verdict.
-
Boundaries. Press the button and the page recovers the Pocock and O'Brien-Fleming constants for 2 through 5 looks by recursive Simpson quadrature over the continuation region, then bisects on the shape constant until the exit probability equals alpha. It prints the recovered value next to the published one so you can see the recovery rather than trust it. There is a second panel for designs that are in no table at all: pick any K, alpha, power and shape and it solves that one too.
-
Sample size. Connor's formula for a paired binary comparison. The discordance rate is a required input and is displayed on every line of output, because a paired N without one is not a usable number. The page also prints the whole discordance curve, the unpaired two-group comparison for contrast, and the sequentially adjusted N using the inflation factor it just computed. A reverse solver takes a paired N somebody quoted at you and tells you what discordance rate was buried in it.
Empty and out-of-range inputs produce a visible error and no output. The discordance rate must exceed the absolute improvement you are sizing for, because two versions cannot differ by more than they disagree; that is checked and refused.
The advice "do not peek at your eval" is everywhere and the size of the penalty almost never is. It is not a small effect. Ten equally spaced looks turns a nominal 5 percent test into roughly a 19 percent test, and the page will show you that on your own parameters rather than asserting it.
The second reason is the discordance rate. Sample-size numbers for paired comparisons circulate with the disagreement rate stripped out, and it is the parameter that dominates the answer. Holding the effect, alpha and power fixed, moving discordance from 5 percent to 40 percent multiplies the required number of cases by roughly eight. Any paired N quoted without that rate has hidden its largest assumption. This page will not print one.
The third reason is that the boundary constants are usually copied out of a table. They do not have to be. The quadrature that produces them is short enough to run in a browser tab in about a second, so the page computes them and shows the published values alongside as a check on itself, including the one place where the two disagree.
Single file, MIT, no build step, no dependency graph to audit, no telemetry.
Everything runs in your browser. The page makes no network requests of any kind: no CDN, no fonts, no analytics, no error reporting, no remote configuration. Your parameters are never transmitted, because there is nothing to transmit them to. The only thing stored on your machine is your light or dark theme choice, in localStorage, and the page works fine if storage is blocked. Open the file with your network disconnected and every feature still works.
- Normal distribution. The error function is evaluated from its Maclaurin series below 2 and from the standard continued fraction above it, both iterated to double precision. Quantiles are bisected on that CDF.
- Binomial sampling. Inversion when the mean count is under 30, transformed rejection above it. The Stirling tail corrections used by the rejection step are computed at load time from exact factorials rather than pasted in as literal constants.
- Randomness. A seeded 32-bit counter-based generator. The seed is an input, so a given seed reproduces a given table exactly.
- Boundaries. Recursive integration of the continuation-region density, composite Simpson with M nodes per look (241 by default), then bisection on the shape constant. The inflation factor comes from a second bisection, on the drift that achieves the requested power under the recovered boundaries.
- Threading. Work runs in a Web Worker built from a Blob, so the page stays responsive while it computes. Where workers are unavailable the identical code runs on the main thread in timed chunks. The status line reports which path ran.
The quadrature recovers seven of the eight boundary constants and three of the four inflation factors to the precision the published tables are printed at. The exception is the Pocock inflation factor at five looks, where this page computes 1.2284 against a published 1.207. Both O'Brien-Fleming factors and the two-look Pocock factor agree. That disagreement is unresolved and is stated on the page itself rather than smoothed over. Press the button and check the number.
Separately, the page does not use the continuity-corrected McNemar statistic as its headline. Run the simulator at one look and read the two columns: the uncorrected statistic lands on the nominal alpha and the corrected one lands visibly below it. No number for that gap is quoted here, because it is a Monte Carlo estimate that moves with the seed and the trial count; the page fills the figure in from the run you just did. A test that comes in under the alpha you asked for is a different test with less power, and it understates the peeking penalty for the same reason.
git clone https://github.com/0xelitesystem/eval-peeking-planner.git
cd eval-peeking-planner
Then open index.html in any browser. Or serve it:
python -m http.server 8000
and visit http://localhost:8000/.
There is no build. One file, index.html, with inline CSS and JavaScript. No package manager, no bundler, no dependencies.
Nothing in this repository is ported, adapted or derived from another project, and no third-party code is vendored or reproduced here. The statistical procedures are implemented directly from their published mathematical definitions: the series and continued-fraction identities for the error function, composite Simpson quadrature, the Pocock and O'Brien-Fleming boundary shapes, McNemar's test, and Connor's paired sample-size formula. The transformed-rejection binomial sampler is implemented from the published description of the algorithm, with its constants derived at load time rather than copied.
This is an independent project. It is not affiliated with, endorsed by, or connected to any statistics vendor, publisher, or model provider named or implied anywhere in it.
- https://github.com/0xelitesystem/eval-significance-calculator
- https://github.com/0xelitesystem/eval-regression-viewer
- https://github.com/0xelitesystem/llm-judge-calibrator
MIT. Copyright (c) 2026 0xelitesystem.