Size your early-stopping window by statistical power instead of by habit.
pip install stoppowerMost people decide a model "converged" by comparing two or three evaluations and loosening the tolerance when detection fails. That tunes the wrong knob. In the measured case (Zenodo 10.5281/zenodo.21630279), the total detection gain decomposes into 7.0× from lengthening the window and only 1.31× from changing the statistic. Tolerance barely moves the needle.
Worse: once the signal-to-noise ratio drops below 1, no tolerance recovers the decision. Type I and Type II error can no longer be separated by any choice of threshold.
For n equally spaced evaluations, the standard error of the OLS slope decreases as n^(-3/2),
against n^(-1/2) for plain averaging:
SE(b) = σ · sqrt( 12 / (n(n² − 1)) )
b* ≥ (z_α + z_β) · SE(b)
from stoppower import window_for, sigma_from_pilot, evaluate, prereg_text, pts
# 1) What window do I need so I don't miss 1 accuracy point per 2500 steps?
plan = window_for(b_star=pts(1.0), sigma=0.0082, eval_every=500)
print(plan) # -> n=11 evaluations = 5000 steps
# 2) How much power does the window I already use actually have?
from stoppower import power_of
power_of(pts(1.0), sigma=0.0082, n=3, eval_every=500) # -> 0.097
# 3) Decide with data in hand
evaluate(steps, accs, b_star=pts(1.0))stoppower design --b-star 0.01 --sigma 0.0082 --eval-every 500 --prereg
stoppower power --b-star 0.01 --sigma 0.0082 --n 3 --eval-every 500
stoppower check history.csv --b-star 0.01 --span 5000 10000Validation history of a model trained for 10,000 steps, evaluated every 500. The question is whether the tail is still improving or it is safe to stop.
$ stoppower check history.csv --b-star 0.01 --span 5000 10000
sigma estimated over the 5000-10000 span: 0.010593
UNDECIDABLE · slope +0.006424 per 2500 steps (95% CI [-0.003474, +0.01632])
t=1.27 p=0.1017 · n=11 evaluations · power=0.63 for b*=0.01
! power 0.63 < 0.8: this window cannot support a convergence claim,
only "no improvement was detected". Lengthen the window before concluding.A two-point criterion would have said "converged". Here the verdict is undecidable, with the number next to it: the window has no power to sustain that claim.
And if you hand it the whole run instead of the stable regime, it warns before answering:
$ stoppower check history.csv --b-star 0.01 --span 500 10000
! noise changes 7.4x within the window (sigma=0.06951 in the 1st half vs
0.009344 in the 2nd, higher at the start): the equation assumes constant noise.
Shorten the window or move it to the stable regime.1. σ is estimated over a span, not over the whole run. Validation noise is not constant. Measured across 8 seeds of the same model:
| step | 500 | 1000 | 1500 | 2000 | 2500 | 5000 | 7500 |
|---|---|---|---|---|---|---|---|
| SD across seeds | .101 | .073 | .038 | .010 | .014 | .009 | .011 |
A factor of 9. Feeding a global σ into the equation is off by a large factor, which is why the span
is a required argument here. homoscedasticity_check warns when the constant-noise assumption does
not even hold inside the window.
2. It does not confuse "no improvement detected" with "converged". evaluate returns
converged=None when power is insufficient, instead of a false green light. That asymmetry is the
whole point: failing to reject improvement is not evidence of convergence unless the study had the
power to see it.
3. It emits the pre-registerable paragraph. prereg_text(plan) produces the text with σ, its
provenance, the level, the power and the resulting window — to paste into your protocol before
looking at the data.
| situation | function |
|---|---|
| validation set is resampled each evaluation | sigma_floor(p, n_val) — analytic bound, no training needed |
| fixed set, you have a pilot run | sigma_from_pilot(steps, values, span=(from, to)) |
| you already have several seeds | sigma_from_runs(runs, at_index=...) |
Fixing a window from a pilot is not the same as recalibrating a tolerance after seeing results: what gets fixed is the design, not the verdict.
The acceptance test reproduces Table 2 of the paper (σ = 0.00821, α = .05 one-sided, power = .80): n=4 → b* = 4.56, n=8 → 1.58, n=16 → 0.55 accuracy points per 2500 steps.
Beyond arithmetic, the nominal power was checked against a Monte Carlo of 4,000 simulations per case: empirical power 0.824 against 0.80 nominal, empirical α 0.054 against 0.05, and 0.092 for n=3 against the 0.097 the formula predicts. The rule is calibrated, not just implemented.
No dependencies (standard library only). It does not train, does not touch your loop, does not decide for you. It does not model hardware effects: switching backends was measured to contribute 0.53× the across-seed variation in the late regime, negligible against what is already reported.
The method (cite this if you use the rule):
Speranza, M. R. (2026). Stopping criteria below the signal-to-noise floor: window length, not tolerance, governs convergence detection in architecture comparisons. 10.5281/zenodo.21630279
The software (cite this if you use the package):
Speranza, M. R. (2026). stoppower: size your early-stopping window by statistical power. 10.5281/zenodo.21711767
MIT.