[PROBE: mass] Add time-origin invariance with aligned calendar shifts - #11
[PROBE: mass] Add time-origin invariance with aligned calendar shifts#11SiavashShams wants to merge 1 commit into
Conversation
|
Thanks @SiavashShams , this is a careful piece of work. The probe itself is right: the 28-year shift keeps every calendar feature aligned over this window (I checked the leap days and the 1,461-week arithmetic), the forcing is byte-identical between variants, and Requested changes1. Drop 2. Drop 3. Don't turn non-finite output into a harness ERROR. 4. Small things
Once 1 to 3 land this is a three-file probe plus its tests, which is what I'd hoped for. Happy to talk through any of these on the issue if you disagree with the reasoning on (1) in particular. |
Closes #10
What this probe asserts
This adds
mass/time-origin-invariance: the same generated weather and catchment are run from 2000-01-01 and 1972-01-01. The 4,015-day records preserve month, day, weekday, day of year and leap-day positions. Only the absolute calendar origin changes; forcing values, static attributes, initialization contract and model seed stay the same.For each required flux or storage, the maximum pointwise difference after spinup must satisfy
max(abs(shifted - control)) / max(mean(abs(control)), 1) <= 1e-9. The one-unit floor handles empty stores. Both variants must independently close their water budgets within 5% of input precipitation. A separate control-run guard rejects zero-runoff or constant-flux outputs.Two small criterion changes support the probe: an opt-in
normalization_floorfor invariance, preserving the existing default, andpaired_closure, which applies the existing closure rule to each named variant. Invariance also rejects nonfinite inputs or arithmetic overflow that could otherwise produce a false pass.The README explains the stationary, rain-only scope. This does not assess snowmelt behavior or changing physical drivers, and a forcing-responsive fixed partition can still pass. The strict tolerance is tested on deterministic CPU references.
Discrimination
The acceptance gate passes with all three gate seeds.
reference_bucketflex_lumpedflex_toposacsma_snow17reference_calendarinvariancereference_degeneratenon_degeneratereference_leakypaired_closureOn five additional seeds (20261001 through 20261005), all 20 physical-model evaluations pass with exactly zero invariance deviation. All five calendar-model evaluations fail only invariance while both water budgets pass. Its smallest normalized deviation is 22.672 against the 1e-9 limit; the largest physical-model closure residual is 1.619e-15 against 0.05.
Validation
ht validate: passes for all 15 probes and 23 models.pytest -q: 174 tests pass, including calendar alignment, identical staged metadata and seeds, repeatability, shorter event windows, both closure budgets and numeric edge cases.ht gate: all 15 probes separate the declared reference models.Checklist
ht validatepasses.ht gate --probe mass/time-origin-invariancepasses.probe.yaml.reference_calendarfails the new invariance check, isolating a fault that closure alone misses.