Adopt upstream #242's golden-file checking - #5
Merged
Conversation
…umented defaults Our --check mode duplicated dsharlet's own open PR dsharlet#242 ("Check stats for correctness"), down to the CheckStatistics method name - and his design is better in three ways, so take his instead of carrying a worse divergence: - Checking is unconditional. Ours was opt-in behind --check, which can be forgotten; a test run that does not assert is not a test. --updateGolden now regenerates, replacing --stats. - Statistics skip a warmup (the first half of the samples). The startup transient is large and sensitive to where buffer boundaries fell, and it was dominating the mean - it is what made the golden files hard to read. - Values are formatted G5. Rounding to five significant figures absorbs last-digit differences between platforms, so the files can be compared as text. That replaces our scale-normalized per-variable tolerance and the per-circuit loose-tolerance list it needed for Pro Co Rat. The golden files are regenerated at the documented defaults (48 kHz, the tool's own default) instead of the undocumented 44.1 kHz the committed ones were made at, so "run the tests" now means what it says with no remembered incantation, and CI no longer passes --sampleRate. Worth recording: dsharlet#242's own regenerated files carry the same provenance quirk. Its Passive 1stOrder Lowpass RC peak is 0.44002, which is the 44.1 kHz grid value; at the 48 kHz default the boxcar-averaged peak is 0.44003, which is what this produces. So regenerating did not fix the mismatch for that PR either - the rate was simply never pinned down. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…table Windows CI caught the one case G5 rounding does not absorb: Floating Input's _v2[t] mean is a cancellation residual around -2.4e-10 on a signal of amplitude 0.5, so five significant figures of it is pure floating-point noise - -2.4347E-10 on Windows against -2.4305E-10 on macOS, differing in every digit while both mean zero. Statistics smaller than 1e-6 of the variable's own range now print as 0. Five golden files change; the other 44 are untouched. Notable: Pro Co Rat now agrees across platforms without any special handling. It previously needed a per-circuit loose tolerance because macOS and Windows diverged by ~4e-2, but that difference lived almost entirely in the startup transient - skipping the warmup removed it. The tolerance machinery was compensating for a statistic that should not have included the transient in the first place. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Windows CI found a second platform disagreement after the first fix: Pro Co Rat's _v28[t] mean printed 0.0018443 there against 0.0018442 here, every other value in all 49 circuits matching exactly. That is the general shape of the problem, not a one-off. min and max are order-of-scale quantities, so G5 of them is ~1e-5 relative - well above the drift between platforms, which is why 48 circuits agreed. The mean is a cancellation residual near zero, so G5 of its own magnitude resolves far below the precision it actually carries, and prints noise. The mean is now quantized to 1e-4 of the variable's envelope. A real regression moves it by orders of magnitude more than that, so this keeps ample resolution to catch one; it just stops asserting on digits that were never meaningful. This replaces the previous absolute snap-to-zero, which only handled the case where the residual was near zero rather than merely small. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces our
--checkwith the design from dsharlet's open PR #242, which does the same job better: unconditional checking, warmup rejection, and G5 rounding that makes the files text-comparable instead of needing per-variable tolerances.Golden files regenerated at the tool's own documented defaults, so CI no longer passes
--sampleRate 44100and "run the tests" needs no remembered incantation.The open question this PR answers: whether G5 rounding absorbs the cross-platform difference that previously required a loose tolerance for Pro Co Rat. macOS passes 49/49 against locally-regenerated files by construction — the Windows leg is the real test. If it fails, the tolerance mechanism has to come back in some form.
🤖 Generated with Claude Code