test: pin the numbers the suite was only stepping around - #283
Merged
Conversation
A mutation sweep over src/lib found a consistent blind spot: the suite tests
which branch runs, rarely what it computes. Coverage cannot see it — the line
executes either way — so 25 constants could be changed with all 881 tests
green. Each finding below was confirmed by breaking the code and watching
nothing fail, then re-confirmed by the new test failing.
quality.ts — the entire deduction table was free. Every per-hour rate, every
per-factor cap, the infeasible and yeast-extreme penalties: all 16 could be
doubled silently, because the tests only asserted which factor fired. The
score is what the user sees, as stars. Adds a describe block that isolates one
factor at a time and pins the exact score, plus the two promises CLAUDE.md
makes about the rates (one moderate deviation stays at five stars; only
stacked problems fall below four). The existing cap test used a 50-point-wide
bound where the answer is exactly 92; the floor test asserted <= 50 where it
is exactly 0.
schedule.ts — the 48 h cold-bulk ceiling, the 1 h pre-fridge bulk, the
3 h room minimum, the room-mode 2:1 budget split and its 90 min final-proof
cap, and the 14/30 degC warning edges were all unpinned; several tests derived
their expectation from the constant under test, so both sides moved together.
Adds a constants tripwire (the idiom INPUT_BOUNDS and WINDOW_STOPS already
use), the leg-length and budget-split assertions, band-edge cases for the
temperature warnings and sweeps that pin both yeast thresholds against
freshEquivalentPercent.
ics.ts — the free/busy set was sampled, not covered: autolyse, bulk-room and
proof-cold could each be dropped from FREE_KINDS without a failure, and a
passive rest marked OPAQUE blocks out the baker's whole calendar day. Now
every step kind is asserted. Same for ACTIVE_NIGHT_KINDS, pinned by membership.
community.md was never parsed by a test. Bad rows are dropped silently by
design, so a typo'd date or URL removes a contributor's recipe from the site
with a green suite. Both data tables now count the link-carrying rows in the
source and assert the parser kept all of them (pizzerias only asserted "at
least one").
Smaller holes, same kind: the stand mixer's 18 degC friction (only the
ordering spiral < stand < hand was pinned), sh= omitted for non-sourdough
links (every other conditional key had an omission test), the TRMNL summary
labels for the two dry yeasts, formatGrams' 1 g and 10 g band edges, and
roundBallWeight's floor — reachable with 1 ball at the 100 g minimum in an
enriched dough, where without it the button hands back a 0 g ball weight.
Adds a cross-locale net over stepCopy: 128 recipe shapes x 5 locales, asserting
nothing rendered still carries a {placeholder}. messages.test.ts catches a
translator dropping a token; nothing caught a code path forgetting to
interpolate one.
Two browser tests for rules only a browser reaches: setReadyBy's startAt floor
(only "not specified" flour gets there, since a stated flour re-picks the
window instead) and the autolyse flag surviving a pre-ferment toggle.
918 unit / 81 browser tests, coverage still 100 %.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
A review of the whole test suite for whether the tests are real. Short answer: they overwhelmingly are — the assertions are specific, the comments say what broke and why, and the e2e specs cover things a unit test structurally cannot reach. I found essentially no fake tests.
What I did find is one consistent blind spot, and it took a mutation sweep to see it: the suite tests which branch runs, rarely what it computes. Coverage cannot catch this — the line executes either way — so 25 constants across
src/lib/could be changed with all 881 tests green.Every finding below was confirmed by breaking the code and watching nothing fail, then re-confirmed by the new test failing on the same mutation.
Findings
quality.ts— the entire deduction table was free. All 16 rates, caps and penalties could be doubled silently; the tests only asserted which factor fired. The score is what the user sees, as stars. Now one factor is isolated at a time and the exact score pinned, plus the two promises CLAUDE.md makes about the rates. The existing cap test used a 50-point-wide bound where the answer is exactly 92; the "floors at 0" test asserted<= 50where it is exactly 0.schedule.ts— six documented numbers unpinned. The 48 h cold-bulk ceiling (the reasonWINDOW_STOPSends at 80 h), the 1 h pre-fridge bulk, the 3 h room minimum, the room-mode 2:1 budget split and its 90 min final-proof cap, and the 14/30 °C warning edges. Several tests derived their expectation from the constant under test, so both sides moved together and nothing was pinned at all..icsfree/busy was sampled, not covered.autolyse,bulk-roomandproof-coldcould each be dropped fromFREE_KINDSwithout a failure — a passive rest markedOPAQUEblocks out the baker's whole calendar day. Same forACTIVE_NIGHT_KINDS, now pinned by membership.community.mdwas never parsed by a test. Bad rows are dropped silently by design, so a typo'd date or URL removes a contributor's recipe from the site with a green suite.pizzerias.mdonly asserted "at least one row". Both now count the link-carrying rows in the source and assert the parser kept all of them.Smaller holes, same kind: the stand mixer's 18 °C friction (only the ordering
spiral < stand < handwas pinned),sh=omitted for non-sourdough links, the TRMNL summary labels for the two dry yeasts,formatGrams' 1 g and 10 g band edges, androundBallWeight's floor — reachable with 1 ball at the 100 g minimum in an enriched dough, where without it the button hands back a 0 g ball weight.New net: 128 recipe shapes × 5 locales asserting nothing rendered still carries a
{placeholder}.messages.test.tscatches a translator dropping a token; nothing caught a code path forgetting to interpolate one. (No leak today.)Two browser tests for rules only a browser reaches:
setReadyBy'sstartAtfloor (only "not specified" flour gets there, since a stated flour re-picks the window instead) and the autolyse flag surviving a pre-ferment toggle. The first fails against the pre-fix mutation; verified.Not changed
No production code. The one reachable-but-untested guard (
roundBallWeight's floor) got a test rather than a deletion — I checked, and it is reachable through the form.Numbers
881 → 918 unit, 79 → 81 browser, coverage still 100 %.
CLAUDE.mdgains the three testing rules this work established.