fuzz: stop morph passing a run that switched nothing - #758
Merged
Conversation
Every term of the morph oracle is about a switch that went wrong, and every one of them is vacuous when no switch happened. With no switch begun the begin/end balance reads 0 -ne 0, the immediate term is gated on $begins, and there are no ghost lines to find, so a run where the layout chord reached nothing left with a green PASS and zero findings. That is the failure mode #752 was about, and this harness could not see it. $chordMisses had the same shape: counted, printed, never asserted on. So gate the run itself. Count the layout chords sent - the startup probe plus one per toggle iteration, excluding the interrupting second chord, which is aimed at a switch still in the air and is meant to be dropped - and require a third of them to have begun a switch, with no more than a fifth refused outright. Both leave with 1, not 2. A chord the desktop refused, or one the router never saw, is a corpus this harness could not establish rather than a defect in the build, and 1 is the code the runner retries. The startup probe reads the trace now instead of stat-ing it. A build that ignores WINTTY_MORPH_TRACE and a build whose chord never reaches the router are the same absent file to Test-Path, and only the second is the one that used to run a whole fuzz to a green verdict. The floor is a third rather than something tighter because this harness loses most of what it sends: a healthy run against a clean build converts 38% to 52% of its chords, measured at 13, 14 and 20 of 34 and at 16 of 31. The margin over the floor is thin by design. Raise it once the harness stops losing chords, not before - a build is not at fault for a chord that never reached it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
vtabs-morph-fuzz.ps1could reportpasswith 0 findings after having toggled nothing. Every term of its oracle is about a switch that went wrong, and all of them are vacuous when no switch happened: with no switch begun the begin/end balance reads0 -ne 0, the immediate term is gated on$begins, and there are no ghost lines to find.$chordMisseshad the same shape - counted, printed, never asserted on.So gate the run itself. Count the layout chords sent - the startup probe plus one per toggle iteration, excluding the interrupting second chord, which is aimed at a switch still in the air and is meant to be dropped - and require a third of them to have begun a switch with no more than a fifth refused.
Both leave with 1, not 2. A chord the desktop refused, or one the router never saw, is a corpus the harness could not establish rather than a defect in the build, and 1 is the code the runner retries.
The startup probe reads the trace now instead of stat-ing it. A build that ignores
WINTTY_MORPH_TRACEand a build whose chord never reaches the router are the same absent file toTest-Path, and only the second is the one that used to run a whole fuzz to a green verdict.The floor is a third because this harness loses most of what it sends. A healthy run against a clean build converts 38% to 52% of its chords: 13, 14 and 20 of 34, and 16 of 31. The margin over the floor is thin by design.
That loss is a separate defect - morph never arms the XAML island, which #752 fixed in the mouse-fuzz harnesses and skipped here. This change makes it visible, not fixed. Raise the floor once the harness stops losing chords, not before: a build is not at fault for a chord that never reached it.
Two things worth knowing about the thin margin:
-Seed, which is the one property this harness otherwise gives you.Validation
just fuzz "-Only morph"just fuzz-selftestSELFTEST OK, exit 0just fuzz-listThe new gate expressions were also run directly over the failure cases: all chords refused, chords sent with the router deaf, and near-zero begins all leave with 1; 13/34, 16/31 and 12/34 pass.
The startup probe caught a real case unprompted. Run outside the suite, which minimizes other windows first,
Focus()succeeds but the chord reaches nothing - previously a silent green, now exit 1. It also happened under the suite at seed 4242, where the retry cleared it.