Give every feature a verification story - #11
Merged
Conversation
Closes the Track 2 gap in the v3.4.0 plan. All 112 features in
Features.json now declare how they are verified, and the routing is
driven by that metadata rather than by a hardcoded list.
The gap was smaller than the plan claimed. Three of the ten features
counted as unverifiable were already verified at runtime by a hardcoded
FeatureId list inside Test-WinSwiftFeature:
if ($FeatureId -in @('RemoveApps','RemoveGamingApps','RemoveHPApps'))
The metadata was incomplete, not the coverage. That is its own problem:
routing by hardcoded list means Features.json does not describe how a
feature is verified, and the unit suite cannot tell a declared gap from
an undeclared one. Real new verification work was five features.
Changes:
- Replace the hardcoded FeatureId branch with dispatch on the declared
VerificationAdapter, making Features.json the single source of truth.
- Add the AppxAbsence adapter for the three app-removal features,
checking installed and provisioned state. Provisioned state matters
because a package can be uninstalled per-user while still provisioned
and due to return on the next servicing pass.
- Add the EdgeRemoved adapter for ForceRemoveEdge, checking the Edge
uninstall key in the 32-bit registry view plus the four autostart
values Remove-EdgeAutostartValue clears. Those are exactly the
artifacts ForceRemoveEdge manipulates.
- Add the StartLayout adapter for the four start menu features,
SHA-256 comparing the on-disk start2.bin against the expected
template. The all-users variants check every user profile plus the
default profile, since new users inherit from it.
- Add the NotApplicable status for entries with no persistent desired
state, reported distinctly and feeding neither the failure nor the
error count so it cannot affect exit code 2. Apps is a value-carrying
CLI parameter and CreateRestorePoint is a one-shot action; neither has
a state to read back.
Tests:
- Assert every feature declares a RegistryKey or a VerificationAdapter,
so a new feature cannot be added without a verification story.
- Pin the exemption list to exactly Apps and CreateRestorePoint, so
NotApplicable cannot become a dumping ground.
- Cover the new adapters, the NotApplicable counting rule, the
no-adapter Unsupported path, and unknown-adapter error handling.
Corrects section 4 of the v3.4.0 plan, which listed the three
already-covered features as gaps, and rebuilds the standalone bundle.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes Track 2 of the v3.4.0 plan. All 112 features now declare how they are verified, and routing is driven by that metadata instead of a hardcoded list.
The gap was smaller than the plan claimed
The plan counted ten unverifiable features. That was measured from
Features.jsonmetadata alone, and it overstated things — three of the ten were already verified at runtime by a hardcoded FeatureId list insideTest-WinSwiftFeature:The metadata was incomplete, not the coverage. That's its own problem though: routing by hardcoded list means
Features.jsondoesn't describe how a feature is verified, and the unit suite can't distinguish a declared gap from an undeclared one — which is exactly why the miscount happened.RemoveApps,RemoveGamingApps,RemoveHPAppsAppxAbsence; routing moved to metadataForceRemoveEdgeEdgeRemovedadapterClearStart,ClearStartAllUsers,ReplaceStart,ReplaceStartAllUsersStartLayoutadapterAppsCreateRestorePointReal new verification work was five features.
Appsturned out to be a parameter ("the selection of apps to remove, specified as a comma separated list") that happens to live inFeatures.json— there is no state to read back, so the plan's single-exemption assumption became two.What's built
Features.jsonis now the single source of truth.AppxAbsence— resolves the app list via the existingGet-WinSwiftFeatureAppIds, checks installed and provisioned state. Provisioned state matters: a package can be uninstalled per-user while still provisioned and due to return on the next servicing pass.EdgeRemoved— checks the Edge uninstall key in the 32-bit registry view plus the four autostart valuesRemove-EdgeAutostartValueclears. Those are exactly the artifactsForceRemoveEdgemanipulates, so their absence is a true applied-state signal.StartLayout— SHA-256 compares the on-diskstart2.binagainst the expected template (bundled blank forClearStart*, caller-supplied forReplaceStart*). All-users variants check every user profile plus the default profile, since new users inherit from it.NotApplicable— new status, reported distinctly, feeding neitherFailedCountnorErrorCountso it cannot affect exit code2.Tests
Two guards that make the gap non-recurring:
RegistryKeyor aVerificationAdapter— a new feature can't be added without a verification story.AppsandCreateRestorePoint, soNotApplicablecan't become a dumping ground for features nobody wanted to verify.Plus coverage for the new adapters, the
NotApplicablecounting rule, the no-adapterUnsupportedpath, and unknown-adapter error handling.Note the existing
RemoveAppstest fixture needed aVerificationAdapteradded — under the old hardcoded routing it didn't need one, which is the same latent problem in miniature.Verification
Tests\Invoke-StaticValidation.ps1— 103 files and all config JSONTrue, content differs →False, file missing →False, missingReplaceStarttemplate → throws (engine surfaces it asError), andEdgeRemoved→Falseon a machine with Edge installedFeatures.jsonassertions replicated by hand: 112/112 covered, 0 gaps, exemptions exactly as pinnedPester deferred to CI — this machine has only Pester 3.4.0 and CI requires 5.7.1, so the six new tests run here for the first time.
Not verified end to end: a full
-Verifysweep across all 112 features on a live machine needs elevation and belongs to the Track 3 integration suite.Also corrects §4 of the plan, which listed the three already-covered features as gaps.