Skip to content

Add the integration test suite - #13

Closed
BiosSystem wants to merge 55 commits into
masterfrom
feat/integration-suite
Closed

Add the integration test suite#13
BiosSystem wants to merge 55 commits into
masterfrom
feat/integration-suite

Conversation

@BiosSystem

@BiosSystem BiosSystem commented Sep 5, 2026

Copy link
Copy Markdown
Owner

Track 3 of the v3.4.0 plan. Unit tests exercise functions in isolation; these run WinSwift.ps1 as a real process, which is the only way to cover startup guards, parameter binding, config loading and exit codes together.

That difference already paid for itself: this work found the admin guard bypass fixed in #12. Three releases of unit tests and static validation never caught it, because none of them ran the script.

Risk tags

Every Describe carries one tag, chosen by what it can do to the machine:

Tag What it does Safe on
ReadOnly Exercises -Verify, which reads state and exits before applying anything Anywhere, including a workstation
DryRun Asserts -DryRun writes nothing. A regression would write here Ephemeral machine: CI or Sandbox
Mutating Deliberately applies changes Windows Sandbox only

Nothing beyond ReadOnly runs unless asked for.

Coverage

  • VerifyContract.Tests.ps1 pins the -Verify exit codes and profile parsing. Every case is deterministic on any machine, which took some care: whether a real tweak is applied depends on host configuration, so none of these assert on one. The machine-independent cases are an exempt-only profile (proving NotApplicable never fails a run, end to end), an unknown feature, an empty profile, a missing path, and an app id that cannot exist.
  • DryRunSafety.Tests.ps1 proves -DryRun reaches the apply pipeline and still changes nothing, asserting against the exact values the feature would write, read from its .reg file.
  • ApplyRoundTrip.Tests.ps1 applies a registry-backed feature and checks both the verification verdict and the individual values underneath it, so a partially applied .reg cannot pass as compliant.
  • RollbackContract.Tests.ps1 is the executable spec for Track 1. Rollback does not exist yet, so the block skips itself until InvokeChanges.ps1 references Restore-RegistryBackupState, then activates on its own. It pins the failure policy from section 3.2, including the load-bearing case that an app removal failure must not trigger rollback.

Writing that contract first is the point: Track 1's open decisions are now assertions rather than prose.

Two findings, recorded rather than papered over

Undo cannot be tested, so plan scenario 2 is not writable. WinSwift.ps1 initialises $script:UndoParams to an empty hashtable, and the only code that ever populates it is Scripts/GUI/Show-MainWindow.ps1. Invoke-UndoFeatures exists, the per-feature RegistryUndoKey metadata exists, and all 112 features declare undo text, but nothing on the command line can select a feature for undo. The test is present and skipped with the reason inline. The larger problem it stands in for is that unattended deployments cannot revert either.

CI runs more than section 5.2 expected. The plan assumed CI would be limited to non-mutating cases. In practice the windows-2025 runner is build 26100 and runs elevated, so it clears both the version gate and the admin guard, and ReadOnly and DryRun both run there. Sandbox is still needed for Mutating, since Server 2025 has no Start menu start2.bin and a limited Appx stack.

Sandbox harness

Sandbox/WinSwift-Tests.wsb maps the repository read-only so a mutating test cannot modify the working tree it was launched from. The bootstrap copies it somewhere writable, installs Pester, and runs the full suite. Everything is discarded when the window closes.

Three defects CI found in the suite itself

  1. Pester only discovers *.Tests.ps1 when handed a directory. The files used the Test-*.ps1 convention from Tests/Unit, which works there only because CI invokes each by explicit path.
  2. The resulting failure passed as green. Invoke-Pester threw, the empty result made $result.FailedCount null, and $null -gt 0 is false. A suite that discovers nothing now fails.
  3. Three files failed at discovery: Pester evaluates -Skip: during discovery, but Test-IsElevated was only dot-sourced in BeforeAll. Resolved in BeforeDiscovery instead.

Static validation covers 110 files, up from 103.

BiosSystem and others added 30 commits July 4, 2026 00:14
Bumps the all group with 2 updates: [actions/checkout](https://github.com/actions/checkout) and [softprops/action-gh-release](https://github.com/softprops/action-gh-release).


Updates `actions/checkout` from 4 to 7
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](actions/checkout@v4...v7)

Updates `softprops/action-gh-release` from 2 to 3
- [Release notes](https://github.com/softprops/action-gh-release/releases)
- [Changelog](https://github.com/softprops/action-gh-release/blob/master/CHANGELOG.md)
- [Commits](softprops/action-gh-release@v2...v3)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: all
- dependency-name: softprops/action-gh-release
  dependency-version: '3'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: all
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps the all group with 1 update: [actions/checkout](https://github.com/actions/checkout).


Updates `actions/checkout` from 4 to 7
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](actions/checkout@v4...v7)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: all
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Update WinSwift logic and docs for Windows 11 24H2 telemetry and Copilot purge

* Update standalone executable for 3.2.0
Release/ holds generated packaging artifacts that are recreated by
build.ps1 and published to GitHub Releases, so it does not belong in
version control. The existing *.zip rule caught only the archive and
left the expanded payload and checksum file showing as untracked noise.
Ports the ForceRemoveEdge portion of upstream Raphire/Win11Debloat
c8a4069 ("Improve error reporting & handling", #741). The commit cannot
be cherry-picked because WinSwift renamed every file it touches, so this
was applied by hand to the one file that had not also diverged in
content.

- Guard the routine behind WhatIf. ForceRemoveEdge is exposed as a
  FeatureId and could previously run destructively under dry-run when
  invoked directly rather than through the app removal path.
- Wrap the routine in try/catch/finally and dispose the three registry
  keys it opens.
- Create the Edge stub with -Force -ErrorAction Stop so a partial stub
  is not silently skipped.
- Capture the uninstaller exit code and report a nonzero result instead
  of continuing into cleanup as though removal succeeded.
- Report per-path failures when removing leftover shortcuts rather than
  discarding them with -ErrorAction SilentlyContinue.
- Replace four blind "reg delete ... *>$null" calls with
  Remove-EdgeAutostartValue, which distinguishes an already-absent value
  from a failure to inspect or remove an existing one.
- Return a boolean result. Both Request-EdgeForceRemove call sites
  discard it with "$null =" so the caller's pipeline output is unchanged.

Rebuilds the standalone bundle, which inlines Scripts/.
Records the 2026-09-05 review of the five upstream commits landed since
the 383f4f3 baseline, with a disposition for each.

Documents why c8a4069 cannot be cherry-picked, including the full
upstream-to-WinSwift path mapping, so the finding is not rediscovered on
the next reconciliation pass. Lists the files whose ports remain
deferred because WinSwift content has diverged from upstream.
The AllUsers branch of Remove-AppxApp ran the 24H2 Copilot/DevHome/MSTeams
DISM fallback twice. The first copy checks the exit code and catches
errors; the second was a bare repeat of the same call with its output
discarded, so a package that resisted the first attempt was simply
retried with no reporting, and every affected removal paid for a second
DISM invocation.

Keeps the guarded copy and drops the bare one. Also realigns the
Get-AppxProvisionedPackage line, which sat 24 columns too deep, and
normalizes this block to CRLF so it matches the rest of the file.

Rebuilds the standalone bundle, which inlines Scripts/.
The DISM fallback passed the caller's wildcard glob straight through:

    DISM /Online /Remove-ProvisionedAppxPackage /PackageName:*Copilot*

/PackageName requires an exact provisioned package name and rejects
wildcards, so every invocation failed with exit code 87
(ERROR_INVALID_PARAMETER). The check tolerated 87 explicitly, which
masked the failure: the fallback reported success while removing
nothing. Copilot, Dev Home, and the new Teams were left provisioned and
free to return on the next servicing pass, which is exactly what this
path exists to prevent.

Resolve the real package names with the same Get-AppxProvisionedPackage
query already used a line above, and run DISM once per match.

The query runs after the Remove-ProvisionedAppxPackage attempt, so it
returns only packages that survived it. When the cmdlet succeeds, which
is the common case, DISM is not invoked at all.

Exit code handling:

- Drop the 87 tolerance. With a real package name, 87 is a genuine
  failure and should be reported, not swallowed.
- Accept 3010 alongside 0. DISM returns it for success pending reboot.
- Reset $global:LASTEXITCODE before each call, matching the convention
  in Import-RegistryFile and Set-StoreSearchSuggestions, so a stale
  value from an earlier native command cannot be misread as this one's
  result.

Rebuilds the standalone bundle, which inlines Scripts/.
The v3.1.0/v3.2.0 modernization plan was the last roadmap on file and
v3.3.0 shipped without a successor. This adds one and points the old
plan at it.

v3.4.0 is scoped as recoverability and proof. The last three releases
each added half a safety story: v3.2.0 added registry backup, v3.3.0
added desired-state verification, but nothing consumes the backup
automatically and ten features cannot be verified at all. Every track in
this plan extends infrastructure that already ships rather than adding a
new subsystem.

Tracks:

- Automatic rollback on failed apply. The backup is already taken in
  Phase 1 before the apply phase, and Restore-RegistryBackupState is a
  working entry point, but InvokeChanges.ps1 has no reference to any
  restore function. The plan covers the run-scoped backup handle, the
  failure policy, and where the call belongs.
- Verification coverage from 102/112 to 112/112, following the
  Appx-absence pattern DisableWidgets already uses in Test-FeatureApplied.
- An integration suite for Tests/Integration, which has been empty since
  July, targeting apply, undo, and both rollback branches.
- Field validation of the 24H2 DISM fallback, which was diagnosed from
  DISM's documented contract rather than observed failing.

Records the sequencing constraint that tracks 2 and 3 land before
track 1, the InvokeChanges.ps1 collision with the deferred upstream port
backlog, and four open decisions that need answers before implementation.
Closes the Track 2 gap in the v3.4.0 plan. All 112 features in
Features.json now declare how they are verified, and the routing is
driven by that metadata rather than by a hardcoded list.

The gap was smaller than the plan claimed. Three of the ten features
counted as unverifiable were already verified at runtime by a hardcoded
FeatureId list inside Test-WinSwiftFeature:

    if ($FeatureId -in @('RemoveApps','RemoveGamingApps','RemoveHPApps'))

The metadata was incomplete, not the coverage. That is its own problem:
routing by hardcoded list means Features.json does not describe how a
feature is verified, and the unit suite cannot tell a declared gap from
an undeclared one. Real new verification work was five features.

Changes:

- Replace the hardcoded FeatureId branch with dispatch on the declared
  VerificationAdapter, making Features.json the single source of truth.
- Add the AppxAbsence adapter for the three app-removal features,
  checking installed and provisioned state. Provisioned state matters
  because a package can be uninstalled per-user while still provisioned
  and due to return on the next servicing pass.
- Add the EdgeRemoved adapter for ForceRemoveEdge, checking the Edge
  uninstall key in the 32-bit registry view plus the four autostart
  values Remove-EdgeAutostartValue clears. Those are exactly the
  artifacts ForceRemoveEdge manipulates.
- Add the StartLayout adapter for the four start menu features,
  SHA-256 comparing the on-disk start2.bin against the expected
  template. The all-users variants check every user profile plus the
  default profile, since new users inherit from it.
- Add the NotApplicable status for entries with no persistent desired
  state, reported distinctly and feeding neither the failure nor the
  error count so it cannot affect exit code 2. Apps is a value-carrying
  CLI parameter and CreateRestorePoint is a one-shot action; neither has
  a state to read back.

Tests:

- Assert every feature declares a RegistryKey or a VerificationAdapter,
  so a new feature cannot be added without a verification story.
- Pin the exemption list to exactly Apps and CreateRestorePoint, so
  NotApplicable cannot become a dumping ground.
- Cover the new adapters, the NotApplicable counting rule, the
  no-adapter Unsupported path, and unknown-adapter error handling.

Corrects section 4 of the v3.4.0 plan, which listed the three
already-covered features as gaps, and rebuilds the standalone bundle.
Ensure-Admin.ps1 is dot-sourced, and `exit` inside a dot-sourced script
does not terminate the caller. Every exit in that guard was therefore
inert: WinSwift printed "must be run as Administrator" and then carried
on into the apply pipeline.

Reproduced non-elevated with stdin redirected, which is what any
scripted or CI invocation looks like. Read-Host returns an empty string
immediately, the prompt reads as declined, `exit 1` does nothing, and
the run proceeds:

    WinSwift must be run as Administrator.
    ...
    [WhatIf] Create registry backup
    [WhatIf] Apply 15 registry changes from 'Disable_Telemetry.reg'
    [WhatIf] Disable Scheduled Task: ...Microsoft Compatibility Appraiser
    exit code 0

Under -DryRun nothing is written. Without it, an unprivileged process
would attempt real registry imports and scheduled task changes, fail
partway through on access denied, and leave a half-applied system.

The same defect breaks the relaunch path: after Start-Process -Verb
RunAs succeeds, `exit 0` does not stop the parent, so the elevated child
and the original non-elevated process run WinSwift concurrently.

Confirmed the mechanism with a minimal repro. A dot-sourced script
invoked with arguments cannot terminate its caller through `exit`, with
or without [CmdletBinding()]; `throw` and caller-side handling both do.

Fix: the guard reports its outcome through $script:ElevationOutcome,
which propagates to the caller because dot-sourcing shares scope, and
WinSwift.ps1 exits on anything other than 'Elevated' before any runtime
module loads. 'Relaunched' exits 0 because the elevated child owns the
run; 'Denied' and 'Failed' exit 1.

Also stop prompting when no console can answer. A redirected read
returned instantly and was indistinguishable from a declined prompt, so
that case now reports why it cannot continue.

Verified after the fix: both the read-only -Verify path and the -DryRun
apply path exit 1 without reaching the pipeline.

Adds source assertions for the outcome contract and its position ahead
of module loading, plus a behavioral test that runs the entry script
unelevated and asserts the pipeline is never reached. That test skips
when already elevated, which is the case on CI runners.
Track 3 of the v3.4.0 plan. Unit tests exercise functions in isolation;
these run WinSwift.ps1 as a real process, which is the only way to cover
startup guards, parameter binding, config loading, and exit codes
together.

Tests are gated by what they can do to the host:

- ReadOnly runs the -Verify path, which reads state and exits before
  applying anything, so it cannot write even if the engine regresses.
  Safe anywhere, and the default.
- DryRun asserts that -DryRun writes nothing. A regression in the
  dry-run guard would write to the host, so it needs an ephemeral
  machine.
- Mutating deliberately applies changes and is Windows Sandbox only.

Nothing beyond ReadOnly runs unless asked for.

Coverage:

- Test-VerifyContract.ps1 pins the -Verify exit codes and profile
  parsing. Every case is deterministic on any machine, since whether a
  real tweak is applied depends on host configuration: an exempt-only
  profile, an unknown feature, an empty profile, a missing path, and an
  app id that cannot exist.
- Test-DryRunSafety.ps1 proves -DryRun reaches the apply pipeline and
  still changes nothing, asserting against the exact values the feature
  would write rather than sweeping the registry.
- Test-ApplyRoundTrip.ps1 applies a registry-backed feature and checks
  both the verification verdict and the individual values underneath it,
  so a partially applied .reg file cannot pass as compliant.
- Test-RollbackContract.ps1 is the executable specification for Track 1.
  Rollback does not exist yet, so the block skips itself until
  InvokeChanges.ps1 references Restore-RegistryBackupState, and
  activates on its own when Track 1 lands. It pins the failure policy,
  including that an app removal failure must not trigger rollback.

Two findings recorded in the README rather than papered over:

- Undo cannot be tested. WinSwift.ps1 initialises $script:UndoParams
  empty and only Show-MainWindow.ps1 ever populates it, so no command
  line can select a feature for undo. Invoke-UndoFeatures and the
  RegistryUndoKey metadata all exist and all 112 features declare undo
  text, but the CLI cannot reach any of it. The test is present and
  skipped with the reason inline. The larger problem it stands in for is
  that unattended deployments cannot revert either.
- CI runs more than section 5.2 of the plan expected. The windows-2025
  runner is build 26100 and elevated, so it clears both the version gate
  and the administrator guard, and both ReadOnly and DryRun run there.
  Sandbox is still required for Mutating.

Adds a Sandbox configuration and bootstrap for the mutating tests, which
maps the repository read-only so a test cannot modify the working tree
it was launched from, and wires the two CI-runnable tags into the
workflow.
The first CI run of the suite reported success while running nothing:

    Pester v5.7.1
    ##[error]No test files were found and no scriptblocks were provided.
    Integration:  passed,  failed,  skipped

Two defects, both in this suite.

Pester only discovers files matching *.Tests.ps1 when handed a
directory. The files followed the Test-*.ps1 convention used in
Tests/Unit, which works there only because CI invokes each of those by
explicit path. Renamed to *.Tests.ps1.

Worse, the failure passed. Invoke-Pester threw, the empty result gave
$result.FailedCount the value $null, and `$null -gt 0` is false, so the
step went green having executed no tests. A suite that discovers nothing
has to fail rather than report success.

The runner now catches discovery errors, treats a null or zero-total
result as a failure, and prints the total alongside the pass and fail
counts so an empty run is visible rather than inferred. The CI step
checks the same condition instead of trusting FailedCount alone.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant