Skip to content

fix(vmtest-harness): narrow the swallowed-die exemption and wire the host-only gates into CI - #5017

Open
mac-duetto wants to merge 2 commits into
mainfrom
fix/vmtest-harness-guard-soundness
Open

fix(vmtest-harness): narrow the swallowed-die exemption and wire the host-only gates into CI#5017
mac-duetto wants to merge 2 commits into
mainfrom
fix/vmtest-harness-guard-soundness

Conversation

@mac-duetto

Copy link
Copy Markdown
Collaborator

Fixes two adversarially-verified defects in vmtest-harness/ and wires the harness's existing host-only test scripts into CI. Closes #5014 and #5015 together because the wiring is what makes the second one load-bearing — both bugs were latent for the same reason: grep -rn "vmtest" .github/ returned zero.

#5014 (MEDIUM) — local x=$(f) was exempted, and it genuinely swallows the child status

tests/check-no-swallowed-die.sh:248 exempted both x=$(f) and local x=$(f). The bare form is correct — it propagates the child status to set -e. The local form is not, and the driver already says so at vmtest:28-29: local is itself a command, so its own status (always 0) wins and the child's is lost.

This is not a lint. vmtest:649-657 records that a runtime exit-code override was tried, reverted, and deliberately declined because this static guard covers it. With the hole open, a failed install run exits 0 while its MEASURE line reports the real failure — on_exit (vmtest:659) exits with $rc, not VMTEST_EXIT.

Changes: delete (local[ \t]+)? from the regex; correct the comment at :243-245, which had claimed both forms propagate; add a local-form case to the selftest.

#5015 (LOW) — the tool-name invariant was red on main

lib/vm.sh:3 declares that file the only one in the harness that may contain the virtualisation tool's name (DOC-1 §12.2, so a second backend can be swapped in). lib/verify.sh:1442 named it in a prose comment — 1,429 lines after that same file's warning at :13-15 that the grep "matches COMMENTS as readily as code". Reworded to name the tool by description.

CI wiring

New .github/workflows/vmtest-harness.yml — one lightweight job modelled byte-for-byte on line-cap.yml (this repo's convention is one gate per file, not a shared lint workflow; generation-artifact-lint.yml:19-21 and agent-assets.yml:13-15 state why). Path-filtered to vmtest-harness/** plus the workflow file itself.

Steps, selftest-before-gate per line-cap.yml's ordering: guard selftest → guard → subshell-classification.shtest-preflight-single-run.sh → the invariant grep as its own failing step.

Two choices worth reviewing:

  • runs-on: macos-14, not ubuntu-latest. The scripts pin /bin/bash 3.2.57 (vmtest:17) and test-preflight-single-run.sh reads BSD ps and renames argv[0] with exec -a. A Linux runner would exercise them under bash 5 with GNU ps and could report green for semantics the real target fails — the exact false negative that lets a swallowed-status construct through. Scripts are invoked as /bin/bash <script> rather than the repo-standard bash scripts/… so the interpreter is pinned regardless of what a future runner image puts on PATH, and one step records the version so the claim is auditable in the log.
  • A paths: filter, which ci: docs-only PRs run the full Rust build (Clippy/Format/MSRV/Test) — add path filtering #4468 normally forbids. That rule binds gates in branch protection, where a path-skipped required check never reports and leaves the PR pending. This gate is not a required context and the harness is self-contained, so the hazard does not apply. Justified inline in the workflow.

All four scripts are host-only — no VM is created, no network is touched. subshell-classification.sh sources the driver through its --source-only hook; test-preflight-single-run.sh puts a stub for the virtualisation CLI on PATH and redirects the run registry and $HOME into a temporary root.

Test ladder — rung 2 (test-only stabilization / test harness)

No Rust changes at all, so the ladder's -p <crate> gate has no crate to name; the harness's own scripts are the equivalent proof, and they are exactly what this PR wires into CI. Rung 1 (comments) also applies to the verify.sh reword. Claiming 2 rather than 1 because the guard's scanning behaviour genuinely changed, so a comment-only gate would not have been enough.

Commands run on /bin/bash 3.2.57 (arm64-apple-darwin25) — the documented target:

$ /bin/bash vmtest-harness/tests/check-no-swallowed-die.sh
check-no-swallowed-die: 106 functions, 48 call `die` directly, 58 classify transitively
check-no-swallowed-die: OK — no classifying function sits in a status-discarding context
EXIT=0

$ /bin/bash vmtest-harness/tests/check-no-swallowed-die-selftest.sh
=== check-no-swallowed-die selftest: the guard must be able to fail ===
ok   control: unmutated harness is GREEN
ok   guard goes RED on for-list                     [for-list]
ok   guard goes RED on process substitution         [process-substitution]
ok   guard goes RED on argument position            [argument-position]
ok   guard goes RED on heredoc substitution         [heredoc-substitution]
ok   guard goes RED on local-form assignment        [argument-position]
ok   guard goes RED on run_watchdog's command argument [run-watchdog-argument]
ok   guard goes RED on unannotated indirect invocation [indirect-invocation]
ok   guard goes RED on backtick command substitution [backtick-substitution]
ok   guard goes RED on a NEWLY die-capable function [argument-position]
ok   guard REFUSES a vacuous scan (emptied sources)
---
11 passed, 0 failed

$ /bin/bash vmtest-harness/tests/subshell-classification.sh
12 passed, 0 failed
subshell-classification: OK

$ /bin/bash vmtest-harness/tests/test-preflight-single-run.sh
test-preflight-single-run.sh: 128 passed, 0 failed, 0 skipped
EXIT=0

$ grep -rlnw 'tart' vmtest-harness --include='*.sh' --include='vmtest'
vmtest-harness/lib/vm.sh

The narrowed guard stays green on the clean tree — zero call sites to repair, zero new false positives, exactly as #5014 measured. Every real site already follows declare-first/assign-second.

Both new gates confirmed load-bearing (a gate that cannot fail proves nothing):

  • Restoring the old regex and planting local _z=$(tsv_scope_packages) into scenarios/install-local.shOK — no classifying function sits in a status-discarding context, EXIT=0. The pre-fix guard passes the mutation; the new selftest case is what closes that.
  • Reverting the verify.sh reword → the invariant step exits 1 and lists both lib/verify.sh and lib/vm.sh.

Repo gates that run regardless of path filter were also run locally and are green: check_generation_artifacts.sh, check_line_cap.sh, check_doc_numbers.sh.

Changelog

No fragment — exempt. scripts/check_changelog_fragment.sh scopes evidence to crates whose crates/<crate>/src/** changed; vmtest-harness/** and .github/workflows/** are outside crates/ entirely, so no crate requires evidence and the gate passes green. Substantively the same answer: this is a test-harness fix plus CI wiring with no user-visible change to any published crate.

Out of scope

Deliberately not done, per the brief: no widening of the invariant grep to *.rs (no Rust in the harness yet), no bash→Rust port work, no refactor of the guard beyond the one-token regex change.

🤖🤖🤖 Generated with trusty-mpm — https://github.com/bobmatnyc/trusty-tools

…host-only gates into CI

Two verified defects in the harness, plus the CI wiring that would have
caught both. Neither bug was reachable by review alone; both were latent
because the harness's four test scripts and its one mechanical invariant
ran nowhere — `grep -rn "vmtest" .github/` returned zero.

#5014 — `local x=$(f)` is not exempt

`check-no-swallowed-die.sh:248` exempted both `x=$(f)` and `local x=$(f)`
from its argument-position finding. The bare form is correct: it propagates
the child status to `set -e`. The `local` form is not, and the driver
already says so at `vmtest:28-29` — `local` is itself a command, so its own
status (always 0) is the one the shell sees and the child's is lost.
Verified on the documented target, /bin/bash 3.2.57: `x=$(f)` yields rc 60,
`local x=$(f)` yields rc 0 and continues past the fatal.

That matters more than "the run continues past a die". The side channel
records the classification but does not restore control flow, and `on_exit`
(`vmtest:659`) exits with `$rc`, not `VMTEST_EXIT` — so a failed install run
exits 0 with a MEASURE line contradicting its own exit status. `vmtest:649-657`
records that a runtime exit-code override was tried, reverted, and declined
*because this static guard covers it*. The guard is a load-bearing
precondition of the exit-code contract, not a lint.

Deletes `(local[ \t]+)?` from the regex and corrects the comment, which had
claimed both forms propagate. Currently latent: every real call site already
follows declare-first/assign-second, so the narrowed guard stays green on the
clean tree with zero call sites to repair and zero new false positives.

The selftest gains a `local`-form case (10 -> 11), because the guard's green
was not evidence of coverage without one. Confirmed load-bearing: restoring
the old regex makes that case pass the mutation, exit 0.

#5015 — the tool-name invariant was red on main

`lib/vm.sh:3` declares that file the only one in the harness that may contain
the virtualisation tool's name, so a second backend can be swapped in
(DOC-1 §12.2). The check listed two files: `lib/verify.sh:1442` named the tool
inside a prose comment, 1,429 lines after that file's own warning at :13-15
that the grep "matches COMMENTS as readily as code". A regression, not an
accepted exception — `MANIFEST.md:1913-1919` records the check green at
`f181a44e`. Reworded to name the tool by description.

CI wiring

New `.github/workflows/vmtest-harness.yml`, one lightweight job modelled on
`line-cap.yml`, path-filtered to `vmtest-harness/**` and the workflow file.
It runs the guard selftest, the guard, `subshell-classification.sh`,
`test-preflight-single-run.sh`, and the invariant grep as its own failing
step. Selftest before gate, matching line-cap.yml's ordering.

Every script is host-only: no VM is created and no network is touched.
`subshell-classification.sh` sources the driver through `--source-only`;
`test-preflight-single-run.sh` puts a stub for the virtualisation CLI on PATH
and redirects the registry and $HOME into a temporary root.

Runs on macos-14, not ubuntu-latest. The scripts pin /bin/bash 3.2.57 and
`test-preflight-single-run.sh` reads BSD `ps` and renames argv[0] with
`exec -a`; a Linux runner would exercise them under bash 5 with GNU `ps` and
could report green for semantics the real target fails. The invariant step
compares against the exact expected path, so an empty result — a vacuous scan
— fails too.

Closes #5014
Closes #5015

🤖🤖🤖 Generated with trusty-mpm — https://github.com/bobmatnyc/trusty-tools
@mac-duetto mac-duetto added trusty-mpm trusty-mpm platform and related work ws/tm-trusty-tools-22 trusty-mpm workstream tm-trusty-tools-22 labels Aug 6, 2026
@mac-duetto mac-duetto self-assigned this Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

trusty-mpm trusty-mpm platform and related work ws/tm-trusty-tools-22 trusty-mpm workstream tm-trusty-tools-22

Projects

None yet

Development

Successfully merging this pull request may close these issues.

vmtest-harness: check-no-swallowed-die exempts local x=$(f), which genuinely swallows child status

1 participant