Skip to content

fix(trusty-installer): probe trusty-mpm /health over HTTP instead of reporting unknown - #4935

Open
mac-duetto wants to merge 1 commit into
mainfrom
fix/tctl-probe-mpm-http
Open

fix(trusty-installer): probe trusty-mpm /health over HTTP instead of reporting unknown#4935
mac-duetto wants to merge 1 commit into
mainfrom
fix/tctl-probe-mpm-http

Conversation

@mac-duetto

Copy link
Copy Markdown
Collaborator

Closes #4925.

What changed

One match arm in crates/trusty-installer/src/commands/probe.rsprobe_member_health
now routes ManageStrategy::OwnVerb through the same HTTP transport as Launchd:

-        ManageStrategy::Launchd => probe_member_http_blocking(binary, binary),
-        ManageStrategy::OwnVerb | ManageStrategy::None => ProbeOutcome::Unprobeable,
+        ManageStrategy::Launchd | ManageStrategy::OwnVerb => {
+            probe_member_http_blocking(binary, binary)
+        }
+        ManageStrategy::None => ProbeOutcome::Unprobeable,

ManageStrategy was overloaded to mean two things that stopped being the same thing
in #4246: how a member is started/stopped, and whether its health is probeable. Those
coincided while the probe was <binary> health --json, a subprocess contract. Once
#4246 moved the transport to HTTP /health, probeability became a property of the
daemon's transport rather than its supervision model — but the match arm still keyed it
off the lifecycle enum, leaving trusty-mpm (the only OwnVerb member) reported as
unknown despite answering GET http://127.0.0.1:7880/health with 200.

Everything else in the diff is doc comments in the eight files that cited mpm as the
live "unprobeable" example, tests, a changelog.d fragment, and lockstep updates to
vmtest-harness/lib/verify.sh and the harness-contracts doc. stable_set.rs is not in
the diff at all — mpm keeps OwnVerb for lifecycle, which is what keeps needs_kickstart
unable to fire for it.

Accepted policy change

This is user-visible and was reviewed and accepted before implementation (see #4925).

A stopped trusty-mpm now yields tctl statusdown / degraded / exit 2, and
tctl installNOT VERIFIED, because mpm is required: true (stable_set.rs:181).
This makes mpm consistent with its declared required flag rather than exempt from it —
the same behaviour trusty-search, trusty-memory and trusty-review already have. CI that
gates on tctl status exit codes may start failing where it previously passed.

required: false demotion was explicitly not the chosen path. No env-var opt-out,
no probeable field, no second probe surface, no --json added to trusty-mpm health,
no mpm special-case in any caller.

Verification

Built the patched binary and ran it against the live daemon, with the pre-fix binary
installed at ~/.cargo/bin/tctl as a same-host, same-moment control:

$ ./target/debug/tctl status          # patched
  trusty-mpm         1.3.4        healthy
$ tctl status                         # pre-fix
  trusty-mpm         1.3.4        unknown

tga correctly stays n/a — the non-daemon arm is intact.

make check fails identically on this branch and on origin/main — the two Tauri
crates trusty-code-gui and trusty-mpm-gui both fail with The frontendDist configuration is set to "ui/dist" but this path doesn't exist, because crates/*/ui/dist
is gitignored and needs a pnpm build. Neither crate appears in this diff. Reproduced on
origin/main in a throwaway worktree, so the CI-equivalent gate (per
.github/workflows/ci.yml:23-28) was run instead:

Check Result
cargo fmt --all --check exit 0
cargo clippy --workspace --all-targets -- -D warnings (3 GUI excludes) exit 0
cargo test --workspace (same excludes) 21972 passed, 0 failed, 139 ignored, 207 suites
trusty-installer package suite 554 passed, 0 failed, 4 ignored
line-cap / sld-lint / doc-numbers / test-pointers / generation-artifacts / changelog-fragment 0 violations

probe.rs is 252 SLOC against the 500 cap (was 217).

Tests

  • Rewrote probe::tests::own_verb_member_is_unknownown_verb_member_is_probed_over_http.
    Inverted rather than deleted, so the reversal is recorded where the carve-out was.
  • Added probe::tests::own_verb_member_refused_when_nothing_listens — drives a real
    probe to Refused with OwnVerb and asserts needs_kickstart(&outcome, OwnVerb) == false.
    This is the safety-critical pin: a confirmed-down mpm must never be eligible for
    launchctl kickstart -k com.trusty.mpm, a label that does not exist.
  • Added probe::tests::non_daemon_member_is_unprobeable — pins that None keeps
    Unprobeable, so the widened arm cannot creep to cover every strategy.
  • Added verify_tail::tests::required_mpm_reporting_down_fails_verification.
  • stable_set::tests::mpm_uses_own_verb and manage_strategy_for_matches_the_stable_set
    are unmodified and green — proof the lifecycle axis was untouched.

The one behavioural non-Rust edit

vmtest-harness/lib/verify.sh widens the H_P accepted-health set for
plist_installed == null members by one word:

-                    accepted="${accepted} unknown"
+                    accepted="${accepted} unknown down"

Without it the harness would go red on every run: the oracle snapshots tctl stack doctor
before anything starts daemons, so post-fix mpm reads down there, and down was
previously accepted only for plist_installed == false.

Net assertive power at that step is unchanged. null ⟺ daemon + OwnVerb ⟺ trusty-mpm
alone (doctor.rs:117-123, stable_set.rs:133); a Launchd member with a missing plist
gets Some(false) and already accepted down. Pre-change mpm read unknown
unconditionally at that site, so it asserted nothing about mpm either way. not_installed
is still rejected. mpm liveness was, and remains, asserted exclusively by
verify_daemon_liveness, which runs after tctl start --json, probes /health directly,
is unconditional in all three scenarios, and fails closed (die 60).

The scoping is clean — the false and true branches are untouched. The formal set
notation in 02-harness-contracts.md was updated in lockstep.

Follow-ups, not in scope

  • tctl stack doctor and tctl stack health treat unknown health as passing #4847 stays open. It asks whether an undetermined health should advance a green verdict —
    a policy question this change does not resolve, even though it removes the only member
    that produced unknown in practice.
  • mpm's /health envelope carries no version field (probe_http.rs:164-165), so mpm
    reports healthy and can never report stale. Version-floor detection does not apply
    to it.
  • Post-widening, verify_stack_doctor cannot distinguish a serving mpm from a dead one,
    leaving verify_daemon_liveness as the sole assertion of mpm liveness. Not a regression —
    it already was — but worth recording.

🤖🤖🤖 Generated with trusty-mpm — https://github.com/bobmatnyc/trusty-tools

…reporting unknown

`probe_member_health` returned `ProbeOutcome::Unprobeable` for any member whose
`ManageStrategy` is `OwnVerb` — trusty-mpm alone — even though the daemon answers
`GET http://127.0.0.1:7880/health`, its `http_addr` discovery file is present, and
`probe_http::fixed_port_for` already maps `"trusty-mpm" => Some(7880)`.

Probeability is a property of the daemon's HTTP transport, not of its
lifecycle-management strategy. The two axes coincided while the probe was a
`<binary> health --json` subprocess and diverged when #4246 moved it to HTTP
`/health`; the match arm kept keying off the lifecycle enum. `OwnVerb` now takes
the same transport as `Launchd`; only `ManageStrategy::None` (a non-daemon) stays
`Unprobeable`.

`ManageStrategy` itself is untouched — it still governs `tctl start|stop|restart`
dispatch and `needs_kickstart`, which independently requires `Launchd`. That is
why this is a one-arm change and why mpm stays `OwnVerb` in `stable_set`: a
confirmed-down mpm now reaches `Refused` yet still cannot be handed to
`launchctl kickstart -k` against the nonexistent `com.trusty.mpm` label.

ACCEPTED POLICY CONSEQUENCE. mpm is `required: true`, and `down` — unlike
`unknown` — fails `VerifyTailReport::build` and degrades `status`'s exit code. A
user who has simply not started mpm now gets `tctl status` / `tctl stack health` /
`tctl stack doctor` -> `degraded`, exit 2, and `tctl install` -> NOT VERIFIED.
This is intended: mpm becomes consistent with its declared `required` flag rather
than exempt from it, exactly as a stopped trusty-search already behaves. No escape
hatch, env-var opt-out or `required: false` demotion was added. It also stops
`tctl up` issuing a redundant `start` against a daemon already known to be
serving.

Tests: `own_verb_member_is_unknown` is replaced by
`own_verb_member_is_probed_over_http` (the pin this issue removes, inverted rather
than deleted), plus `own_verb_member_refused_when_nothing_listens`,
`non_daemon_member_is_unprobeable`, and
`required_mpm_reporting_down_fails_verification`. The `OwnVerb` row of
`needs_kickstart_only_for_confirmed_down_launchd` is now load-bearing for a live
code path and says so. `stable_set::mpm_uses_own_verb` and
`manage_strategy_for_matches_the_stable_set` are unchanged and green — proof the
lifecycle axis was not touched.

Also corrects the doc comments and vmtest-harness sites that recorded the #4246
carve-out as a current invariant, and widens the harness oracle's `H_P` to accept
`down` for a non-launchd member pre-start (the same ordering artefact it already
accepts for a launchd member with no plist); a genuinely dead mpm still fails at
`verify_daemon_liveness`, which probes it after `tctl start --json`.

Closes #4925

🤖🤖🤖 Generated with trusty-mpm — https://github.com/bobmatnyc/trusty-tools
@mac-duetto mac-duetto self-assigned this Aug 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Probe trusty-mpm's /health over HTTP instead of reporting it unknown

1 participant