fix(probe): retry stale staged replies; carry probe failures into health#106
Merged
Conversation
Two probe-robustness fixes from the #138 bench restart-roulette: Retry stale staged replies (#103): an ungracefully interrupted master leaves its last requested_opcode staged on the peripheral, so the next probe's query can read that old frame back before the new SET_REPLY takes effect. The probe treated any unexpected reply opcode as fatal and excluded the device until the next restart - observed twice in a row on the bench with a different victim each time (whichever slice the previous runtime instance was polling when it was killed). Version and caps queries now re-issue the whole SET_REPLY round-trip up to 3 times before giving up; a persistently wrong reply still excludes. Carry probe failures into the missing-device report (#104): a configured device whose address was probed and failed landed in missing_expected_ids with the generic reason 'expected device not found during startup'. The probe failure (status + detail) now travels with the id - into the readiness failed_devices reason, a missing_detail health metric, and the startup log line - so operators see WHY a device is absent without the provider log. The runtime-side visibility gap (provider AVAILABLE with missing configured devices, install health passing) is anolishq/anolis#185. Tests: ScriptedTransport gains a stale-reply queue; new cases cover stale-then-recover, persistently-stale exclusion, and detail propagation on the existing unresponsive-expected-device flow. Closes #103 Closes #104
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #103, closes #104 — the probe-robustness pair from the #138 bench restart-roulette.
#103 — stale staged replies are transient, not fatal
An ungracefully interrupted master leaves its last
requested_opcodestaged on the peripheral; the next probe's query can read that old frame back before its own SET_REPLY takes effect. The probe treated any unexpected reply opcode as fatal → device excluded until the next restart (observed twice consecutively on the bench, different victim each time — whichever slice the dying runtime was polling).probe_device's version and caps queries now go throughquery_reply(), which re-issues the whole SET_REPLY round-trip up to 3 times when the reply opcode doesn't match. Re-staging is exactly the recovery the failure mode needs; a persistently wrong reply still excludes the device.#104 — say WHY a device is missing
Bench journal showed bread already reports probe-excluded configured devices via
missing_expected_ids(the "vanishes" framing in the issue was wrong — corrected there). What was missing is the reason:failed_devicessaid "expected device not found during startup" even when the address was probed and failed with a specific error. The probe failure now travels with the id into the readiness reason, amissing_detailhealth metric, and the startup log line:The runtime-side half (provider shows AVAILABLE with missing configured devices; install health passes) is filed as anolishq/anolis#185.
Tests
ScriptedTransportgains a stale-reply queue (served ahead of the script regardless of the requested opcode — models the real staging behavior). New: stale-then-recover probes Supported; 5 stale frames exhaust the budget and exclude with the right detail; the existing unresponsive-expected-device test now asserts the propagated reason. 88/88, warnings-as-errors clean.