feat(health): real per-device last_seen and I/O counters#105
Merged
Conversation
Track cumulative per-address statistics in the CRUMBS session, where every attempt is visible and the existing mutex already serializes: ok/failed operation counts, retried_attempts (every attempt beyond an operation's first), and the timestamp of the last successful operation. Argument/state validation failures never touch the wire and are not counted. device_health() now surfaces them per device: last_seen is the real last-contact time (probes included) instead of the provider-start constant stamped on every device, and io_ok / io_failed / io_retried_attempts land in DeviceHealth.metrics. Devices with no successful contact yet - including missing-expected devices - leave last_seen unset rather than fabricated. The retry counter is the observability half of the DCMT saga: retries that eventually succeed no longer erase the underlying failures from health output (a ~30% per-transaction failure rate hid behind the retry policy for weeks, feastorg/Slice_DCMT#3). Closes #87
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #87 — and completes the observability lesson from the DCMT saga.
Design
Stats live in the CRUMBS
Session, not the adapters: that's the only layer where every attempt is visible (retries are internal toSession), and its mutex already serializes the hot path — recording is a map bump under the lock already held. Per address:ok/failed— completed operations (post-retry outcome)retried_attempts— every attempt beyond an operation's first, including retries that eventually succeeded. This is the counter that would have exposed I2C: DCMT fails to serve ~30% of CRUMBS state queries (RLHT on the same bus: 0/360) feastorg/Slice_DCMT#3 in week one: a ~30% per-transaction failure rate hid behind 3-attempt retries for weeks because success erased the evidence.last_successtimestamp — real device contact (startup probes included)Validation failures (bad address, oversized frame, not-open) never reach the wire and are not counted.
Health surface
device_health()maps the device's address to its stats:last_seenbecomes the genuine last-contact time, andio_ok/io_failed/io_retried_attemptsland inDeviceHealth.metrics. Per the issue: devices with no successful contact (including missing-expected) leavelast_seenunset — the started_at fabrication is gone.Tests
last_seenunset +io_ok=0before any device I/O → set + counter moved after a successful read, per device (rlht0moves, untoucheddcmt0stays unset).