feat(observability): unsampled tick heartbeat, interval-relative breach, slow_ticks doctor check, host-load stream (vp-qvqk) - #92
Merged
Conversation
bourgois
previously approved these changes
Jul 17, 2026
bourgois
left a comment
Collaborator
There was a problem hiding this comment.
Approving per /review — dashboard client regen verified, observability change reviewed APPROVE. CI green (64/64, no failures).
…ch, slow_ticks doctor check, host-load stream (vp-qvqk) Four independent fixes to controller tick-health observability: 1. controller.tick_completed now emits on EVERY completed tick. The old breach-or-every-10th sampling made the stream a biased sample: fast ticks were silently omitted, so period/median arithmetic over the stream was valid only while every tick breached - and would report a phantom regression the moment the controller got healthy. 2. threshold_breach is now computed against 2x the configured [daemon] patrol_interval (legacy absolute 5s only when the interval is unknown or non-positive). The old constant 5s threshold was ON for 100% of ticks in a 30-55s regime - a canary carrying zero bits. 3. New slow_ticks supervisor-doctor check consumes threshold_breach over the doctor window and emits doctor.alert - the consumer that makes the flag load-bearing (previously emitted, never read). 4. New host.load_sample event (load1/5/15, cores, runnable-process count, summed per-process %CPU) at patrol cadence on its own goroutine, so a wedged tick cannot stall the series that attributes the wedge. Runnable + %CPU discriminate CPU oversubscription from blocked-on-I/O - Darwin load averages alone cannot. host.load_sample carries a typed payload struct but stays out of KnownEventTypes and the payload registry (same deferral as provider.health_gate_alert): registering it would sweep the payload into the generated EventPayload union ahead of the SSE-projection follow-up. openapi.json + genclient regenerated for the threshold_breach description change only. Local gates: go vet ./... clean; make dashboard-check green; make test-fast-parallel green on darwin except the pre-existing internal/productmetrics PATH_MAX reds documented in vp-zq8h (reproduced there on unmodified eb74364).
…h doc-string The tick-heartbeat change updated internal/api/openapi.json (+ Go genclient + schema mirrors) but not the dashboardspa TS client generated from the same spec, tripping Preflight/generated-artifacts. Regenerated via npm run generate:client — 1-line doc-string propagation, no runtime change.
bourgois
approved these changes
Jul 17, 2026
bourgois
left a comment
Collaborator
There was a problem hiding this comment.
Re-approve after CHANGELOG rebase onto main. Reviewed APPROVE; CI green.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements all four asks of vp-qvqk (gascity-core observability: the tick-health canary was a 5s constant in a 30-55s regime, breach=true on 102/102 events with nothing consuming it; tick sampling was biased; no host-load stream existed).
What changed
cmd/gc/city_runtime.go):controller.tick_completednow fires once per completed reconcile tick. The old breach-or-every-10th gate made the stream a biased sample — fast ticks were silently omitted, so period/median arithmetic over it was valid only while every tick breached, and would have reported a phantom regression the moment the controller got healthy. One event per tick is patrol-cadence volume (~2/min), not a hot path.threshold_breachis nowduration >= 2 × [daemon] patrol_interval(legacy absolute 5s only when the interval is unknown/non-positive, so the flag can never degenerate to always-true). A canary must be calibrated to the regime it watches.internal/supervisordoctor+cmd/gc/supervisor_doctor.go): newslow_ticksdoctor check readsthreshold_breachover the doctor window (bounded tail scan, ≤512 events) and emits adoctor.alertwhen any inspected tick breached — the consumer that makes the flag load-bearing.cmd/gc/host_load.go,internal/events/hostload_payloads.go): newhost.load_sampleevent (load1/5/15, logical cores, runnable-process count, summed per-process %CPU) at patrol cadence on its own goroutine, so a wedged tick cannot stall the series that attributes the wedge. Runnable + %CPU discriminate CPU oversubscription from blocked-on-I/O — Darwin load averages alone cannot (they count uninterruptible waits).Wire discipline
host.load_samplecarries a typed payload struct but is deliberately left out ofKnownEventTypesand the payload registry — the same deferral asprovider.health_gate_alert— becauseRegisterPayloadwould sweep it into the generatedEventPayloadOpenAPI union ahead of the SSE-projection follow-up. Subscribers receive it via the custom-event envelope.openapi.json/genclientregenerated for thethreshold_breachdescription change only (1 line each).Gates
go vet ./...clean;make dashboard-checkgreen; spec-sync + genclient-sync tests green.slow_ticksalert + clean-window + outside-window doctor-subset tests.make test-fast-parallel: green except the pre-existinginternal/productmetricsPATH_MAX failures documented in vp-zq8h (reproduced there on unmodifiedeb743642c, this branch's exact base). Local pre-push gate bypassed for that reason; this PR's CI is the authoritative gate.Bead: vp-qvqk (voxist-platform store). Precedent for repo/PR shape: vp-7mjx → #91, vp-qgil → #90.