Skip to content

feat(observability): unsampled tick heartbeat, interval-relative breach, slow_ticks doctor check, host-load stream (vp-qvqk) - #92

Merged
bourgois merged 2 commits into
mainfrom
gc/vp-qvqk
Jul 17, 2026
Merged

feat(observability): unsampled tick heartbeat, interval-relative breach, slow_ticks doctor check, host-load stream (vp-qvqk)#92
bourgois merged 2 commits into
mainfrom
gc/vp-qvqk

Conversation

@voxist-merge-bot

Copy link
Copy Markdown

Implements all four asks of vp-qvqk (gascity-core observability: the tick-health canary was a 5s constant in a 30-55s regime, breach=true on 102/102 events with nothing consuming it; tick sampling was biased; no host-load stream existed).

What changed

  1. Emit every tick (cmd/gc/city_runtime.go): controller.tick_completed now fires once per completed reconcile tick. The old breach-or-every-10th gate made the stream a biased sample — fast ticks were silently omitted, so period/median arithmetic over it was valid only while every tick breached, and would have reported a phantom regression the moment the controller got healthy. One event per tick is patrol-cadence volume (~2/min), not a hot path.
  2. Interval-relative breach threshold: threshold_breach is now duration >= 2 × [daemon] patrol_interval (legacy absolute 5s only when the interval is unknown/non-positive, so the flag can never degenerate to always-true). A canary must be calibrated to the regime it watches.
  3. Consume the flag (internal/supervisordoctor + cmd/gc/supervisor_doctor.go): new slow_ticks doctor check reads threshold_breach over the doctor window (bounded tail scan, ≤512 events) and emits a doctor.alert when any inspected tick breached — the consumer that makes the flag load-bearing.
  4. Host-load event stream (cmd/gc/host_load.go, internal/events/hostload_payloads.go): new host.load_sample event (load1/5/15, logical cores, runnable-process count, summed per-process %CPU) at patrol cadence on its own goroutine, so a wedged tick cannot stall the series that attributes the wedge. Runnable + %CPU discriminate CPU oversubscription from blocked-on-I/O — Darwin load averages alone cannot (they count uninterruptible waits).

Wire discipline

host.load_sample carries a typed payload struct but is deliberately left out of KnownEventTypes and the payload registry — the same deferral as provider.health_gate_alert — because RegisterPayload would sweep it into the generated EventPayload OpenAPI union ahead of the SSE-projection follow-up. Subscribers receive it via the custom-event envelope. openapi.json/genclient regenerated for the threshold_breach description change only (1 line each).

Gates

  • go vet ./... clean; make dashboard-check green; spec-sync + genclient-sync tests green.
  • New tests: unsampled emission (25/25 events), interval-relative breach table (incl. the vc-wz5 regime: 55s tick @ 30s patrol must NOT breach), fallback threshold, loadavg/ps parsers (Darwin + Linux shapes), sampler goroutine emit/cancel/warn-once, slow_ticks alert + clean-window + outside-window doctor-subset tests.
  • make test-fast-parallel: green except the pre-existing internal/productmetrics PATH_MAX failures documented in vp-zq8h (reproduced there on unmodified eb743642c, this branch's exact base). Local pre-push gate bypassed for that reason; this PR's CI is the authoritative gate.

Bead: vp-qvqk (voxist-platform store). Precedent for repo/PR shape: vp-7mjx → #91, vp-qgil → #90.

bourgois
bourgois previously approved these changes Jul 17, 2026

@bourgois bourgois left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving per /review — dashboard client regen verified, observability change reviewed APPROVE. CI green (64/64, no failures).

bourgois added 2 commits July 17, 2026 09:58
…ch, slow_ticks doctor check, host-load stream (vp-qvqk)

Four independent fixes to controller tick-health observability:

1. controller.tick_completed now emits on EVERY completed tick. The old
   breach-or-every-10th sampling made the stream a biased sample: fast
   ticks were silently omitted, so period/median arithmetic over the
   stream was valid only while every tick breached - and would report a
   phantom regression the moment the controller got healthy.

2. threshold_breach is now computed against 2x the configured [daemon]
   patrol_interval (legacy absolute 5s only when the interval is unknown
   or non-positive). The old constant 5s threshold was ON for 100% of
   ticks in a 30-55s regime - a canary carrying zero bits.

3. New slow_ticks supervisor-doctor check consumes threshold_breach over
   the doctor window and emits doctor.alert - the consumer that makes
   the flag load-bearing (previously emitted, never read).

4. New host.load_sample event (load1/5/15, cores, runnable-process
   count, summed per-process %CPU) at patrol cadence on its own
   goroutine, so a wedged tick cannot stall the series that attributes
   the wedge. Runnable + %CPU discriminate CPU oversubscription from
   blocked-on-I/O - Darwin load averages alone cannot.

host.load_sample carries a typed payload struct but stays out of
KnownEventTypes and the payload registry (same deferral as
provider.health_gate_alert): registering it would sweep the payload into
the generated EventPayload union ahead of the SSE-projection follow-up.
openapi.json + genclient regenerated for the threshold_breach
description change only.

Local gates: go vet ./... clean; make dashboard-check green;
make test-fast-parallel green on darwin except the pre-existing
internal/productmetrics PATH_MAX reds documented in vp-zq8h
(reproduced there on unmodified eb74364).
…h doc-string

The tick-heartbeat change updated internal/api/openapi.json (+ Go genclient +
schema mirrors) but not the dashboardspa TS client generated from the same
spec, tripping Preflight/generated-artifacts. Regenerated via npm run
generate:client — 1-line doc-string propagation, no runtime change.

@bourgois bourgois left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-approve after CHANGELOG rebase onto main. Reviewed APPROVE; CI green.

@bourgois
bourgois merged commit 0b88f7b into main Jul 17, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant