Skip to content

fix: keep Stream metric scrapes off durable storage - #232

Merged
smiggleworth merged 2 commits into
mainfrom
fix/stream-metrics-scrape-liveness
Sep 3, 2026
Merged

fix: keep Stream metric scrapes off durable storage#232
smiggleworth merged 2 commits into
mainfrom
fix/stream-metrics-scrape-liveness

Conversation

@smiggleworth

@smiggleworth smiggleworth commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Summary

  • maintain Stream event and watermark metrics in a monotonic in-process projection
  • initialize the projection from persisted state at startup and advance it only after successful durable commits
  • render raw Prometheus metrics without storage scans or actor mailbox work while preserving refreshed structured admin metrics
  • yield Stream maintenance after one bucket so synchronous family actors service client commands between strict storage commits
  • pin the latest Midge main revision with bounded-memory compaction and update the auto-memory expectation

Why

Fitz Destroyer sustained pressure showed that repeated /metrics scrapes scanned growing Stream history and competed with data-plane storage and actor work. After removing that scrape path, sustained pressure exposed multi-bucket Stream maintenance monopolizing the same synchronous family actor. This keeps both observability and maintenance within the documented liveness boundary.

Validation

  • cargo fmt --all -- --check
  • cargo test --workspace --quiet (1,615 unit tests plus all integration and doc tests)
  • cargo clippy --locked --workspace --all-targets --all-features -- -D warnings -D clippy::pedantic
  • Fitz Destroyer eight-minute soak, four clients, all seven domains: passed with zero warnings, zero assertion failures, zero failed stages, and exact Queue reconciliation; 15 typed Queue 4005 retries recovered; Stream max latency 6.37s under the 10s progress gate

AI-assisted: yes

@smiggleworth
smiggleworth merged commit fbf2239 into main Sep 3, 2026
1 check passed
@smiggleworth
smiggleworth deleted the fix/stream-metrics-scrape-liveness branch September 3, 2026 14:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant