perf(dq): #1c snapshot pagination + #5b incremental rollup (chaos-proven)#16
Merged
Conversation
…windows (#1c) A single fat snapshot (span can't drop below 1) with a large blob fan-out materialized its whole readDelta slice + resolveBlobs payloads at once and could OOM the single writer, which then crash-looped on the identical delta. The (cursor, to] insert delta is now read+decoded+written in row-key-ordered windows bounded by DUCKDB window byte budget (64 MiB) / row cap (100k): intermediate windows are written idempotently WITHOUT advancing the cursor, and only the final window's transaction advances it — so "cursor advanced <=> every window durable" still holds. A crash mid-span leaves the cursor un-advanced; restart re-reads the whole span and the cloud_event_id anti-join collapses windows that already landed. Proven: crash-mid-span exactly-once recovery (embedded), paginated fat snapshot under two concurrent materializers on a real Postgres catalog (exactly-once, no dupes), and small-snapshot single-window (common path unchanged).
The steady-state rollup recomputed each dirty subject's signals_latest over its
ENTIRE retained history every flush (O(history) x active fleet). It is now folded
INCREMENTALLY inside the decode-commit transaction (O(batch)):
- recency (timestamp/value_*/loc_*/loc_ts) is recomputed from the base but BOUNDED
to `timestamp >= the row's prior latest`, so it prunes every older day-partition
yet stays exact and self-healing (the new latest is in the batch or the prior row
still qualifies; deduped via ORDER BY timestamp DESC, cloud_event_id ASC).
- count = prev.count + a NOT-EXISTS delta over DISTINCT (subject,name,timestamp),
captured BEFORE the base insert so a redelivery, a same-(s,n,ts) collision, and a
replayed window each contribute 0 (idempotent). first_seen min-folds.
Composes with the #1c windows (each window folds in its own tx) and with concurrent
writers (temp scratch tables are per-connection; the fold rides the atomic commit).
The full-history rollupSelectSQL is retained only for the boot/DR rebuild and the
deferred bulk-backfill catch-up. events_latest still uses the recompute (far lower
volume); mirroring the fold to it is a follow-up.
Proven: differential test (incremental == RecomputeRollup) across redelivery,
same-timestamp collision, out-of-order, multi-window, dormant re-report; crash-replay
idempotency; a 25-round randomized fuzz; and exactness under two concurrent paginated
materializers on a real Postgres catalog.
…nts serving From a 4-lens adversarial regression review of the load-review PRs: - #5b count: captureRollupDelta inherited the anti-join's 30d probe clamp, so an ancient (>30d) redelivery counted as new and signals_latest.count drifted +1 vs RecomputeRollup. The exact-timestamp match is already partition-pruned; dropped the redundant BETWEEN so an ancient redelivery finds its existing row (count stays exact). - #8 location floor: the 90d gap-fill lookback returned (0,0) for a vehicle whose last GPS fix was 90d-1y old but still within LAKE_DECODED_RETENTION — a behavior regression vs the pre-#8 unbounded LocationAt. Widened past any realistic retention (the base is retention-pruned, so it costs nothing) to restore the prior semantics. - #5a events_latest serving: GetEventSummaries served the rollup as soon as the table EXISTED, so during the first-create migration rebuild (or for a brand-new subject before the next flush) it returned zero events. Now falls back to the live base scan when the rollup is empty for the subject. - #5a events_latest migration (Q3b): the first-create rebuild used an in-memory flag set from pre-DDL existence, so a crash mid-rebuild silently stranded dormant subjects on restart. Re-derived from DB state (events non-empty AND rollup empty) each boot, crash- safe like the loc_ts marker. Tests: ancient-redelivery count parity, empty-rollup fallback, crash-mid-migration restart rebuild; location equivalence updated. Full suite + PG concurrency green.
zer0stars
force-pushed
the
fix/load-review-exactly-once
branch
from
0b8e7a5 to
95b1c9d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Exactly-once-sensitive load fixes (dq) — #1c + #5b
Stacked on
fix/load-review-2026-07-07(PR #15). These are the two items that PR deferred because they touch the chaos-proven commit path; both are now implemented and proven under crash-replay + real-Postgres concurrency, not just unit tests.#1c — paginate a single oversized snapshot (HIGH)
A single fat snapshot (span can't drop below 1) with a large blob fan-out materialized its whole
readDeltaslice +resolveBlobspayloads and could OOM the single writer, which then crash-looped on the identical delta.(cursor, to]insert delta is read+decoded+written in row-key-ordered windows bounded by a byte budget (64 MiB) / row cap (100k). Intermediate windows are written idempotently without advancing the cursor; only the final window's transaction advances it — so "cursor advanced ⟺ every window durable" still holds. A crash mid-span leaves the cursor un-advanced; restart re-reads the whole span and thecloud_event_idanti-join collapses windows that already landed.#5b — maintain signals_latest incrementally at commit (MEDIUM, highest-traffic API)
The rollup recomputed each dirty subject over its entire retained history every flush. It is now folded incrementally inside the decode-commit transaction (O(batch)):
timestamp >= the row's prior latest— prunes old day-partitions, stays exact + self-healing (new latest is in the batch or the prior row still qualifies; deduped byORDER BY timestamp DESC, cloud_event_id ASC).prev.count + NOT-EXISTS deltaover DISTINCT(subject,name,timestamp), captured before the base insert so a redelivery, a same-(s,n,ts)collision, and a replayed window each contribute 0 (idempotent). first_seen min-folds.events_lateststill uses the recompute (far lower volume) — mirroring is a follow-up.RecomputeRollupacross redelivery / same-timestamp collision / out-of-order / multi-window / dormant; crash-replay idempotency; a 25-round randomized fuzz; and exactness under two concurrent paginated materializers on real Postgres.Verification
go build/go vet/go test ./...(14 pkgs ok) /golangci-lint run ./...(0 issues) /go test -raceon the new paths — all green. Real-PG proofs run withPG_CATALOG_DSNset.🤖 Generated with Claude Code
https://claude.ai/code/session_01SAgMMjdGVCD7M1yLy7BUPq