perf(dq): SORT rollups by subject for per-vehicle read pruning (scale review)#17
Open
zer0stars wants to merge 1 commit into
Open
perf(dq): SORT rollups by subject for per-vehicle read pruning (scale review)#17zer0stars wants to merge 1 commit into
zer0stars wants to merge 1 commit into
Conversation
… reads prune (scale review #4) The rollup tables were PARTITIONED BY (subject_bucket) but never SORTED, so a per-vehicle latest / dataSummary / availableSignals / location read pruned to one of 256 buckets and then scanned the WHOLE bucket (~fleet/256 rows) to return ~50 — the most common telemetry query, growing linearly with fleet size. SET SORTED BY (subject) clusters each bucket's parquet files by subject so zone maps prune to a few row groups. - First-create: emit SET SORTED BY (subject) alongside SET PARTITIONED for both rollups (matches the base tables), gated to creation like the other layout ALTERs. - Existing catalogs: migrateLatestSort applies it once at boot, gated on the sort being ABSENT (re-derived from ducklake_sort_info each boot, mirroring din's isSorted) so a normal reboot mints zero schema changes. Best-effort — a metadata hiccup logs and retries next boot rather than blocking the single writer. Existing files re-sort lazily via rollup rewrites/merges; new writes sort immediately. Tests: rollups sorted on fresh create; an existing partitioned-but-unsorted rollup is migrated (and idempotent on re-boot). Verified under concurrent materializers on real PG.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Telematics-scale fix (dq) — #4 SORT the rollups by subject
From the telematics-at-scale review.
signals_latest/events_latestwerePARTITIONED BY (subject_bucket)but neverSORTED, so the most common query (per-vehicle latest / dataSummary / availableSignals / location) pruned to one of 256 buckets and then scanned the WHOLE bucket (~fleet/256 rows) to return ~50 — growing linearly with fleet size.SET SORTED BY (subject)clusters each bucket's parquet files by subject so zone maps prune to a few row groups.SET SORTED BY (subject)alongsideSET PARTITIONEDfor both rollups (matches the base tables), gated to creation like the other layout ALTERs.migrateLatestSortapplies it once at boot, gated on the sort being ABSENT (re-derived fromducklake_sort_infoeach boot, mirroring din'sisSorted) so a normal reboot mints zero schema changes. Best-effort — a metadata hiccup logs and retries rather than blocking the single writer. Existing files re-sort lazily; new writes sort immediately.Adversarial review confirmed: no read-result change (physical-order only), the best-effort skip gate is load-bearing AND reliable (din runs the identical probe on the same PG catalog with a passing round-trip test). Operational caveat (documented in code): the one-time ALTER opens the known "din maintenance flush under concurrent read → FATAL" window; the materializer self-heals but query pods heal via K8s liveness — prefer deploying this release during a low-traffic / low-maintenance window.
Verification
go build/test ./...(14 pkgs ok) /golangci-lint(0 issues). Tests: rollups sorted on fresh create; an existing partitioned-but-unsorted rollup is migrated (idempotent on re-boot). Verified under concurrent materializers on real Postgres.🤖 Generated with Claude Code
https://claude.ai/code/session_01SAgMMjdGVCD7M1yLy7BUPq