Skip to content

fix(dq): partition decoded tables by real dates, not day-of-month#20

Merged
elffjs merged 1 commit into
mainfrom
fix/date-partition-triple
Jul 10, 2026
Merged

fix(dq): partition decoded tables by real dates, not day-of-month#20
elffjs merged 1 commit into
mainfrom
fix/date-partition-triple

Conversation

@elffjs

@elffjs elffjs commented Jul 10, 2026

Copy link
Copy Markdown
Member

Problem

day(x) in DuckLake is a component extraction — day-of-month 1–31, not a date truncation — so PARTITIONED BY (subject_bucket, day("timestamp")) on lake.signals / lake.events produced 31 cycling time buckets that mix every month forever (day=9 collects Jul 9, Aug 9, Dec 9, …). Partitions never close, and the anti-join dedup probe's "day partition" pruning was actually pruning on day-of-month.

Fix

Use the documented DuckLake daily-partition shape — the year/month/day triple:

ALTER TABLE lake.signals SET PARTITIONED BY (subject_bucket, year("timestamp"), month("timestamp"), day("timestamp"))
ALTER TABLE lake.events  SET PARTITIONED BY (subject_bucket, year("timestamp"), month("timestamp"), day("timestamp"))

The signals_latest / events_latest rollups stay subject_bucket-only, unchanged.

Notes

  • Known trade-off (documented in setupStatements): ducklake_merge_adjacent_files only consolidates within one partition, so a low-traffic (bucket, day) keeps its tiny files forever. Accepted for layout legibility.
  • Deploy: the ALTERs only run at table creation, so an existing catalog keeps the old spec — deployed environments (all droppable per team) should start from a fresh catalog.
  • Companion PR: fix(din): partition raw_events by real dates, not day-of-month din#10 (same bug on raw_events).

Testing

Full go test ./... passes.

🤖 Generated with Claude Code

DuckLake's day(x) is a component extraction — day-of-month 1-31, not a
date truncation — so PARTITIONED BY (subject_bucket, day("timestamp"))
on lake.signals/lake.events cycled 31 time buckets that mixed every
month forever. Partitions never closed, and the anti-join dedup probe's
"day partition" pruning was pruning day-of-month, not dates.

Replace it with the year/month/day triple, the documented DuckLake
daily-partition shape. The _latest rollups stay subject_bucket-only.

Companion to DIMO-Network/din fix/date-partition-triple (raw_events).
No live-catalog migration: the ALTERs only run at table creation, so
deployed environments (all droppable) need a fresh catalog on deploy.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@elffjs elffjs merged commit 9bccfb0 into main Jul 10, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant