Tracking the operator follow-up from PR #50 (raised in review).
Before the delta-rs fork fix (pinned in PR #50), light OPTIMIZE under a date = today filter rewrote files with partitionValues={date} only, silently NULLing project_id and hiding every compacted row from project-scoped queries.
Required before/immediately after deploying PR #50 to prod:
- Audit: walk
_delta_log (bounded — an unfiltered SQL count times out) for Add entries whose partitionValues lack project_id, attributing them to a commit op (bench/delta_audit.py is a starting point).
- Backfill: rewrite affected files with the full partition values, or re-OPTIMIZE the affected partitions on the fixed binary.
The regression test optimize_preserves_all_partition_values guards the fix itself; this issue only covers existing prod data.
Tracking the operator follow-up from PR #50 (raised in review).
Before the delta-rs fork fix (pinned in PR #50), light OPTIMIZE under a
date = todayfilter rewrote files with partitionValues={date} only, silently NULLingproject_idand hiding every compacted row from project-scoped queries.Required before/immediately after deploying PR #50 to prod:
_delta_log(bounded — an unfiltered SQL count times out) for Add entries whosepartitionValueslackproject_id, attributing them to a commit op (bench/delta_audit.pyis a starting point).The regression test
optimize_preserves_all_partition_valuesguards the fix itself; this issue only covers existing prod data.