enhance: add column-level writer policies - #537
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: jiaqizho The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #537 +/- ##
==========================================
+ Coverage 74.89% 74.99% +0.10%
==========================================
Files 149 149
Lines 13935 14005 +70
Branches 2093 2105 +12
==========================================
+ Hits 10436 10503 +67
- Misses 3499 3502 +3
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
22c25ad to
07883f6
Compare
Add shared writer properties for disabling per-column compression and statistics by column name. The new keys are exposed through the C++ API and C FFI as VECTOR_STR properties, with unknown column names warned once during writer initialization. Apply the policies in the Parquet writer by disabling compression as UNCOMPRESSED plus plain non-dictionary encoding for configured columns, and by disabling statistics for configured columns while preserving the default no-statistics behavior for BINARY and FIXED_SIZE_BINARY. Keep Vortex behavior unchanged in this pass: non-empty column policy properties log a warning and are ignored until Vortex column-level layout support is added.
07883f6 to
56148fe
Compare
Add shared writer properties for disabling per-column compression and statistics by column name. The new keys are exposed through the C++ API and C FFI as VECTOR_STR properties, with unknown column names warned once during writer initialization.
Apply the policies in the Parquet writer by disabling compression as UNCOMPRESSED plus plain non-dictionary encoding for configured columns, and by disabling statistics for configured columns while preserving the default no-statistics behavior for BINARY and FIXED_SIZE_BINARY.
Keep Vortex behavior unchanged in this pass: non-empty column policy properties log a warning and are ignored until Vortex column-level layout support is added.