Skip to content

enhance: add column-level writer policies - #537

Open
jiaqizho wants to merge 1 commit into
milvus-io:mainfrom
jiaqizho:parquet-disable-compress-fsb
Open

enhance: add column-level writer policies#537
jiaqizho wants to merge 1 commit into
milvus-io:mainfrom
jiaqizho:parquet-disable-compress-fsb

Conversation

@jiaqizho

@jiaqizho jiaqizho commented May 21, 2026

Copy link
Copy Markdown
Collaborator

Add shared writer properties for disabling per-column compression and statistics by column name. The new keys are exposed through the C++ API and C FFI as VECTOR_STR properties, with unknown column names warned once during writer initialization.

Apply the policies in the Parquet writer by disabling compression as UNCOMPRESSED plus plain non-dictionary encoding for configured columns, and by disabling statistics for configured columns while preserving the default no-statistics behavior for BINARY and FIXED_SIZE_BINARY.

Keep Vortex behavior unchanged in this pass: non-empty column policy properties log a warning and are ignored until Vortex column-level layout support is added.

@sre-ci-robot
sre-ci-robot requested review from sunby and tedxu May 21, 2026 02:55
@sre-ci-robot

Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: jiaqizho
To complete the pull request process, please assign shaoting-huang after the PR has been reviewed.
You can assign the PR to them by writing /assign @shaoting-huang in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@codecov

codecov Bot commented May 21, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.02913% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 74.99%. Comparing base (746f8b9) to head (56148fe).

Files with missing lines Patch % Lines
cpp/src/writer.cpp 96.42% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #537      +/-   ##
==========================================
+ Coverage   74.89%   74.99%   +0.10%     
==========================================
  Files         149      149              
  Lines       13935    14005      +70     
  Branches     2093     2105      +12     
==========================================
+ Hits        10436    10503      +67     
- Misses       3499     3502       +3     
Flag Coverage Δ
cpp 78.12% <99.02%> (+0.09%) ⬆️
python 44.74% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@jiaqizho
jiaqizho force-pushed the parquet-disable-compress-fsb branch from 22c25ad to 07883f6 Compare May 21, 2026 07:22
Add shared writer properties for disabling per-column compression and statistics by column name. The new keys are exposed through the C++ API and C FFI as VECTOR_STR properties, with unknown column names warned once during writer initialization.

Apply the policies in the Parquet writer by disabling compression as UNCOMPRESSED plus plain non-dictionary encoding for configured columns, and by disabling statistics for configured columns while preserving the default no-statistics behavior for BINARY and FIXED_SIZE_BINARY.

Keep Vortex behavior unchanged in this pass: non-empty column policy properties log a warning and are ignored until Vortex column-level layout support is added.
@jiaqizho
jiaqizho force-pushed the parquet-disable-compress-fsb branch from 07883f6 to 56148fe Compare June 4, 2026 02:31
@jiaqizho jiaqizho changed the title enhance: skip compression for fixed-size binary Parquet columns enhance: add column-level writer policies Jun 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants