fix: validate schema type compatibility when reading parquet files - #447
fix: validate schema type compatibility when reading parquet files#447jiaqizho wants to merge 1 commit into
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: jiaqizho The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #447 +/- ##
==========================================
- Coverage 73.07% 73.07% -0.01%
==========================================
Files 130 130
Lines 12940 12939 -1
Branches 1902 1902
==========================================
- Hits 9456 9455 -1
Misses 3484 3484
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
When reading parquet files, we weren't checking whether the caller's read schema has compatible field types with the actual file schema. This could lead to subtle issues — for example, if someone passes a struct type for a field that's actually int64 in the file, Arrow's C Data Interface would happily import it with the wrong memory layout, which can cause memory corruption or ASAN violations. This adds a ValidateSchemaCompatibility check in ParquetFormatReader::open() that compares each field in the read schema against the file schema. If a field exists in both but has a different type, we now fail early with a clear error message instead of silently misinterpreting the data. The read_schema is threaded through from FormatReader::create() down to ParquetFormatReader so the validation happens right after we read the file's actual schema. Fields that only exist in the read schema (schema evolution case) are skipped. Signed-off-by: jiaqizho <jiaqi.zhou@zilliz.com>
55e70aa to
bba363a
Compare
2 participants
When reading parquet files, we weren't checking whether the caller's read schema has compatible field types with the actual file schema. This could lead to subtle issues — for example, if someone passes a struct type for a field that's actually int64 in the file, Arrow's C Data Interface would happily import it with the wrong memory layout, which can cause memory corruption or ASAN violations.
This adds a ValidateSchemaCompatibility check in ParquetFormatReader::open() that compares each field in the read schema against the file schema. If a field exists in both but has a different type, we now fail early with a clear error message instead of silently misinterpreting the data.
The read_schema is threaded through from FormatReader::create() down to ParquetFormatReader so the validation happens right after we read the file's actual schema. Fields that only exist in the read schema (schema evolution case) are skipped.