Fix map key path handling in Delta schema extractors#826
Merged
vinishjail97 merged 3 commits intoJun 5, 2026
Conversation
vinishjail97
reviewed
Jun 1, 2026
f975962 to
1cde813
Compare
vinishjail97
approved these changes
Jun 5, 2026
vinishjail97
left a comment
Contributor
There was a problem hiding this comment.
Thanks for the contribution @brishi19791
Contributor
|
Changed the PR description to reflect the current state of the code. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #825
Both
DeltaSchemaExtractorandDeltaKernelSchemaExtractorwere passingMAP_VALUE_FIELD_NAMEwhen recursing into a map key, so nested fields under a complex (struct/array/map) map key were qualified with<map>._one_field_value.<child>instead of<map>._one_field_key.<child>. This collides with the value-side fully-qualified paths and breaks any downstream lookup keyed offInternalField.parentPath(column stats, field resolution, etc.).Changes
DeltaSchemaExtractor: passMAP_KEY_FIELD_NAME(instead ofMAP_VALUE_FIELD_NAME) when building the key schema's parent path.DeltaKernelSchemaExtractor:BinaryTypebranch withoriginalMetadata != nullbefore readingXTABLE_LOGICAL_TYPE. Map/array recursion passesnullmetadata, so a binary nested inside a complex type previously NPE'd here.NotSupportedExceptionon an unknownDataTypeinstead of silently falling through (which would build anInternalSchemawith anulldataType and""name).TestDeltaSchemaExtractor/TestDeltaKernelSchemaExtractor: addtestMapWithStructKey, covering a map whose key is a 2-field struct (nullable + non-nullable) and whose value is also a struct, asserting both_one_field_keyand_one_field_valuenestedparentPathpropagation.TestDeltaKernelSchemaExtractor: addtestBinaryInComplexTypesDoesNotThrow, a regression for aBinaryTypenested inMapType/ArrayTypereceivingnulloriginalMetadata.