Skip to content

Fix map key path handling in Delta schema extractors#826

Merged
vinishjail97 merged 3 commits into
apache:mainfrom
brishi19791:dev/rishi/fix-delta-map-key-path
Jun 5, 2026
Merged

Fix map key path handling in Delta schema extractors#826
vinishjail97 merged 3 commits into
apache:mainfrom
brishi19791:dev/rishi/fix-delta-map-key-path

Conversation

@brishi19791

@brishi19791 brishi19791 commented Jun 1, 2026

Copy link
Copy Markdown
Contributor

Fixes #825

Both DeltaSchemaExtractor and DeltaKernelSchemaExtractor were passing MAP_VALUE_FIELD_NAME when recursing into a map key, so nested fields under a complex (struct/array/map) map key were qualified with <map>._one_field_value.<child> instead of <map>._one_field_key.<child>. This collides with the value-side fully-qualified paths and breaks any downstream lookup keyed off InternalField.parentPath (column stats, field resolution, etc.).

Changes

  • DeltaSchemaExtractor: pass MAP_KEY_FIELD_NAME (instead of MAP_VALUE_FIELD_NAME) when building the key schema's parent path.
  • DeltaKernelSchemaExtractor:
    • Same map-key path fix.
    • Guard the BinaryType branch with originalMetadata != null before reading XTABLE_LOGICAL_TYPE. Map/array recursion passes null metadata, so a binary nested inside a complex type previously NPE'd here.
    • Throw NotSupportedException on an unknown DataType instead of silently falling through (which would build an InternalSchema with a null dataType and "" name).
  • Tests:
    • TestDeltaSchemaExtractor / TestDeltaKernelSchemaExtractor: add testMapWithStructKey, covering a map whose key is a 2-field struct (nullable + non-nullable) and whose value is also a struct, asserting both _one_field_key and _one_field_value nested parentPath propagation.
    • TestDeltaKernelSchemaExtractor: add testBinaryInComplexTypesDoesNotThrow, a regression for a BinaryType nested in MapType/ArrayType receiving null originalMetadata.

@brishi19791 brishi19791 force-pushed the dev/rishi/fix-delta-map-key-path branch from f975962 to 1cde813 Compare June 2, 2026 16:32

@vinishjail97 vinishjail97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution @brishi19791

@vinishjail97

Copy link
Copy Markdown
Contributor

Changed the PR description to reflect the current state of the code.

@vinishjail97 vinishjail97 merged commit a4e729d into apache:main Jun 5, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Delta schema extractors produce incorrect parentPath for nested fields under a complex map key

2 participants