[#12104] fix(spark-connector): Map ExternalType to StringType to prevent UnsupportedOperationException#12105
Open
jiangxt2 wants to merge 3 commits into
Open
[#12104] fix(spark-connector): Map ExternalType to StringType to prevent UnsupportedOperationException#12105jiangxt2 wants to merge 3 commits into
jiangxt2 wants to merge 3 commits into
Conversation
…o prevent UnsupportedOperationException Add ExternalType → StringType mapping in SparkTypeConverter.toSparkType() base class. This prevents tables with ExternalType columns (e.g. ClickHouse IPv4/IPv6, Doris LARGEINT/BITMAP, Glue unknown types) from throwing UnsupportedOperationException during loadTable() in the Spark connector. Placing the fix in the base class covers all subclasses across Spark versions 3.3/3.4/3.5, including JDBC, Hive, and Glue connectors. Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the Spark connector’s type conversion logic to safely handle Gravitino Types.ExternalType by mapping it to Spark StringType, preventing UnsupportedOperationException during table loads across Spark 3.3/3.4/3.5 connectors.
Changes:
- Add
ExternalType -> DataTypes.StringTypehandling in the sharedSparkTypeConverter.toSparkType()implementation. - Add/extend unit coverage for JDBC type converters (including Spark 3.4-specific converter).
- Document the
ExternalTypemapping behavior and add integration tests that exercise real-world “external” types (ClickHouse IPv4, Doris LARGEINT).
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| spark-connector/spark-common/src/main/java/org/apache/gravitino/spark/connector/SparkTypeConverter.java | Add ExternalType handling to avoid Spark table-load crashes. |
| spark-connector/spark-common/src/test/java/org/apache/gravitino/spark/connector/jdbc/TestSparkJdbcTypeConverter.java | Add unit test ensuring ExternalType converts to StringType. |
| spark-connector/v3.4/spark/src/test/java/org/apache/gravitino/spark/connector/jdbc/TestSparkJdbcTypeConverter34.java | Add Spark 3.4-specific unit test coverage for ExternalType conversion. |
| docs/spark-connector/spark-connector.md | Document UUID and ExternalType being represented as StringType. |
| catalogs/catalog-jdbc-doris/src/test/java/org/apache/gravitino/catalog/doris/integration/test/CatalogDoris4xIT.java | Add docker IT verifying LARGEINT round-trip/load behavior doesn’t break table loading. |
| catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/integration/test/CatalogClickHouseIT.java | Add docker IT verifying ClickHouse IPv4 is exposed as ExternalType in metadata. |
Code Coverage Report
Files
|
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
…merge duplicate comments - Replace CAST(ip AS STRING) with CAST(col AS BIGINT) in docs, since ExternalType is already mapped to StringType - Collapse two redundant "Verify that..." comments into one line Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Add
ExternalType → DataTypes.StringTypemapping inSparkTypeConverter.toSparkType()base class. This prevents tables withExternalTypecolumns from throwingUnsupportedOperationExceptionduringloadTable()in the Spark connector.Placing the fix in the base class covers all subclasses across Spark versions 3.3/3.4/3.5 (JDBC, Hive, and Glue connectors) with a single change.
Why are the changes needed?
SparkTypeConverter.toSparkType()handles ~20 Gravitino native types but has no branch forTypes.ExternalType. When a JDBC catalog table contains database-specific types (e.g. ClickHouseIPv4/IPv6, DorisLARGEINT/BITMAP), the Gravitino server maps them toExternalType, which then falls through to the finalthrowintoSparkType()— crashing the entire table load.Affected catalogs include ClickHouse (16+ types), Doris (7 types), MySQL, PostgreSQL, OceanBase, Hologres, and Glue.
The fix follows the existing pattern used for
UUID(also mapped toStringTypebecause Spark has no native UUID type).Fix: #12104
Does this PR introduce any user-facing change?
Yes — tables with
ExternalTypecolumns are now loadable in Spark. The columns are presented asStringType, and users canCASTto the desired type as needed, e.g.CAST(ip AS STRING).How was this patch tested?
TestSparkJdbcTypeConverter.testConvertExternalTypeToSparkString(spark-common) +TestSparkJdbcTypeConverter34(v3.4), covering LARGEINT/BITMAP/IPv4CatalogClickHouseIT.testIPv4ColumnIsExternalType— verifies ClickHouse IPv4 columns are exposed as ExternalType in Gravitino metadataCatalogDoris4xIT.testLargeintColumnIsExternalType— verifies Doris LARGEINT columns round-trip correctly