Describe what's wrong
When loading a JDBC catalog table through the Spark connector, if any column is mapped to Types.ExternalType in Gravitino metadata, loadTable() throws UnsupportedOperationException and the entire table becomes unusable.
Root cause: SparkTypeConverter.toSparkType() handles ~20 Gravitino native types but does not have a branch for Types.ExternalType. Any column with this type falls through to the final throw statement.
The problem affects all JDBC catalogs where toGravitino() produces ExternalType for database-specific types:
| Catalog |
Types producing ExternalType |
| ClickHouse |
IPv4, IPv6, Date32, DateTime64, Decimal(>38), LowCardinality, Map, Tuple, Array, JSON, Enum, Variant, Geo, Nested, Dynamic, AggregateFunction |
| Doris |
LARGEINT, BITMAP, HLL, JSON, Variant, IPV4, IPV6 |
| PostgreSQL |
interval, custom types, and unknown-type catch-all |
| MySQL |
TINYTEXT, MEDIUMTEXT, YEAR, ENUM, SET, JSON, GEOMETRY and other types |
| OceanBase |
Unknown-type catch-all |
| Hologres |
Unknown-type catch-all |
| Glue (via Hive connector) |
Unknown Hive types |
The Glue catalog uses SparkHiveTypeConverter which also delegates unknown types to the base class SparkTypeConverter.toSparkType(), triggering the same crash.
Error message and/or stacktrace
java.lang.UnsupportedOperationException: Not support ExternalType: IPv4
at org.apache.gravitino.spark.connector.SparkTypeConverter.toSparkType(SparkTypeConverter.java:196)
at org.apache.gravitino.spark.connector.jdbc.SparkJdbcTypeConverter.toSparkType(SparkJdbcTypeConverter.java:39)
at org.apache.gravitino.spark.connector.utils.GravitinoTableInfoHelper.getSchema(GravitinoTableInfoHelper.java:89)
How to reproduce
-
Register a ClickHouse catalog in Gravitino with a table containing an IPv4 column:
CREATE TABLE test.ipv4_table (id Int32, ip IPv4) ENGINE = MergeTree ORDER BY id;
-
Load the table through the Spark connector:
SELECT * FROM clickhouse_catalog.test.ipv4_table;
-
Result: UnsupportedOperationException — the entire table is inaccessible.
Proposed fix
Add an ExternalType → DataTypes.StringType mapping in SparkTypeConverter.toSparkType() before the final throw. This follows the existing pattern used for UUID (also mapped to StringType because Spark has no native UUID type). Placing the fix in the base class automatically covers all subclasses (JDBC, Hive, and Glue connectors across all Spark versions 3.3/3.4/3.5).
Describe what's wrong
When loading a JDBC catalog table through the Spark connector, if any column is mapped to
Types.ExternalTypein Gravitino metadata,loadTable()throwsUnsupportedOperationExceptionand the entire table becomes unusable.Root cause:
SparkTypeConverter.toSparkType()handles ~20 Gravitino native types but does not have a branch forTypes.ExternalType. Any column with this type falls through to the finalthrowstatement.The problem affects all JDBC catalogs where
toGravitino()producesExternalTypefor database-specific types:The Glue catalog uses
SparkHiveTypeConverterwhich also delegates unknown types to the base classSparkTypeConverter.toSparkType(), triggering the same crash.Error message and/or stacktrace
How to reproduce
Register a ClickHouse catalog in Gravitino with a table containing an
IPv4column:Load the table through the Spark connector:
Result:
UnsupportedOperationException— the entire table is inaccessible.Proposed fix
Add an
ExternalType → DataTypes.StringTypemapping inSparkTypeConverter.toSparkType()before the finalthrow. This follows the existing pattern used forUUID(also mapped toStringTypebecause Spark has no native UUID type). Placing the fix in the base class automatically covers all subclasses (JDBC, Hive, and Glue connectors across all Spark versions 3.3/3.4/3.5).