Skip to content

fix(spark-connector): Map ExternalType to StringType to prevent UnsupportedOperationException #12104

Description

@jiangxt2

Describe what's wrong

When loading a JDBC catalog table through the Spark connector, if any column is mapped to Types.ExternalType in Gravitino metadata, loadTable() throws UnsupportedOperationException and the entire table becomes unusable.

Root cause: SparkTypeConverter.toSparkType() handles ~20 Gravitino native types but does not have a branch for Types.ExternalType. Any column with this type falls through to the final throw statement.

The problem affects all JDBC catalogs where toGravitino() produces ExternalType for database-specific types:

Catalog Types producing ExternalType
ClickHouse IPv4, IPv6, Date32, DateTime64, Decimal(>38), LowCardinality, Map, Tuple, Array, JSON, Enum, Variant, Geo, Nested, Dynamic, AggregateFunction
Doris LARGEINT, BITMAP, HLL, JSON, Variant, IPV4, IPV6
PostgreSQL interval, custom types, and unknown-type catch-all
MySQL TINYTEXT, MEDIUMTEXT, YEAR, ENUM, SET, JSON, GEOMETRY and other types
OceanBase Unknown-type catch-all
Hologres Unknown-type catch-all
Glue (via Hive connector) Unknown Hive types

The Glue catalog uses SparkHiveTypeConverter which also delegates unknown types to the base class SparkTypeConverter.toSparkType(), triggering the same crash.

Error message and/or stacktrace

java.lang.UnsupportedOperationException: Not support ExternalType: IPv4
    at org.apache.gravitino.spark.connector.SparkTypeConverter.toSparkType(SparkTypeConverter.java:196)
    at org.apache.gravitino.spark.connector.jdbc.SparkJdbcTypeConverter.toSparkType(SparkJdbcTypeConverter.java:39)
    at org.apache.gravitino.spark.connector.utils.GravitinoTableInfoHelper.getSchema(GravitinoTableInfoHelper.java:89)

How to reproduce

  1. Register a ClickHouse catalog in Gravitino with a table containing an IPv4 column:

    CREATE TABLE test.ipv4_table (id Int32, ip IPv4) ENGINE = MergeTree ORDER BY id;
  2. Load the table through the Spark connector:

    SELECT * FROM clickhouse_catalog.test.ipv4_table;
  3. Result: UnsupportedOperationException — the entire table is inaccessible.

Proposed fix

Add an ExternalType → DataTypes.StringType mapping in SparkTypeConverter.toSparkType() before the final throw. This follows the existing pattern used for UUID (also mapped to StringType because Spark has no native UUID type). Placing the fix in the base class automatically covers all subclasses (JDBC, Hive, and Glue connectors across all Spark versions 3.3/3.4/3.5).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions