Skip to content

[#12064] fix(lance): complete V3 type compatibility#12080

Draft
nevzheng wants to merge 5 commits into
apache:mainfrom
nevzheng:codex/v3-phase2-lance
Draft

[#12064] fix(lance): complete V3 type compatibility#12080
nevzheng wants to merge 5 commits into
apache:mainfrom
nevzheng:codex/v3-phase2-lance

Conversation

@nevzheng

Copy link
Copy Markdown
Collaborator

What changes were proposed in this pull request?

Define and document Lance compatibility for the five Gravitino V3-native type families.

Type family Outcome
Nanosecond timestamps Round-trip exact precision-9 timestamps.
Variant Reject before dataset mutation.
Unknown/Null Round-trip nullable Unknown through Arrow Null; reject unsupported non-null use.
Geometry Round-trip through GeoArrow WKB while preserving CRS.
Geography Round-trip through GeoArrow WKB while preserving CRS and edge-algorithm metadata.

Why are the changes needed?

Lance and Arrow provide exact representations for four families. Variant has no lossless target and must fail without overwriting the existing dataset.

Fix: #12064

Related: #12056

Does this PR introduce any user-facing change?

Yes. Timestamp, nullable Unknown, Geometry, and Geography gain exact mappings, while Variant fails before mutation.

How was this patch tested?

  • Added native Lance schema round-trip coverage.
  • Verified rejected Variant overwrite leaves the original dataset and metadata intact.
  • Covered the Lance-specific module suites and native integration paths.
  • Updated docs/lakehouse-generic-lance-table.md.

@github-actions

Copy link
Copy Markdown

Code Coverage Report

Overall Project 67.93% +0.02% 🟢
Files changed 71.05% 🟢

Module Coverage
aliyun 1.72% 🔴
api 47.82% 🟢
authorization-common 85.96% 🟢
aws 42.04% 🟢
azure 2.47% 🔴
catalog-common 9.92% 🔴
catalog-fileset 80.23% 🟢
catalog-glue 66.91% 🟢
catalog-hive 79.42% 🟢
catalog-jdbc-clickhouse 81.74% 🟢
catalog-jdbc-common 44.22% 🟢
catalog-jdbc-doris 81.6% 🟢
catalog-jdbc-hologres 54.03% 🟢
catalog-jdbc-mysql 79.23% 🟢
catalog-jdbc-oceanbase 80.91% 🟢
catalog-jdbc-postgresql 82.29% 🟢
catalog-jdbc-starrocks 78.51% 🟢
catalog-kafka 77.01% 🟢
catalog-lakehouse-generic 60.45% +3.12% 🟢
catalog-lakehouse-hudi 79.1% 🟢
catalog-lakehouse-iceberg 86.48% 🟢
catalog-lakehouse-paimon 84.25% 🟢
catalog-model 77.72% 🟢
cli 44.5% 🟢
client-java 78.29% 🟢
common 51.95% 🟢
core 82.53% 🟢
filesystem-hadoop3 77.3% 🟢
flink 0.0% 🔴
flink-common 47.09% 🟢
flink-runtime 0.0% 🔴
gcp 14.12% 🔴
hadoop-auth 68.0% 🟢
hadoop-common 12.7% 🔴
hive-metastore-common 53.29% 🟢
iceberg-common 63.63% 🟢
iceberg-rest-server 74.97% 🟢
idp-basic 86.02% 🟢
integration-test-common 0.0% 🔴
jobs 66.17% 🟢
lance-common 25.68% +17.53% 🔴
lance-rest-server 64.84% 🟢
lineage 53.02% 🟢
optimizer 83.24% 🟢
optimizer-api 21.95% 🔴
server 86.1% 🟢
server-common 75.78% 🟢
spark 28.57% 🔴
spark-common 46.01% 🟢
tencent 69.84% 🟢
trino-connector 40.29% 🟢
Files
Module File Coverage
catalog-lakehouse-generic LanceTableOperations.java 62.5% 🟢
lance-common LanceDataTypeConverter.java 84.86% 🟢

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Subtask] Complete V3 type compatibility for the Lance catalog

1 participant