Skip to content

[Bug] unix_timestamp support level disagrees with documented TimestampNTZ tz-conversion divergence #4682

Description

@andygrove

Describe the bug

CometUnixTimestamp (spark/src/main/scala/org/apache/comet/serde/datetime.scala) gives inconsistent signals for TimestampNTZType input:

  • getUnsupportedReasons states TimestampNTZType is not supported because Comet incorrectly applies timezone conversion to TimestampNTZ values.
  • isSupportedInputType returns true for TimestampNTZType, and getSupportLevel reports Compatible() for it.

So the EXPLAIN-time reason and the runtime behavior disagree. If the native path really does apply session-timezone conversion to TIMESTAMP_NTZ (as the documented divergence on unix_timestamp in docs/source/contributor-guide/spark_expressions_support.md suggests), then Comet produces a different value than Spark for non-UTC session timezones. If it does not, the reason text is stale.

This is the unix_timestamp / UnixTimestamp analog of the hour/minute/second TimestampNTZ tz-conversion bug tracked in #3180.

Steps to reproduce

With a non-UTC spark.sql.session.timeZone, evaluate unix_timestamp (and to_unix_timestamp) on a TIMESTAMP_NTZ column and compare Comet against Spark.

Expected behavior

The EXPLAIN-time reason must match runtime behavior. If the native path applies session-timezone conversion to TIMESTAMP_NTZ, the support level for that input type should be Incompatible(Some(...)) (cross-referencing #3180); if it does not, the stale reason text should be removed and the inconsistency resolved.

Additional context

Split out from #4502 (item 1), surfaced by the audit-comet-expression skill run in #4448. Cross-references #3180 (same tz-conversion divergence for hour/minute/second on TimestampNTZ).

Metadata

Metadata

Assignees

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions