Describe the bug
CometUnixTimestamp (spark/src/main/scala/org/apache/comet/serde/datetime.scala) gives inconsistent signals for TimestampNTZType input:
getUnsupportedReasons states TimestampNTZType is not supported because Comet incorrectly applies timezone conversion to TimestampNTZ values.
isSupportedInputType returns true for TimestampNTZType, and getSupportLevel reports Compatible() for it.
So the EXPLAIN-time reason and the runtime behavior disagree. If the native path really does apply session-timezone conversion to TIMESTAMP_NTZ (as the documented divergence on unix_timestamp in docs/source/contributor-guide/spark_expressions_support.md suggests), then Comet produces a different value than Spark for non-UTC session timezones. If it does not, the reason text is stale.
This is the unix_timestamp / UnixTimestamp analog of the hour/minute/second TimestampNTZ tz-conversion bug tracked in #3180.
Steps to reproduce
With a non-UTC spark.sql.session.timeZone, evaluate unix_timestamp (and to_unix_timestamp) on a TIMESTAMP_NTZ column and compare Comet against Spark.
Expected behavior
The EXPLAIN-time reason must match runtime behavior. If the native path applies session-timezone conversion to TIMESTAMP_NTZ, the support level for that input type should be Incompatible(Some(...)) (cross-referencing #3180); if it does not, the stale reason text should be removed and the inconsistency resolved.
Additional context
Split out from #4502 (item 1), surfaced by the audit-comet-expression skill run in #4448. Cross-references #3180 (same tz-conversion divergence for hour/minute/second on TimestampNTZ).
Describe the bug
CometUnixTimestamp(spark/src/main/scala/org/apache/comet/serde/datetime.scala) gives inconsistent signals forTimestampNTZTypeinput:getUnsupportedReasonsstatesTimestampNTZType is not supported because Comet incorrectly applies timezone conversion to TimestampNTZ values.isSupportedInputTypereturnstrueforTimestampNTZType, andgetSupportLevelreportsCompatible()for it.So the EXPLAIN-time reason and the runtime behavior disagree. If the native path really does apply session-timezone conversion to
TIMESTAMP_NTZ(as the documented divergence onunix_timestampindocs/source/contributor-guide/spark_expressions_support.mdsuggests), then Comet produces a different value than Spark for non-UTC session timezones. If it does not, the reason text is stale.This is the
unix_timestamp/UnixTimestampanalog of the hour/minute/second TimestampNTZ tz-conversion bug tracked in #3180.Steps to reproduce
With a non-UTC
spark.sql.session.timeZone, evaluateunix_timestamp(andto_unix_timestamp) on aTIMESTAMP_NTZcolumn and compare Comet against Spark.Expected behavior
The EXPLAIN-time reason must match runtime behavior. If the native path applies session-timezone conversion to
TIMESTAMP_NTZ, the support level for that input type should beIncompatible(Some(...))(cross-referencing #3180); if it does not, the stale reason text should be removed and the inconsistency resolved.Additional context
Split out from #4502 (item 1), surfaced by the
audit-comet-expressionskill run in #4448. Cross-references #3180 (same tz-conversion divergence for hour/minute/second on TimestampNTZ).