GlueSync partition comparator false-positives on volatile metadata, causing OOM on large-partition tables

[jamespfaulkner]

Title: GlueSync partition comparator false-positives on volatile metadata, causing OOM on large-partition tables

Summary

HiveToGluePartitionComparator.equals() classifies functionally-identical Hive/Glue partitions as "different" due to unfiltered comparison of volatile/generated metadata, inflating partitionsToUpdate and causing an OutOfMemoryError during a manual gluesync CLI run on a large-partition table.

Observed failure

Manual CLI backfill (gluesync --database-name-regex "^<database>$" --table-name-regex ".*") crashed:

INFO  [main] com.expediagroup.apiary.extensions.gluesync.cli.GlueSyncCli - Syncing table: <table> in database: <database>
INFO  [main] com.expediagroup.apiary.extensions.gluesync.listener.ApiaryGlueSync - <table> table already exists in glue, updating....
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
	at com.expediagroup.apiary.extensions.gluesync.listener.service.HiveToGlueTransformer.extractColumns(HiveToGlueTransformer.java:174)
	at com.expediagroup.apiary.extensions.gluesync.listener.service.HiveToGlueTransformer.transformPartition(HiveToGlueTransformer.java:117)
	at com.expediagroup.apiary.extensions.gluesync.listener.service.GluePartitionService.batchUpdatePartitions(GluePartitionService.java:389)
	at com.expediagroup.apiary.extensions.gluesync.listener.service.GluePartitionService.synchronizePartitions(GluePartitionService.java:322)
	at com.expediagroup.apiary.extensions.gluesync.cli.GlueSyncCli.syncTable(GlueSyncCli.java:171)
	at com.expediagroup.apiary.extensions.gluesync.cli.GlueSyncCli.syncAll(GlueSyncCli.java:127)
	at com.expediagroup.apiary.extensions.gluesync.cli.GlueSyncCliParser.main(GlueSyncCliParser.java:80)

Manually verified afterward: table schema matches between Hive and Glue, and partition count matches between Hive and Glue. The underlying data has not diverged. This points to a comparator bug, not a real sync gap.

Root cause

HiveToGluePartitionComparator.equals() (hive-event-listeners/apiary-gluesync-listener/src/main/java/com/expediagroup/apiary/extensions/gluesync/listener/service/HiveToGluePartitionComparator.java:34-73):

1. Unfiltered exact map equality on parameters — at the partition level (line 57), storage-descriptor level (line 111), and SerDe level (line 162). No exclusion of volatile/generated keys such as transient_lastDdlTime, numRows, rawDataSize, totalSize, numFiles, COLUMN_STATS_ACCURATE. Any of these changing (which happens on almost every Hive touch) makes Objects.equals(...) return false for otherwise-identical partitions.
2. Exact lastAccessTime equality with zero tolerance (lines 61-64), likely compounded by a units bug — Hive's lastAccessTime is epoch seconds, but the code does new Date(hiveLastAccess), which expects millis. Hive also updates lastAccessTime on reads, so this field will differ across nearly every partition regardless of the units bug.

Because nearly every partition therefore gets misclassified as "needs update," GluePartitionService.synchronizePartitions (GluePartitionService.java:274-339) builds an oversized partitionsToUpdate map, and batchUpdatePartitions (:377-406) re-transforms every one of them (rebuilding columns/sort-orders/SerDe info from scratch per partition, with no caching) before batching — for a table with a large partition count, this exhausts the CLI's heap.

Suggested fix

• Exclude known-volatile keys (transient_lastDdlTime, stats fields) from the parameters equality check at all three levels (partition, storage descriptor, SerDe), or compare only a defined allowlist of semantically-relevant keys.
• Fix (or remove) the lastAccessTime comparison — it is not schema/data-relevant and appears to have a seconds/millis conversion bug.
• Consider follow-up: synchronizePartitions/batchUpdatePartitions hold the full partition diff in memory and re-transform every partition from scratch with no caching (no reuse between the equality check and the eventual transformPartition call, and GluePartitionService.update() calls transformPartition twice for the same partition). Even with the comparator fixed, this remains a scaling risk for genuinely large partition counts and may be worth a follow-up streaming/caching pass.

Environment

• A Hive Metastore-backed environment with Glue sync enabled, on a table with a large partition count
• Manual CLI invocation, not the embedded listener's live-event path