From 106ae4a1e09582c1924bbae2e847ac5df9e906fa Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Fri, 23 Oct 2026 12:35:30 +0100 Subject: [PATCH 01/26] Add Apache DataFusion Comet 1.2.0 release blog post (draft) --- .../blog/2026-11-03-datafusion-comet-1.2.0.md | 441 ++++++++++++++++++ 1 file changed, 441 insertions(+) create mode 100644 content/blog/2026-11-03-datafusion-comet-1.2.0.md diff --git a/content/blog/2026-11-03-datafusion-comet-1.2.0.md b/content/blog/2026-11-03-datafusion-comet-1.2.0.md new file mode 100644 index 00000000..b4e461f1 --- /dev/null +++ b/content/blog/2026-11-03-datafusion-comet-1.2.0.md @@ -0,0 +1,441 @@ +--- +layout: post +title: Apache DataFusion Comet 1.2.0 Release +date: 2026-11-03 +author: pmc +categories: [subprojects] +--- + + + +[TOC] + +The Apache DataFusion PMC is pleased to announce version 1.2.0 of the [Comet](https://datafusion.apache.org/comet/) subproject. + +Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for +improved performance and efficiency without requiring any code changes. + +This release covers roughly five weeks of development since 1.1.0: 388 commits from 40 contributors. See the +[change log] for the full list of changes. + +[change log]: https://github.com/apache/datafusion-comet/blob/branch-1.2/docs/source/changelog/1.2.0.md + +1.2.0 is mostly about one thing: **getting the same answers as Spark**. A native engine that is fast but returns +different results is not useful, so much of this cycle went into sweeping whole classes of differences instead of +fixing one expression at a time. + +The highlights: + +- **Spark-exact timestamps, floats and collations**: native timestamp results carry the right timezone, and + `-0.0`, `0.0` and `NaN` follow Spark in hashing, comparison, sorting, aggregation and array functions. +- **Memory, spill and shuffle at scale**: fixes for memory that Spark has already reclaimed, much smaller JVM + shuffle memory use with many partitions, and smaller shuffle files. +- **Iceberg**: native writes are a documented, opt-in feature with support for format-version 3 tables, and the + native scan reads fewer nested fields and more files at once. +- **More of the plan runs natively**, and Comet reaches new places: a native Delta Lake scan (experimental), + Arrow Python UDFs (experimental), and Spark 4.2 as a supported version. + +## Results That Match Spark + +Comet's promise is that a query returns what Spark would return. In earlier releases, we fixed differences as +users found them. In 1.2.0 we worked through them systematically, starting with the two areas that produced the +most reports. + +### Timestamps and timezones + +Spark labels a timestamp with the session timezone, and Comet's native code has to produce the same label, not +only the same instant. Several native expressions got this wrong. The timezone work is now complete, and covers: + +- Native timestamp results carry the correct timezone label: `timestamp_seconds` results are labelled as UTC + timestamps ([#6341]), `date_trunc` keeps its input's label ([#6345]), and `CASE` and `COALESCE` relabel + timestamp branches instead of panicking ([#6347]). +- Session timezone IDs are normalized before they reach native code ([#6351]). +- `date_trunc` with a format that differs from row to row uses the same kernel as a literal format ([#6354]). +- The driver warns when the native and JVM timezone databases differ ([#6353]), since a mismatch is the one + thing Comet cannot hide. +- The test harness now fails a test when a native plan mislabels a timestamp, so this class of bug can't come + back unnoticed ([#6868], [#6930]). + +Thanks to [@andygrove] for the sweep and [@LinSimon-901101] for the label checks. The new +[timezone handling page](https://datafusion.apache.org/comet/contributor-guide/timezones.html) in the +contributor guide explains the rules. + +### Floating point: `-0.0`, `0.0` and `NaN` + +Spark treats `-0.0` and `0.0` as equal and every `NaN` as one value in some places, and as different values in +others, depending on the operation and the Spark version. DataFusion makes different choices. The differences +are easy to miss in testing and show up as wrong joins, groups and sort orders. In 1.2.0, these now follow Spark: + +- **Hashing**: `hash`, `xxhash64` and native shuffle hash `NaN` as Spark does ([#6413]). +- **Comparison and ordering**: `min`, `max`, `greatest` and `least` ([#6457], [#6760], [#6761]), sort and + window keys nested in arrays and structs ([#6475], [#6968]), and `percentile` for negative `NaN`s ([#6829]). +- **Array functions**: `array_remove`, `sort_array`, `array_distinct` and `array_union` ([#6518], [#6563]), and + `arrays_overlap` on the Spark versions where `-0.0` and `0.0` are one value ([#6989]). +- **Arithmetic**: `signum(-0.0)` ([#6556]) and the zero check in `Remainder` ([#6896]). +- **Maps**: Comet no longer builds a map natively when its key type holds a `FLOAT` or `DOUBLE`, because Spark 4.0+ + normalizes those keys and rejects duplicates in a way native code did not ([#6992]). + +The contributor guide now has a [floating point page](https://datafusion.apache.org/comet/contributor-guide/floating_point.html) +that says which Spark rule each expression follows and how to test for it. A sweep suite crosses edge values +with every operator context, so the next DataFusion upgrade that shifts float behavior will fail a test instead +of a customer. + +Thanks to [@andygrove] for the sweep, [@viirya] for the `greatest` and `least` fixes, [@rich7420] for +`percentile`, [@zhangfengcdt] for `signum`, and [@Weijun-H] for nested null ordering. + +### Collations and more wrong-results fixes + +Strings with a non-default collation compare differently from ordinary strings. Comet now falls back to Spark, or +runs the expression through [codegen dispatch], for collated casts, sort keys and comparisons instead of +comparing bytes ([#5302], [#6206], [#6471], [#6480]). Thanks to [@stantheman0128], [@LinSimon-901101] and +[@viirya]. + +Other fixes in this release changed the answer a query returned: + +- **Decimal `AVG`** returned `NULL` or a wrong value for sums that overflowed an intermediate `i128`, and + mixed up empty and overflowing partial states. 1.2.0 repairs the division and the buffers, and falls back to + Spark for the cases it cannot yet do correctly ([#5420], thanks to [@sunchao], and [#7017]). +- **Variance, covariance and correlation** now update and merge in Spark's order, so values that are large and + close together give Spark's result, and `regr_r2` returns `NULL` or `DIVIDE_BY_ZERO` where Spark does ([#6076], thanks to [@rich7420]). +- **Broadcasts** of sliced string, binary, array or map batches could return wrong values or fail with + `OversizedAllocationException` ([#6900], thanks to [@comphead]). +- **Join output order**: Spark drops a sort above a hash join because the join keeps its streamed side's order. + Comet's native hash join did not keep it for some inputs, so the query could return unsorted partitions + ([#6835], [#6810], thanks to [@dwsmith1983]). +- **ANSI mode**: `array_append` now evaluates its item on rows whose array is `NULL`, so it raises the errors + Spark raises ([#7025], thanks to [@kazuyukitanimura]), and `corr` raises `DIVIDE_BY_ZERO` ([#6830], thanks to [@kazuyukitanimura]). + +[codegen dispatch]: https://datafusion.apache.org/blog/2026/06/20/datafusion-comet-0.17.0/ + +## Memory, Spill and Shuffle + +1.1.0 made Comet's memory use visible. 1.2.0 fixes more of the ways it went wrong, and makes shuffle cheaper. + +### Memory + +- **Memory is returned sooner.** Native plans return their memory before the Spark task ends ([#6261]), and a + native acquire or shuffle page allocation that fails after Spark has dropped the task's entry is retried + instead of failing the task ([#6310], [#6403], [#6813]). +- **Spilling works in more places.** A final aggregate that spilled reads its spill files back past its memory + share ([#6544]), and the memory reservation of grouped integer `SUM` state is now counted ([#6370]). +- **Container sizing.** The native memory limit follows how YARN and Kubernetes size executor containers + ([#6375]). +- **Spill files are compressed with lz4 by default** (`spark.comet.exec.spill.compression.codec`, [#6698]). The + PR reports TPC-H q21 going from 380 s to 250 s, against 367 s for Spark. +- **The memory usage log is in the Spark event log**, so it can be read after the application ends ([#6401], + [#6250], [#6271]). + +Thanks to [@andygrove] for most of this work, with [@dwsmith1983] and [@viirya] on the retry fixes and [@comphead] on +spill compression and the event log. + +### Shuffle + +- **JVM shuffle with many partitions.** The bypass-merge writer gave every partition a page of Spark's page size, + 32 to 64 MB, as soon as it buffered a row. With more partitions than a task's memory could hold pages for, + every new page forced a spill. Pages now start small and grow ([#7059]). The PR reports `REPARTITION(64)` on + TPC-H `orders` going from 266 s and 2.48 GB to 1.1 s and 156 MB, and on `lineitem` from 15.5 minutes to 3.3 s. +- **Smaller shuffle files.** Offsets are stored as lengths and compressed with zstd, which makes shuffle files with + string, binary, array and map columns 9% to 40% smaller in the PR's benchmarks ([#6943]). +- **Faster spills.** A JVM shuffle spill writes every partition in one native call ([#7065]), which the PR reports + as 21% to 28% faster for 1,000 partitions and 45% to 49% faster when spilling. Spill pointers are sorted in + the sorter's scratch space ([#7125]), and shuffle blocks are read in 64 KiB pieces ([#6805]). +- **Broadcast.** Every task that read a Comet broadcast decoded the whole broadcast for itself. Tasks on an + executor now share one decoded copy ([#7016]). In the PR's benchmark, a 1M-row build side read by 128 tasks on + `local[8]` went from 1,321 ms to 684 ms. + +Thanks to [@andygrove] for this work, [@viirya] for the spill pointer sort and [@comphead] for the block reads. + +## Iceberg + +### Native writes + +Native Iceberg writes are **off by default** in 1.2.0, as they were in 1.1.0 when they were experimental. We +considered turning them on during this cycle and decided against it for this release: it is their first +release as a documented, user-facing feature, and we would rather have users opt in and report what they find +before the default changes. The default is revisited after the 1.2.0 branch is cut. + +To use them, set `spark.comet.write.iceberg.enabled=true`. This replaces the pair of settings used in 1.1.0, so see +[Upgrading to 1.2.0](#upgrading-to-120) if you enabled the feature then. What changed in the writer: + +- **Format-version 3 tables** can be written natively ([#6672]). +- **Per-location credentials** apply to native reads and writes ([#6478]). +- **Files match iceberg-java more closely**: dictionary encoding is chosen the way iceberg-java does ([#6241]), + partitions past year 262142 match ([#6239]), and float and double identity partitions are written natively + ([#6851]). +- **Memory**: write buffers are charged to the task's memory pool ([#6247]), and when the pool refuses a new + buffer the writer closes fanout partitions early instead of failing ([#6773]). +- **Speed**: the writer's dictionary pre-pass is 1.8x to 3.9x faster ([#6937]). +- **Fallbacks**: more unsupported writes are detected up front and run through iceberg-java, such as writes with + unsupported S3 settings, a void partition field and a scan that reports an output ordering ([#6441], [#6502], + [#6677], [#7043]). + +The [Iceberg guide](https://datafusion.apache.org/comet/user-guide/latest/iceberg.html#writes) lists the +supported operations, every fallback, and every known difference from iceberg-java. + +### Reads and row-level operations + +- The native scan reads only the nested fields Spark keeps ([#6725]) and opens **two data files at once** by + default ([#7132]), which hides the latency of opening files on tables with many small files or on S3. +- `MERGE INTO` can run natively in `MergeRowsExec` on Spark 3.5+ (`spark.comet.exec.mergeRows.enabled`, off by + default, [#5318]), with `MergeSummary` support on Spark 4.1+ ([#6640]) and Spark 4.2 write transactions + ([#6587]). + +Thanks to [@andygrove] for the writer work, [@snmvaughan] for per-location credentials, [@unikdahal] for the +row-level operations, [@comphead] for nested-field pruning, [@sam-1112], [@0lai0] and [@parthchandra] for +fallback gates. + +## More of the Plan Runs Natively + +- **On by default**: existence joins ([#4587], thanks to [@coderfender]), conversion of `OneRowRelation` ([#6666]), + aggregates with no grouping keys ([#6899]), `min` and `max` over strings and binary ([#6927]), and broadcast + joins whose build side holds an unsupported scan ([#6786], thanks to [@parthchandra]). +- **New operators and expressions**: `SortAggregateExec` ([#4565]), the `PivotFirst` fast path for `PIVOT` + ([#4798]), HyperLogLog sketch functions ([#4802]), distinct-combined `collect_list` ([#4727]), + `AtLeastNNonNulls` ([#6180]), `replace` ([#6680]), hashing of wide decimals ([#6548]), `RANGE` window frames + with an offset on a `DATE` key ([#4974]), and interval multiply and divide ([#4715], [#4901]). +- **Opt-in conversions**: a native `RangeExec` ([#6553]), operators above typed Dataset operations ([#6564]), + native shuffle over row input ([#6607]), and join filter pushdown through deterministic residuals ([#6431]). + Each conversion of RDD, range, cache and Data Source V1 inputs has its own `spark.comet.convert.*` setting + ([#6602]). + +### The in-memory cache stays opt-in + +Comet can store Spark's in-memory cache in Arrow format and read it natively. During this cycle we planned to +turn it on by default, and made hardening changes for that: serializer and Kryo gating ([#6360], [#6537]), +EXPLAIN and SQL-tab fidelity ([#6453], [#6574], [#6577]), planner statistics ([#6412]) and a reader class +generated once per executor ([#6860]). The audit it prompted found cases where default settings could return +rows in the wrong order, so we decided to keep the feature opt-in (`spark.comet.exec.inMemoryCache.enabled`) for +1.2.0. One of those cases is fixed ([#6835]); the rest are tracked in [#6834]. + +## Reaching New Places + +### Delta Lake (experimental) + +A new contrib module plans Delta Lake scans on the JVM and runs them in Comet's native Parquet scan, including +deletion vectors, row-group pruning and page pruning ([#5365]). It is opt-in: build the separate +`comet-contrib-delta` jar and set `spark.comet.scan.delta.enabled=true`. See the +[Delta Lake guide](https://datafusion.apache.org/comet/user-guide/latest/delta.html). Thanks to [@dwsmith1983]. + +### Python UDFs (experimental) + +Scalar Arrow Python UDFs can run in native Comet, passing Arrow data to Python without going through rows +([#6130]). It needs a native library built with the `python-udf` Cargo feature and PyArrow on every executor, and +is enabled with `spark.comet.exec.nativeArrowPythonUDF.enabled`. The PR reports a vectorized +`pyarrow.compute.negate` over 50 million rows running 3.6x faster than Spark, and a UDF that loops in Python +running 3.4x slower, so try it on your own workload before relying on it. Thanks to [@viirya]. Custom Rust UDFs +through Arrow FFI ([#4459]) remain experimental, and a new setting disables or restricts native library loading +([#6775]). + +### Spark 4.2 + +Spark 4.2.0 is now a supported version rather than an experimental one. Comet's tests and Spark's SQL tests both +run against it nightly ([#4950], [#6417]). + +## Performance + +- **`CASE WHEN` and `IF`** are up to 12x faster ([#6350]), and **string-to-timestamp casts** up to 5.6x faster + ([#5130]). Casts from integers, decimals, floats and booleans to timestamp are vectorized ([#6969]). +- **Spark-to-Arrow conversion**: columns are written in bulk ([#6566]), unsafe rows are converted straight from + their memory ([#6730]), and JVM columnar shuffle converts strings, maps and arrays faster ([#6604]). +- **Columnar to row**: a direct path is up to 1.9x faster on decimal-heavy schemas, though the PR measured it 29% + slower on a plain long, int, double and string schema ([#5496], thanks to [@peterxcli]). +- **Scans**: the next Parquet file's footer and page index load while the current file is read, which the PR + reports as up to 1.9x faster on high-latency storage ([#7094]). Opt-in TopK threshold pushdown reaches the Parquet + readers ([#6181], [#6263], thanks to [@pingzh]). +- **Joins**: decoded probe batches are filtered before intermediate joins ([#6433], thanks to [@sunchao]). + +## Upgrading to 1.2.0 + +A few changes in 1.2.0 can affect a deployment. The [Comet Upgrade Guide] has the details. + +[Comet Upgrade Guide]: https://datafusion.apache.org/comet/user-guide/latest/migration-guide.html + +- **Native Iceberg writes use one setting.** 1.1.0's `spark.comet.iceberg.write.enabled` (a testing setting that + also needed `spark.comet.write.iceberg.splitOperator.enabled`) is now `spark.comet.write.iceberg.enabled`, and + Comet ignores the old name. Parquet writes were renamed in the same way, from `spark.comet.parquet.write.enabled` + to `spark.comet.write.parquet.enabled` ([#6494]). Set the new names to keep native writes. +- **The native Iceberg scan reads two files at once.** `spark.comet.scan.icebergNative.dataFileConcurrencyLimit` now + defaults to 2. A task can use more off-heap memory, and a query without `ORDER BY` can return rows in a + different order. Set it to `1` to restore the old behavior. +- **Spark-to-Arrow conversions have their own settings.** Converting ranges, cached tables, RDDs, queries without a + `FROM` clause and Data Source V1 relations through `spark.comet.sparkToColumnar.supportedOperatorList` is + deprecated. Use the matching `spark.comet.convert.*` setting. The old setting still works and logs a warning. +- **Native libraries are extracted into private directories** ([#6987], thanks to [@manuzhang]), and the packaging + of the published artifacts changed: `comet-common`, the parent pom and `comet-spark-integration` are no longer + published ([#6888]). + +## Compatibility + +Supported platforms: + +- **Spark 3.4.3** with Java 17 and Scala 2.12/2.13 (deprecated) +- **Spark 3.5.9** with Java 17 and Scala 2.12/2.13 +- **Spark 4.0.4** with Java 17/21 and Scala 2.13 +- **Spark 4.1.3** with Java 17/21 and Scala 2.13 +- **Spark 4.2.0** with Java 17 and Scala 2.13 + +See the [Spark Version Compatibility] page for known limitations specific to each version. + +[Spark Version Compatibility]: https://datafusion.apache.org/comet/user-guide/latest/compatibility/spark-versions.html + +This release builds on **DataFusion 55.2** and **Arrow 59.2** ([#6766]). + +## Get Started with Comet 1.2.0 + +Follow the [Comet 1.2.0 Installation Guide](https://datafusion.apache.org/comet/user-guide/1.2/installation.html) +to get up and running, then point Comet at your existing Spark workloads. + +[@andygrove]: https://github.com/andygrove +[@LinSimon-901101]: https://github.com/LinSimon-901101 +[@viirya]: https://github.com/viirya +[@rich7420]: https://github.com/rich7420 +[@zhangfengcdt]: https://github.com/zhangfengcdt +[@Weijun-H]: https://github.com/Weijun-H +[@stantheman0128]: https://github.com/stantheman0128 +[@sunchao]: https://github.com/sunchao +[@comphead]: https://github.com/comphead +[@dwsmith1983]: https://github.com/dwsmith1983 +[@kazuyukitanimura]: https://github.com/kazuyukitanimura +[@snmvaughan]: https://github.com/snmvaughan +[@unikdahal]: https://github.com/unikdahal +[@sam-1112]: https://github.com/sam-1112 +[@0lai0]: https://github.com/0lai0 +[@parthchandra]: https://github.com/parthchandra +[@coderfender]: https://github.com/coderfender +[@peterxcli]: https://github.com/peterxcli +[@pingzh]: https://github.com/pingzh +[@manuzhang]: https://github.com/manuzhang + +[#6341]: https://github.com/apache/datafusion-comet/pull/6341 +[#6345]: https://github.com/apache/datafusion-comet/pull/6345 +[#6347]: https://github.com/apache/datafusion-comet/pull/6347 +[#6351]: https://github.com/apache/datafusion-comet/pull/6351 +[#6354]: https://github.com/apache/datafusion-comet/pull/6354 +[#6353]: https://github.com/apache/datafusion-comet/pull/6353 +[#6868]: https://github.com/apache/datafusion-comet/pull/6868 +[#6930]: https://github.com/apache/datafusion-comet/pull/6930 +[#6413]: https://github.com/apache/datafusion-comet/pull/6413 +[#6457]: https://github.com/apache/datafusion-comet/pull/6457 +[#6760]: https://github.com/apache/datafusion-comet/pull/6760 +[#6761]: https://github.com/apache/datafusion-comet/pull/6761 +[#6475]: https://github.com/apache/datafusion-comet/pull/6475 +[#6968]: https://github.com/apache/datafusion-comet/pull/6968 +[#6829]: https://github.com/apache/datafusion-comet/pull/6829 +[#6518]: https://github.com/apache/datafusion-comet/pull/6518 +[#6563]: https://github.com/apache/datafusion-comet/pull/6563 +[#6989]: https://github.com/apache/datafusion-comet/pull/6989 +[#6556]: https://github.com/apache/datafusion-comet/pull/6556 +[#6896]: https://github.com/apache/datafusion-comet/pull/6896 +[#6992]: https://github.com/apache/datafusion-comet/pull/6992 +[#5302]: https://github.com/apache/datafusion-comet/pull/5302 +[#6206]: https://github.com/apache/datafusion-comet/pull/6206 +[#6471]: https://github.com/apache/datafusion-comet/pull/6471 +[#6480]: https://github.com/apache/datafusion-comet/pull/6480 +[#5420]: https://github.com/apache/datafusion-comet/pull/5420 +[#7017]: https://github.com/apache/datafusion-comet/pull/7017 +[#6076]: https://github.com/apache/datafusion-comet/pull/6076 +[#6900]: https://github.com/apache/datafusion-comet/pull/6900 +[#6835]: https://github.com/apache/datafusion-comet/pull/6835 +[#6810]: https://github.com/apache/datafusion-comet/pull/6810 +[#7025]: https://github.com/apache/datafusion-comet/pull/7025 +[#6830]: https://github.com/apache/datafusion-comet/pull/6830 +[#6261]: https://github.com/apache/datafusion-comet/pull/6261 +[#6310]: https://github.com/apache/datafusion-comet/pull/6310 +[#6403]: https://github.com/apache/datafusion-comet/pull/6403 +[#6813]: https://github.com/apache/datafusion-comet/pull/6813 +[#6544]: https://github.com/apache/datafusion-comet/pull/6544 +[#6370]: https://github.com/apache/datafusion-comet/pull/6370 +[#6375]: https://github.com/apache/datafusion-comet/pull/6375 +[#6698]: https://github.com/apache/datafusion-comet/pull/6698 +[#6401]: https://github.com/apache/datafusion-comet/pull/6401 +[#6250]: https://github.com/apache/datafusion-comet/pull/6250 +[#6271]: https://github.com/apache/datafusion-comet/pull/6271 +[#7059]: https://github.com/apache/datafusion-comet/pull/7059 +[#6943]: https://github.com/apache/datafusion-comet/pull/6943 +[#7065]: https://github.com/apache/datafusion-comet/pull/7065 +[#7125]: https://github.com/apache/datafusion-comet/pull/7125 +[#6805]: https://github.com/apache/datafusion-comet/pull/6805 +[#7016]: https://github.com/apache/datafusion-comet/pull/7016 +[#6672]: https://github.com/apache/datafusion-comet/pull/6672 +[#6478]: https://github.com/apache/datafusion-comet/pull/6478 +[#6241]: https://github.com/apache/datafusion-comet/pull/6241 +[#6239]: https://github.com/apache/datafusion-comet/pull/6239 +[#6851]: https://github.com/apache/datafusion-comet/pull/6851 +[#6247]: https://github.com/apache/datafusion-comet/pull/6247 +[#6773]: https://github.com/apache/datafusion-comet/pull/6773 +[#6937]: https://github.com/apache/datafusion-comet/pull/6937 +[#6441]: https://github.com/apache/datafusion-comet/pull/6441 +[#6502]: https://github.com/apache/datafusion-comet/pull/6502 +[#6677]: https://github.com/apache/datafusion-comet/pull/6677 +[#7043]: https://github.com/apache/datafusion-comet/pull/7043 +[#6725]: https://github.com/apache/datafusion-comet/pull/6725 +[#7132]: https://github.com/apache/datafusion-comet/pull/7132 +[#5318]: https://github.com/apache/datafusion-comet/pull/5318 +[#6640]: https://github.com/apache/datafusion-comet/pull/6640 +[#6587]: https://github.com/apache/datafusion-comet/pull/6587 +[#4587]: https://github.com/apache/datafusion-comet/pull/4587 +[#6666]: https://github.com/apache/datafusion-comet/pull/6666 +[#6899]: https://github.com/apache/datafusion-comet/pull/6899 +[#6927]: https://github.com/apache/datafusion-comet/pull/6927 +[#6786]: https://github.com/apache/datafusion-comet/pull/6786 +[#4565]: https://github.com/apache/datafusion-comet/pull/4565 +[#4798]: https://github.com/apache/datafusion-comet/pull/4798 +[#4802]: https://github.com/apache/datafusion-comet/pull/4802 +[#4727]: https://github.com/apache/datafusion-comet/pull/4727 +[#6180]: https://github.com/apache/datafusion-comet/pull/6180 +[#6680]: https://github.com/apache/datafusion-comet/pull/6680 +[#6548]: https://github.com/apache/datafusion-comet/pull/6548 +[#4974]: https://github.com/apache/datafusion-comet/pull/4974 +[#4715]: https://github.com/apache/datafusion-comet/pull/4715 +[#4901]: https://github.com/apache/datafusion-comet/pull/4901 +[#6553]: https://github.com/apache/datafusion-comet/pull/6553 +[#6564]: https://github.com/apache/datafusion-comet/pull/6564 +[#6607]: https://github.com/apache/datafusion-comet/pull/6607 +[#6431]: https://github.com/apache/datafusion-comet/pull/6431 +[#6602]: https://github.com/apache/datafusion-comet/pull/6602 +[#6360]: https://github.com/apache/datafusion-comet/pull/6360 +[#6537]: https://github.com/apache/datafusion-comet/pull/6537 +[#6453]: https://github.com/apache/datafusion-comet/pull/6453 +[#6574]: https://github.com/apache/datafusion-comet/pull/6574 +[#6577]: https://github.com/apache/datafusion-comet/pull/6577 +[#6412]: https://github.com/apache/datafusion-comet/pull/6412 +[#6860]: https://github.com/apache/datafusion-comet/pull/6860 +[#6834]: https://github.com/apache/datafusion-comet/pull/6834 +[#5365]: https://github.com/apache/datafusion-comet/pull/5365 +[#6130]: https://github.com/apache/datafusion-comet/pull/6130 +[#4459]: https://github.com/apache/datafusion-comet/pull/4459 +[#6775]: https://github.com/apache/datafusion-comet/pull/6775 +[#4950]: https://github.com/apache/datafusion-comet/pull/4950 +[#6417]: https://github.com/apache/datafusion-comet/pull/6417 +[#6350]: https://github.com/apache/datafusion-comet/pull/6350 +[#5130]: https://github.com/apache/datafusion-comet/pull/5130 +[#6969]: https://github.com/apache/datafusion-comet/pull/6969 +[#6566]: https://github.com/apache/datafusion-comet/pull/6566 +[#6730]: https://github.com/apache/datafusion-comet/pull/6730 +[#6604]: https://github.com/apache/datafusion-comet/pull/6604 +[#5496]: https://github.com/apache/datafusion-comet/pull/5496 +[#7094]: https://github.com/apache/datafusion-comet/pull/7094 +[#6181]: https://github.com/apache/datafusion-comet/pull/6181 +[#6263]: https://github.com/apache/datafusion-comet/pull/6263 +[#6433]: https://github.com/apache/datafusion-comet/pull/6433 +[#6494]: https://github.com/apache/datafusion-comet/pull/6494 +[#6987]: https://github.com/apache/datafusion-comet/pull/6987 +[#6888]: https://github.com/apache/datafusion-comet/pull/6888 +[#6766]: https://github.com/apache/datafusion-comet/pull/6766 From 4adb30d242217bf43cffac482acf786725ea7662 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Sun, 25 Oct 2026 05:10:50 -0600 Subject: [PATCH 02/26] Reframe Spark compatibility intro in Comet 1.2.0 blog post --- content/blog/2026-11-03-datafusion-comet-1.2.0.md | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/content/blog/2026-11-03-datafusion-comet-1.2.0.md b/content/blog/2026-11-03-datafusion-comet-1.2.0.md index b4e461f1..317516e0 100644 --- a/content/blog/2026-11-03-datafusion-comet-1.2.0.md +++ b/content/blog/2026-11-03-datafusion-comet-1.2.0.md @@ -37,9 +37,9 @@ This release covers roughly five weeks of development since 1.1.0: 388 commits f [change log]: https://github.com/apache/datafusion-comet/blob/branch-1.2/docs/source/changelog/1.2.0.md -1.2.0 is mostly about one thing: **getting the same answers as Spark**. A native engine that is fast but returns -different results is not useful, so much of this cycle went into sweeping whole classes of differences instead of -fixing one expression at a time. +Comet has always aimed to return the same results as Spark. Much of this cycle went into the edge cases that +remained, working through whole categories at a time (timezones, floating-point `-0.0` and `NaN`, collations) +and adding tests that keep them fixed. The highlights: @@ -54,9 +54,8 @@ The highlights: ## Results That Match Spark -Comet's promise is that a query returns what Spark would return. In earlier releases, we fixed differences as -users found them. In 1.2.0 we worked through them systematically, starting with the two areas that produced the -most reports. +The differences that remain between Comet and Spark tend to cluster in edge cases that share a root cause. In +1.2.0 we went through these by category, starting with the two areas that produced the most reports. ### Timestamps and timezones From 3adb7bac52a67327a3184fcfe85d6ae109b72e85 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Wed, 28 Oct 2026 08:02:04 -0600 Subject: [PATCH 03/26] Update Comet 1.2.0 blog post for final branch-1.2 contents --- .../blog/2026-11-03-datafusion-comet-1.2.0.md | 47 +++++++++++++++---- 1 file changed, 39 insertions(+), 8 deletions(-) diff --git a/content/blog/2026-11-03-datafusion-comet-1.2.0.md b/content/blog/2026-11-03-datafusion-comet-1.2.0.md index 317516e0..b10a377c 100644 --- a/content/blog/2026-11-03-datafusion-comet-1.2.0.md +++ b/content/blog/2026-11-03-datafusion-comet-1.2.0.md @@ -32,7 +32,7 @@ The Apache DataFusion PMC is pleased to announce version 1.2.0 of the [Comet](ht Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for improved performance and efficiency without requiring any code changes. -This release covers roughly five weeks of development since 1.1.0: 388 commits from 40 contributors. See the +This release covers roughly five weeks of development since 1.1.0: 396 commits from 40 contributors. See the [change log] for the full list of changes. [change log]: https://github.com/apache/datafusion-comet/blob/branch-1.2/docs/source/changelog/1.2.0.md @@ -118,6 +118,9 @@ Other fixes in this release changed the answer a query returned: - **Join output order**: Spark drops a sort above a hash join because the join keeps its streamed side's order. Comet's native hash join did not keep it for some inputs, so the query could return unsorted partitions ([#6835], [#6810], thanks to [@dwsmith1983]). +- **Spark patch releases**: Spark 4.0.5, 4.1.4 and 4.2.1 change what `sequence` returns when `stop - start` + overflows a long, and what casting a zero with a large exponent such as `'0e79'` to decimal returns. Comet + follows the behavior of the Spark version it runs on ([#7171], thanks to [@snmvaughan]). - **ANSI mode**: `array_append` now evaluates its item on rows whose array is `NULL`, so it raises the errors Spark raises ([#7025], thanks to [@kazuyukitanimura]), and `corr` raises `DIVIDE_BY_ZERO` ([#6830], thanks to [@kazuyukitanimura]). @@ -133,7 +136,9 @@ Other fixes in this release changed the answer a query returned: native acquire or shuffle page allocation that fails after Spark has dropped the task's entry is retried instead of failing the task ([#6310], [#6403], [#6813]). - **Spilling works in more places.** A final aggregate that spilled reads its spill files back past its memory - share ([#6544]), and the memory reservation of grouped integer `SUM` state is now counted ([#6370]). + share ([#6544]), a partial merge aggregate whose input is sorted on some of its grouping keys spills instead + of failing the task ([#7222]), and the memory reservation of grouped integer `SUM` state is now counted + ([#6370]). - **Container sizing.** The native memory limit follows how YARN and Kubernetes size executor containers ([#6375]). - **Spill files are compressed with lz4 by default** (`spark.comet.exec.spill.compression.codec`, [#6698]). The @@ -204,15 +209,17 @@ fallback gates. - **On by default**: existence joins ([#4587], thanks to [@coderfender]), conversion of `OneRowRelation` ([#6666]), aggregates with no grouping keys ([#6899]), `min` and `max` over strings and binary ([#6927]), and broadcast - joins whose build side holds an unsupported scan ([#6786], thanks to [@parthchandra]). + joins whose build side holds an unsupported scan ([#6786], thanks to [@parthchandra]). A Text scan on the + build side stays in Spark unless `spark.comet.convert.text.enabled` is set, because converting it replaces + bytes that are not valid UTF-8 with U+FFFD ([#7123]). - **New operators and expressions**: `SortAggregateExec` ([#4565]), the `PivotFirst` fast path for `PIVOT` ([#4798]), HyperLogLog sketch functions ([#4802]), distinct-combined `collect_list` ([#4727]), `AtLeastNNonNulls` ([#6180]), `replace` ([#6680]), hashing of wide decimals ([#6548]), `RANGE` window frames with an offset on a `DATE` key ([#4974]), and interval multiply and divide ([#4715], [#4901]). - **Opt-in conversions**: a native `RangeExec` ([#6553]), operators above typed Dataset operations ([#6564]), native shuffle over row input ([#6607]), and join filter pushdown through deterministic residuals ([#6431]). - Each conversion of RDD, range, cache and Data Source V1 inputs has its own `spark.comet.convert.*` setting - ([#6602]). + Each conversion of RDD, range, cache, Text and Data Source V1 inputs has its own `spark.comet.convert.*` + setting ([#6602], [#7123]). ### The in-memory cache stays opt-in @@ -242,6 +249,17 @@ running 3.4x slower, so try it on your own workload before relying on it. Thanks through Arrow FFI ([#4459]) remain experimental, and a new setting disables or restricts native library loading ([#6775]). +### Parquet writes to S3 with the magic committer + +The native Parquet writer can now write to S3 when the S3A magic committer commits the write ([#6940]). Spark 4.1 +configures that committer by default when spark-hadoop-cloud is on the classpath. S3A creates each file through +the same call Spark's own writer makes, and the native writer supplies only the bytes, so the committer and S3A's +write settings, such as server-side encryption, ACLs and storage class, apply as they do for Spark. Other S3 +writes fall back to Spark. The native Parquet writer is still for testing only and is off by default +(`spark.comet.write.parquet.enabled`). See the +[Parquet guide](https://datafusion.apache.org/comet/user-guide/latest/parquet.html#writing). Thanks to +[@andygrove]. + ### Spark 4.2 Spark 4.2.0 is now a supported version rather than an experimental one. Comet's tests and Spark's SQL tests both @@ -274,8 +292,14 @@ A few changes in 1.2.0 can affect a deployment. The [Comet Upgrade Guide] has th defaults to 2. A task can use more off-heap memory, and a query without `ORDER BY` can return rows in a different order. Set it to `1` to restore the old behavior. - **Spark-to-Arrow conversions have their own settings.** Converting ranges, cached tables, RDDs, queries without a - `FROM` clause and Data Source V1 relations through `spark.comet.sparkToColumnar.supportedOperatorList` is - deprecated. Use the matching `spark.comet.convert.*` setting. The old setting still works and logs a warning. + `FROM` clause, Data Source V1 relations and Text scans through + `spark.comet.sparkToColumnar.supportedOperatorList` is deprecated. Use the matching `spark.comet.convert.*` + setting. The old setting still works and logs a warning. +- **Three known differences from Spark are new in 1.2.0**, because the operators involved now run natively where + 1.1.0 fell back: `collect_set` over `FLOAT` or `DOUBLE` next to `count(DISTINCT ...)` on Spark before 4.2 + ([#7203]), `RANGE` window frames over a `DATE` column with an offset of roughly 92 million days or more + ([#5022]), and memory accounting for `min` and `max` over strings in a hash aggregate ([#6936]). The upgrade + guide gives a setting that avoids each one. - **Native libraries are extracted into private directories** ([#6987], thanks to [@manuzhang]), and the packaging of the published artifacts changed: `comet-common`, the parent pom and `comet-spark-integration` are no longer published ([#6888]). @@ -416,7 +440,7 @@ to get up and running, then point Comet at your existing Spark workloads. [#6577]: https://github.com/apache/datafusion-comet/pull/6577 [#6412]: https://github.com/apache/datafusion-comet/pull/6412 [#6860]: https://github.com/apache/datafusion-comet/pull/6860 -[#6834]: https://github.com/apache/datafusion-comet/pull/6834 +[#6834]: https://github.com/apache/datafusion-comet/issues/6834 [#5365]: https://github.com/apache/datafusion-comet/pull/5365 [#6130]: https://github.com/apache/datafusion-comet/pull/6130 [#4459]: https://github.com/apache/datafusion-comet/pull/4459 @@ -438,3 +462,10 @@ to get up and running, then point Comet at your existing Spark workloads. [#6987]: https://github.com/apache/datafusion-comet/pull/6987 [#6888]: https://github.com/apache/datafusion-comet/pull/6888 [#6766]: https://github.com/apache/datafusion-comet/pull/6766 +[#7171]: https://github.com/apache/datafusion-comet/pull/7171 +[#7222]: https://github.com/apache/datafusion-comet/pull/7222 +[#7123]: https://github.com/apache/datafusion-comet/pull/7123 +[#7203]: https://github.com/apache/datafusion-comet/issues/7203 +[#5022]: https://github.com/apache/datafusion-comet/issues/5022 +[#6936]: https://github.com/apache/datafusion-comet/issues/6936 +[#6940]: https://github.com/apache/datafusion-comet/pull/6940 From 883f2b0c511bf79e1d314153a2ac2f637d715faa Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Sat, 31 Oct 2026 06:59:42 -0600 Subject: [PATCH 04/26] Add recent branch-1.2 backports to Comet 1.2.0 blog post --- .../blog/2026-11-03-datafusion-comet-1.2.0.md | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/content/blog/2026-11-03-datafusion-comet-1.2.0.md b/content/blog/2026-11-03-datafusion-comet-1.2.0.md index b10a377c..354ce956 100644 --- a/content/blog/2026-11-03-datafusion-comet-1.2.0.md +++ b/content/blog/2026-11-03-datafusion-comet-1.2.0.md @@ -32,7 +32,7 @@ The Apache DataFusion PMC is pleased to announce version 1.2.0 of the [Comet](ht Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for improved performance and efficiency without requiring any code changes. -This release covers roughly five weeks of development since 1.1.0: 396 commits from 40 contributors. See the +This release covers roughly five weeks of development since 1.1.0: 400 commits from 40 contributors. See the [change log] for the full list of changes. [change log]: https://github.com/apache/datafusion-comet/blob/branch-1.2/docs/source/changelog/1.2.0.md @@ -121,6 +121,16 @@ Other fixes in this release changed the answer a query returned: - **Spark patch releases**: Spark 4.0.5, 4.1.4 and 4.2.1 change what `sequence` returns when `stop - start` overflows a long, and what casting a zero with a large exponent such as `'0e79'` to decimal returns. Comet follows the behavior of the Spark version it runs on ([#7171], thanks to [@snmvaughan]). +- **Missing nested Parquet fields**: when every nested field a query asks for is missing from a Parquet file, + Comet could return a non-null struct, array or map where Spark returns `NULL`, which changed `IS NULL`, filters + and `count`. Comet now follows Spark's version-dependent rules, and falls back for nested reads of name-mapped + Iceberg tables ([#7085], thanks to [@rich7420]). +- **Nested binary read as strings**: a nested Parquet `BINARY` value read as a string that held invalid UTF-8 + became `NULL`. It is now decoded the way Comet's `CAST(binary AS STRING)` decodes it, replacing ill-formed bytes + with U+FFFD ([#7106], thanks to [@manuzhang]). +- **Identity broadcasts**: a Spark operator that keeps every row of a Comet broadcast could see the last row of + each batch repeated in place of the others, because the columnar-to-row converters reuse their row object. + The rows are now copied ([#7260], thanks to [@pingzh]). - **ANSI mode**: `array_append` now evaluates its item on rows whose array is `NULL`, so it raises the errors Spark raises ([#7025], thanks to [@kazuyukitanimura]), and `corr` raises `DIVIDE_BY_ZERO` ([#6830], thanks to [@kazuyukitanimura]). @@ -139,6 +149,9 @@ Other fixes in this release changed the answer a query returned: share ([#6544]), a partial merge aggregate whose input is sorted on some of its grouping keys spills instead of failing the task ([#7222]), and the memory reservation of grouped integer `SUM` state is now counted ([#6370]). +- **Settings reach every stage.** A job launched from a query's RDD, such as through `Dataset.rdd`, ran its map + stages with every `spark.comet.*` setting at its default, including the memory pool. Those stages now use the + session's settings ([#7302], thanks to [@comphead]). - **Container sizing.** The native memory limit follows how YARN and Kubernetes size executor containers ([#6375]). - **Spill files are compressed with lz4 by default** (`spark.comet.exec.spill.compression.codec`, [#6698]). The @@ -469,3 +482,7 @@ to get up and running, then point Comet at your existing Spark workloads. [#5022]: https://github.com/apache/datafusion-comet/issues/5022 [#6936]: https://github.com/apache/datafusion-comet/issues/6936 [#6940]: https://github.com/apache/datafusion-comet/pull/6940 +[#7085]: https://github.com/apache/datafusion-comet/pull/7085 +[#7106]: https://github.com/apache/datafusion-comet/pull/7106 +[#7260]: https://github.com/apache/datafusion-comet/pull/7260 +[#7302]: https://github.com/apache/datafusion-comet/pull/7302 From b6d45d11a9cc6406a6bc88fdbcd686a51b0a2528 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Sat, 31 Oct 2026 08:41:29 -0600 Subject: [PATCH 05/26] Review pass on Comet 1.2.0 blog post --- .../blog/2026-11-03-datafusion-comet-1.2.0.md | 289 +++++++++--------- 1 file changed, 152 insertions(+), 137 deletions(-) diff --git a/content/blog/2026-11-03-datafusion-comet-1.2.0.md b/content/blog/2026-11-03-datafusion-comet-1.2.0.md index 354ce956..567ded77 100644 --- a/content/blog/2026-11-03-datafusion-comet-1.2.0.md +++ b/content/blog/2026-11-03-datafusion-comet-1.2.0.md @@ -43,84 +43,86 @@ and adding tests that keep them fixed. The highlights: -- **Spark-exact timestamps, floats and collations**: native timestamp results carry the right timezone, and - `-0.0`, `0.0` and `NaN` follow Spark in hashing, comparison, sorting, aggregation and array functions. -- **Memory, spill and shuffle at scale**: fixes for memory that Spark has already reclaimed, much smaller JVM - shuffle memory use with many partitions, and smaller shuffle files. +- **Results that match Spark**: native timestamp results carry the right timezone, `-0.0`, `0.0` and `NaN` + follow Spark in hashing, comparison, sorting, aggregation and array functions, and fixes to decimal `AVG`, + nested types, broadcasts and join order. +- **Memory, spill and shuffle at scale**: memory returned to Spark sooner, fewer memory-related task failures, + much smaller JVM shuffle memory use with many partitions, and smaller shuffle files. - **Iceberg**: native writes are a documented, opt-in feature with support for format-version 3 tables, and the native scan reads fewer nested fields and more files at once. -- **More of the plan runs natively**, and Comet reaches new places: a native Delta Lake scan (experimental), - Arrow Python UDFs (experimental), and Spark 4.2 as a supported version. +- **More of the plan runs natively**: existence joins, `SortAggregateExec`, the `PIVOT` fast path and + HyperLogLog sketch functions, among others. +- **New places to run**: a native Delta Lake scan (experimental), Arrow Python UDFs (experimental), and Spark 4.2 + as a supported version. ## Results That Match Spark -The differences that remain between Comet and Spark tend to cluster in edge cases that share a root cause. In -1.2.0 we went through these by category, starting with the two areas that produced the most reports. +Timestamps and floating point produced the most reports, so we started there, followed by collations. Individual +fixes in other areas come after that. ### Timestamps and timezones Spark labels a timestamp with the session timezone, and Comet's native code has to produce the same label, not -only the same instant. Several native expressions got this wrong. The timezone work is now complete, and covers: +only the same instant. Several native expressions got this wrong. 1.2.0 fixes them: -- Native timestamp results carry the correct timezone label: `timestamp_seconds` results are labelled as UTC +- Native timestamp results carry the correct timezone label: `timestamp_seconds` results are labeled as UTC timestamps ([#6341]), `date_trunc` keeps its input's label ([#6345]), and `CASE` and `COALESCE` relabel timestamp branches instead of panicking ([#6347]). - Session timezone IDs are normalized before they reach native code ([#6351]). - `date_trunc` with a format that differs from row to row uses the same kernel as a literal format ([#6354]). -- The driver warns when the native and JVM timezone databases differ ([#6353]), since a mismatch is the one - thing Comet cannot hide. +- The driver warns when the native and JVM timezone databases differ ([#6353]), because Comet cannot correct + for that difference. - The test harness now fails a test when a native plan mislabels a timestamp, so this class of bug can't come back unnoticed ([#6868], [#6930]). -Thanks to [@andygrove] for the sweep and [@LinSimon-901101] for the label checks. The new -[timezone handling page](https://datafusion.apache.org/comet/contributor-guide/timezones.html) in the -contributor guide explains the rules. +Thanks to [@andygrove] for the timezone sweep and [@LinSimon-901101] for the label checks. The new +[timezone handling page] in the contributor guide explains the rules. + +[timezone handling page]: https://datafusion.apache.org/comet/contributor-guide/timezones.html ### Floating point: `-0.0`, `0.0` and `NaN` -Spark treats `-0.0` and `0.0` as equal and every `NaN` as one value in some places, and as different values in -others, depending on the operation and the Spark version. DataFusion makes different choices. The differences -are easy to miss in testing and show up as wrong joins, groups and sort orders. In 1.2.0, these now follow Spark: +In some operations Spark treats `-0.0` and `0.0` as equal and every `NaN` as one value; in others it treats them +as distinct. Which rule applies depends on the operation and the Spark version, and DataFusion makes different +choices. The differences are easy to miss in testing and show up as wrong joins, groups and sort orders. In +1.2.0, these follow Spark: - **Hashing**: `hash`, `xxhash64` and native shuffle hash `NaN` as Spark does ([#6413]). -- **Comparison and ordering**: `min`, `max`, `greatest` and `least` ([#6457], [#6760], [#6761]), sort and - window keys nested in arrays and structs ([#6475], [#6968]), and `percentile` for negative `NaN`s ([#6829]). +- **Comparison and ordering**: `min`, `max`, `greatest` and `least` ([#6457]), sort and window keys nested in + arrays and structs ([#6475]), and `percentile` for negative `NaN`s ([#6829]). - **Array functions**: `array_remove`, `sort_array`, `array_distinct` and `array_union` ([#6518], [#6563]), and `arrays_overlap` on the Spark versions where `-0.0` and `0.0` are one value ([#6989]). - **Arithmetic**: `signum(-0.0)` ([#6556]) and the zero check in `Remainder` ([#6896]). - **Maps**: Comet no longer builds a map natively when its key type holds a `FLOAT` or `DOUBLE`, because Spark 4.0+ normalizes those keys and rejects duplicates in a way native code did not ([#6992]). -The contributor guide now has a [floating point page](https://datafusion.apache.org/comet/contributor-guide/floating_point.html) -that says which Spark rule each expression follows and how to test for it. A sweep suite crosses edge values -with every operator context, so the next DataFusion upgrade that shifts float behavior will fail a test instead -of a customer. +The contributor guide now has a [floating point page] that says which Spark rule each expression follows and how +to test for it. A sweep suite crosses edge values with every operator context, so a DataFusion upgrade that +shifts float behavior will fail a test before it reaches users. -Thanks to [@andygrove] for the sweep, [@viirya] for the `greatest` and `least` fixes, [@rich7420] for -`percentile`, [@zhangfengcdt] for `signum`, and [@Weijun-H] for nested null ordering. +Thanks to [@andygrove] for the float sweep, [@rich7420] for `percentile` and [@zhangfengcdt] for `signum`. -### Collations and more wrong-results fixes +[floating point page]: https://datafusion.apache.org/comet/contributor-guide/floating_point.html -Strings with a non-default collation compare differently from ordinary strings. Comet now falls back to Spark, or -runs the expression through [codegen dispatch], for collated casts, sort keys and comparisons instead of -comparing bytes ([#5302], [#6206], [#6471], [#6480]). Thanks to [@stantheman0128], [@LinSimon-901101] and -[@viirya]. +### Collations -Other fixes in this release changed the answer a query returned: +Strings with a non-default collation compare differently from ordinary strings. For collated casts, sort keys and +comparisons, Comet now falls back to Spark or runs the expression through [codegen dispatch] instead of comparing +bytes ([#5302], [#6206], [#6471], [#6480]). Thanks to [@stantheman0128], [@LinSimon-901101] and [@viirya]. -- **Decimal `AVG`** returned `NULL` or a wrong value for sums that overflowed an intermediate `i128`, and - mixed up empty and overflowing partial states. 1.2.0 repairs the division and the buffers, and falls back to - Spark for the cases it cannot yet do correctly ([#5420], thanks to [@sunchao], and [#7017]). -- **Variance, covariance and correlation** now update and merge in Spark's order, so values that are large and - close together give Spark's result, and `regr_r2` returns `NULL` or `DIVIDE_BY_ZERO` where Spark does ([#6076], thanks to [@rich7420]). -- **Broadcasts** of sliced string, binary, array or map batches could return wrong values or fail with - `OversizedAllocationException` ([#6900], thanks to [@comphead]). -- **Join output order**: Spark drops a sort above a hash join because the join keeps its streamed side's order. - Comet's native hash join did not keep it for some inputs, so the query could return unsorted partitions - ([#6835], [#6810], thanks to [@dwsmith1983]). -- **Spark patch releases**: Spark 4.0.5, 4.1.4 and 4.2.1 change what `sequence` returns when `stop - start` - overflows a long, and what casting a zero with a large exponent such as `'0e79'` to decimal returns. Comet - follows the behavior of the Spark version it runs on ([#7171], thanks to [@snmvaughan]). +[codegen dispatch]: https://datafusion.apache.org/blog/2026/06/20/datafusion-comet-0.17.0/ + +### Other wrong-results fixes + +- **Decimal `AVG`**: results were `NULL` or wrong for sums that overflowed an intermediate `i128`, and empty and + overflowing partial states were mixed up. 1.2.0 repairs the division and the buffers ([#5420], thanks to + [@sunchao]) and falls back to Spark for the cases it cannot yet do correctly ([#7017]). +- **Variance, covariance and correlation**: these now update and merge in Spark's order, so values that are large + and close together give Spark's result, and `regr_r2` returns `NULL` or `DIVIDE_BY_ZERO` where Spark does + ([#6076], thanks to [@rich7420]). +- **Nested values in comparisons and sorts**: `greatest` and `least` compare struct arguments by position and + order nulls inside nested values first ([#6760], [#6761], thanks to [@viirya]), and native sort keeps Spark's + null ordering inside nested values ([#6968], thanks to [@Weijun-H]). - **Missing nested Parquet fields**: when every nested field a query asks for is missing from a Parquet file, Comet could return a non-null struct, array or map where Spark returns `NULL`, which changed `IS NULL`, filters and `count`. Comet now follows Spark's version-dependent rules, and falls back for nested reads of name-mapped @@ -128,13 +130,18 @@ Other fixes in this release changed the answer a query returned: - **Nested binary read as strings**: a nested Parquet `BINARY` value read as a string that held invalid UTF-8 became `NULL`. It is now decoded the way Comet's `CAST(binary AS STRING)` decodes it, replacing ill-formed bytes with U+FFFD ([#7106], thanks to [@manuzhang]). +- **Sliced broadcasts**: broadcasts of sliced string, binary, array or map batches could return wrong values or + fail with `OversizedAllocationException` ([#6900], thanks to [@comphead]). - **Identity broadcasts**: a Spark operator that keeps every row of a Comet broadcast could see the last row of - each batch repeated in place of the others, because the columnar-to-row converters reuse their row object. - The rows are now copied ([#7260], thanks to [@pingzh]). + each batch repeated in place of the others. Comet now copies each row ([#7260], thanks to [@pingzh]). +- **Join output order**: Spark drops a sort above a hash join because the join keeps its streamed side's order. + Comet's native hash join did not keep it for some inputs, so the query could return unsorted partitions + ([#6835], [#6810], thanks to [@dwsmith1983]). +- **Spark patch releases**: Spark 4.0.5, 4.1.4 and 4.2.1 change what `sequence` returns when `stop - start` + overflows a long, and the result of casting a zero with a large exponent, such as `'0e79'`, to decimal. Comet + follows the behavior of the Spark version it runs on ([#7171], thanks to [@snmvaughan]). - **ANSI mode**: `array_append` now evaluates its item on rows whose array is `NULL`, so it raises the errors - Spark raises ([#7025], thanks to [@kazuyukitanimura]), and `corr` raises `DIVIDE_BY_ZERO` ([#6830], thanks to [@kazuyukitanimura]). - -[codegen dispatch]: https://datafusion.apache.org/blog/2026/06/20/datafusion-comet-0.17.0/ + Spark raises, and `corr` raises `DIVIDE_BY_ZERO` ([#7025], [#6830], thanks to [@kazuyukitanimura]). ## Memory, Spill and Shuffle @@ -142,57 +149,55 @@ Other fixes in this release changed the answer a query returned: ### Memory -- **Memory is returned sooner.** Native plans return their memory before the Spark task ends ([#6261]), and a - native acquire or shuffle page allocation that fails after Spark has dropped the task's entry is retried - instead of failing the task ([#6310], [#6403], [#6813]). -- **Spilling works in more places.** A final aggregate that spilled reads its spill files back past its memory - share ([#6544]), a partial merge aggregate whose input is sorted on some of its grouping keys spills instead - of failing the task ([#7222]), and the memory reservation of grouped integer `SUM` state is now counted - ([#6370]). -- **Settings reach every stage.** A job launched from a query's RDD, such as through `Dataset.rdd`, ran its map - stages with every `spark.comet.*` setting at its default, including the memory pool. Those stages now use the - session's settings ([#7302], thanks to [@comphead]). +- **Memory is returned sooner.** Native plans return their memory before the Spark task ends ([#6261]). When + Spark has already dropped a task's memory entry, a failed native memory request or shuffle page allocation is + now retried instead of failing the task ([#6310], [#6403], [#6813]). +- **Spilling works in more places.** A final aggregate that spilled can now read its spill files back even when + they exceed its memory share ([#6544]). A partial merge aggregate whose input is sorted on some of its grouping + keys spills instead of failing the task ([#7222]). Grouped integer `SUM` state now counts toward the memory + reservation ([#6370]). - **Container sizing.** The native memory limit follows how YARN and Kubernetes size executor containers ([#6375]). -- **Spill files are compressed with lz4 by default** (`spark.comet.exec.spill.compression.codec`, [#6698]). The - PR reports TPC-H q21 going from 380 s to 250 s, against 367 s for Spark. -- **The memory usage log is in the Spark event log**, so it can be read after the application ends ([#6401], +- **Spill files are compressed with lz4 by default.** The codec is set by + `spark.comet.exec.spill.compression.codec` ([#6698]). The PR reports TPC-H q21 going from 380 s to 250 s, + against 367 s for Spark. +- **The memory usage log is in the Spark event log.** It can be read after the application ends ([#6401], [#6250], [#6271]). -Thanks to [@andygrove] for most of this work, with [@dwsmith1983] and [@viirya] on the retry fixes and [@comphead] on -spill compression and the event log. +Thanks to [@andygrove] for most of this work, with [@dwsmith1983] and [@viirya] on the retry fixes and +[@comphead] on spill compression and the event log. -### Shuffle +### Shuffle and broadcast - **JVM shuffle with many partitions.** The bypass-merge writer gave every partition a page of Spark's page size, - 32 to 64 MB, as soon as it buffered a row. With more partitions than a task's memory could hold pages for, - every new page forced a spill. Pages now start small and grow ([#7059]). The PR reports `REPARTITION(64)` on - TPC-H `orders` going from 266 s and 2.48 GB to 1.1 s and 156 MB, and on `lineitem` from 15.5 minutes to 3.3 s. -- **Smaller shuffle files.** Offsets are stored as lengths and compressed with zstd, which makes shuffle files with - string, binary, array and map columns 9% to 40% smaller in the PR's benchmarks ([#6943]). -- **Faster spills.** A JVM shuffle spill writes every partition in one native call ([#7065]), which the PR reports - as 21% to 28% faster for 1,000 partitions and 45% to 49% faster when spilling. Spill pointers are sorted in - the sorter's scratch space ([#7125]), and shuffle blocks are read in 64 KiB pieces ([#6805]). + 32 to 64 MB, as soon as it buffered a row. When a task's memory could not hold one page per partition, every + new page forced a spill. Pages now start small and grow ([#7059]). The PR reports `REPARTITION(64)` on TPC-H + `orders` going from 266 s and 2.48 GB to 1.1 s and 156 MB, and on `lineitem` from 15.5 minutes to 3.3 s. +- **Smaller shuffle files.** Shuffle files with string, binary, array and map columns are 9% to 40% smaller in + the PR's benchmarks, because offsets are now stored as lengths and compressed with zstd ([#6943]). +- **Faster spills.** A JVM shuffle spill writes every partition in one native call ([#7065]). The PR reports this + as 21% to 28% faster with 1,000 partitions, and 45% to 49% faster when the writer spills every 100,000 rows. + Spill pointers are now sorted in the sorter's scratch space ([#7125]). +- **Shuffle reads.** Shuffle blocks are read in 64 KiB pieces ([#6805]). - **Broadcast.** Every task that read a Comet broadcast decoded the whole broadcast for itself. Tasks on an executor now share one decoded copy ([#7016]). In the PR's benchmark, a 1M-row build side read by 128 tasks on `local[8]` went from 1,321 ms to 684 ms. -Thanks to [@andygrove] for this work, [@viirya] for the spill pointer sort and [@comphead] for the block reads. +Thanks to [@viirya] for the spill pointer sort, [@comphead] for the block reads and [@andygrove] for the rest. ## Iceberg ### Native writes -Native Iceberg writes are **off by default** in 1.2.0, as they were in 1.1.0 when they were experimental. We -considered turning them on during this cycle and decided against it for this release: it is their first -release as a documented, user-facing feature, and we would rather have users opt in and report what they find -before the default changes. The default is revisited after the 1.2.0 branch is cut. +Native Iceberg writes stay **off by default** in 1.2.0, as they were in 1.1.0 when they were experimental. We +considered turning them on, but this is their first release as a documented, user-facing feature, and we would +rather users opt in and report what they find first. We plan to revisit the default in a later release. To use them, set `spark.comet.write.iceberg.enabled=true`. This replaces the pair of settings used in 1.1.0, so see [Upgrading to 1.2.0](#upgrading-to-120) if you enabled the feature then. What changed in the writer: -- **Format-version 3 tables** can be written natively ([#6672]). -- **Per-location credentials** apply to native reads and writes ([#6478]). +- **Format-version 3 tables**: written natively ([#6672]). +- **Per-location credentials**: applied to native reads and writes ([#6478]). - **Files match iceberg-java more closely**: dictionary encoding is chosen the way iceberg-java does ([#6241]), partitions past year 262142 match ([#6239]), and float and double identity partitions are written natively ([#6851]). @@ -200,48 +205,51 @@ To use them, set `spark.comet.write.iceberg.enabled=true`. This replaces the pai buffer the writer closes fanout partitions early instead of failing ([#6773]). - **Speed**: the writer's dictionary pre-pass is 1.8x to 3.9x faster ([#6937]). - **Fallbacks**: more unsupported writes are detected up front and run through iceberg-java, such as writes with - unsupported S3 settings, a void partition field and a scan that reports an output ordering ([#6441], [#6502], - [#6677], [#7043]). + unsupported S3 settings or through a void partition field whose source column was dropped ([#6441], [#6502], + [#6677]). -The [Iceberg guide](https://datafusion.apache.org/comet/user-guide/latest/iceberg.html#writes) lists the -supported operations, every fallback, and every known difference from iceberg-java. +The [Iceberg guide] lists the supported operations, every fallback, and every known difference from iceberg-java. + +[Iceberg guide]: https://datafusion.apache.org/comet/user-guide/latest/iceberg.html#writes ### Reads and row-level operations - The native scan reads only the nested fields Spark keeps ([#6725]) and opens **two data files at once** by - default ([#7132]), which hides the latency of opening files on tables with many small files or on S3. -- `MERGE INTO` can run natively in `MergeRowsExec` on Spark 3.5+ (`spark.comet.exec.mergeRows.enabled`, off by - default, [#5318]), with `MergeSummary` support on Spark 4.1+ ([#6640]) and Spark 4.2 write transactions - ([#6587]). + default ([#7132]), which hides the latency of opening files on tables with many small files or on S3. See + [Upgrading to 1.2.0](#upgrading-to-120) for the trade-offs and how to turn it off. +- The native scan falls back to Spark when Iceberg reports an output ordering, since it does not keep that order + and Spark would otherwise drop the sort above it ([#7043]). +- `MERGE INTO` can run natively in `MergeRowsExec` on Spark 3.5+ ([#5318]). It is off by default; enable it with + `spark.comet.exec.mergeRows.enabled`. It also supports `MergeSummary` on Spark 4.1+ ([#6640]) and Spark 4.2 + write transactions ([#6587]). Thanks to [@andygrove] for the writer work, [@snmvaughan] for per-location credentials, [@unikdahal] for the -row-level operations, [@comphead] for nested-field pruning, [@sam-1112], [@0lai0] and [@parthchandra] for +row-level operations, [@comphead] for nested-field pruning, and [@sam-1112], [@0lai0] and [@parthchandra] for fallback gates. ## More of the Plan Runs Natively -- **On by default**: existence joins ([#4587], thanks to [@coderfender]), conversion of `OneRowRelation` ([#6666]), - aggregates with no grouping keys ([#6899]), `min` and `max` over strings and binary ([#6927]), and broadcast - joins whose build side holds an unsupported scan ([#6786], thanks to [@parthchandra]). A Text scan on the - build side stays in Spark unless `spark.comet.convert.text.enabled` is set, because converting it replaces - bytes that are not valid UTF-8 with U+FFFD ([#7123]). +- **On by default**: existence joins ([#4587], thanks to [@coderfender]), queries without a `FROM` clause + (`OneRowRelation`, [#6666]), aggregates with neither grouping keys nor aggregate functions ([#6899]), and `min` + and `max` over strings and binary ([#6927]). Broadcast joins whose build side holds an unsupported scan now run + natively too ([#6786], thanks to [@parthchandra]), except that a Text scan on the build side stays in Spark + unless `spark.comet.convert.text.enabled` is set, because converting it replaces bytes that are not valid UTF-8 + with U+FFFD ([#7123]). - **New operators and expressions**: `SortAggregateExec` ([#4565]), the `PivotFirst` fast path for `PIVOT` ([#4798]), HyperLogLog sketch functions ([#4802]), distinct-combined `collect_list` ([#4727]), `AtLeastNNonNulls` ([#6180]), `replace` ([#6680]), hashing of wide decimals ([#6548]), `RANGE` window frames with an offset on a `DATE` key ([#4974]), and interval multiply and divide ([#4715], [#4901]). - **Opt-in conversions**: a native `RangeExec` ([#6553]), operators above typed Dataset operations ([#6564]), native shuffle over row input ([#6607]), and join filter pushdown through deterministic residuals ([#6431]). - Each conversion of RDD, range, cache, Text and Data Source V1 inputs has its own `spark.comet.convert.*` - setting ([#6602], [#7123]). ### The in-memory cache stays opt-in -Comet can store Spark's in-memory cache in Arrow format and read it natively. During this cycle we planned to -turn it on by default, and made hardening changes for that: serializer and Kryo gating ([#6360], [#6537]), -EXPLAIN and SQL-tab fidelity ([#6453], [#6574], [#6577]), planner statistics ([#6412]) and a reader class -generated once per executor ([#6860]). The audit it prompted found cases where default settings could return -rows in the wrong order, so we decided to keep the feature opt-in (`spark.comet.exec.inMemoryCache.enabled`) for -1.2.0. One of those cases is fixed ([#6835]); the rest are tracked in [#6834]. +Comet can store Spark's in-memory cache in Arrow format and read it natively. We planned to turn this on by +default in 1.2.0 and hardened it first: serializer and Kryo gating ([#6360], [#6537]), EXPLAIN and SQL-tab +fidelity ([#6453], [#6574], [#6577]), planner statistics ([#6412]) and a reader class generated once per executor +([#6860]). Auditing the feature turned up cases where default settings could return rows in the wrong order, so +it stays opt-in (`spark.comet.exec.inMemoryCache.enabled`) for 1.2.0. One of those cases, the join output order +fix above, is in this release ([#6835]); the rest are tracked in [#6834]. ## Reaching New Places @@ -249,29 +257,31 @@ rows in the wrong order, so we decided to keep the feature opt-in (`spark.comet. A new contrib module plans Delta Lake scans on the JVM and runs them in Comet's native Parquet scan, including deletion vectors, row-group pruning and page pruning ([#5365]). It is opt-in: build the separate -`comet-contrib-delta` jar and set `spark.comet.scan.delta.enabled=true`. See the -[Delta Lake guide](https://datafusion.apache.org/comet/user-guide/latest/delta.html). Thanks to [@dwsmith1983]. +`comet-contrib-delta` jar and set `spark.comet.scan.delta.enabled=true`. See the [Delta Lake guide]. Thanks to +[@dwsmith1983]. + +[Delta Lake guide]: https://datafusion.apache.org/comet/user-guide/latest/delta.html ### Python UDFs (experimental) Scalar Arrow Python UDFs can run in native Comet, passing Arrow data to Python without going through rows -([#6130]). It needs a native library built with the `python-udf` Cargo feature and PyArrow on every executor, and -is enabled with `spark.comet.exec.nativeArrowPythonUDF.enabled`. The PR reports a vectorized +([#6130]). They need a native library built with the `python-udf` Cargo feature and PyArrow on every executor, +and are enabled with `spark.comet.exec.nativeArrowPythonUDF.enabled`. The PR reports a vectorized `pyarrow.compute.negate` over 50 million rows running 3.6x faster than Spark, and a UDF that loops in Python -running 3.4x slower, so try it on your own workload before relying on it. Thanks to [@viirya]. Custom Rust UDFs -through Arrow FFI ([#4459]) remain experimental, and a new setting disables or restricts native library loading -([#6775]). +running 3.4x slower, so try it on your own workload before relying on it. Thanks to [@viirya]. + +Custom Rust UDFs through Arrow FFI ([#4459]) remain experimental. A new setting disables or restricts native +library loading ([#6775]). ### Parquet writes to S3 with the magic committer -The native Parquet writer can now write to S3 when the S3A magic committer commits the write ([#6940]). Spark 4.1 -configures that committer by default when spark-hadoop-cloud is on the classpath. S3A creates each file through -the same call Spark's own writer makes, and the native writer supplies only the bytes, so the committer and S3A's -write settings, such as server-side encryption, ACLs and storage class, apply as they do for Spark. Other S3 -writes fall back to Spark. The native Parquet writer is still for testing only and is off by default -(`spark.comet.write.parquet.enabled`). See the -[Parquet guide](https://datafusion.apache.org/comet/user-guide/latest/parquet.html#writing). Thanks to -[@andygrove]. +The native Parquet writer can now write to S3 through the S3A magic committer ([#6940]), which Spark 4.1 +configures by default when spark-hadoop-cloud is on the classpath. S3A creates each file, so its write settings, +such as server-side encryption, ACLs and storage class, apply as they do for Spark. Other S3 writes fall back to +Spark. The native Parquet writer is still for testing only and is off by default +(`spark.comet.write.parquet.enabled`). See the [Parquet guide]. Thanks to [@andygrove]. + +[Parquet guide]: https://datafusion.apache.org/comet/user-guide/latest/parquet.html#writing ### Spark 4.2 @@ -280,15 +290,16 @@ run against it nightly ([#4950], [#6417]). ## Performance -- **`CASE WHEN` and `IF`** are up to 12x faster ([#6350]), and **string-to-timestamp casts** up to 5.6x faster - ([#5130]). Casts from integers, decimals, floats and booleans to timestamp are vectorized ([#6969]). +- **Casts and conditionals**: `CASE WHEN` and `IF` are up to 12x faster ([#6350]), string-to-timestamp casts up + to 5.6x faster ([#5130]), and casts from integers, decimals, floats and booleans to timestamp are vectorized + ([#6969]). - **Spark-to-Arrow conversion**: columns are written in bulk ([#6566]), unsafe rows are converted straight from their memory ([#6730]), and JVM columnar shuffle converts strings, maps and arrays faster ([#6604]). - **Columnar to row**: a direct path is up to 1.9x faster on decimal-heavy schemas, though the PR measured it 29% slower on a plain long, int, double and string schema ([#5496], thanks to [@peterxcli]). - **Scans**: the next Parquet file's footer and page index load while the current file is read, which the PR - reports as up to 1.9x faster on high-latency storage ([#7094]). Opt-in TopK threshold pushdown reaches the Parquet - readers ([#6181], [#6263], thanks to [@pingzh]). + reports as up to 1.9x faster on high-latency storage ([#7094]). Opt-in TopK threshold pushdown reaches the + Parquet readers ([#6181], [#6263], thanks to [@pingzh]). - **Joins**: decoded probe batches are filtered before intermediate joins ([#6433], thanks to [@sunchao]). ## Upgrading to 1.2.0 @@ -297,24 +308,28 @@ A few changes in 1.2.0 can affect a deployment. The [Comet Upgrade Guide] has th [Comet Upgrade Guide]: https://datafusion.apache.org/comet/user-guide/latest/migration-guide.html -- **Native Iceberg writes use one setting.** 1.1.0's `spark.comet.iceberg.write.enabled` (a testing setting that - also needed `spark.comet.write.iceberg.splitOperator.enabled`) is now `spark.comet.write.iceberg.enabled`, and - Comet ignores the old name. Parquet writes were renamed in the same way, from `spark.comet.parquet.write.enabled` - to `spark.comet.write.parquet.enabled` ([#6494]). Set the new names to keep native writes. +- **Native Iceberg writes use one setting.** In 1.1.0, native Iceberg writes were a testing feature that needed + both `spark.comet.iceberg.write.enabled` and `spark.comet.write.iceberg.splitOperator.enabled`. The setting is + now `spark.comet.write.iceberg.enabled`, and Comet ignores the old name. Parquet writes were renamed the same + way, from `spark.comet.parquet.write.enabled` to `spark.comet.write.parquet.enabled` ([#6494]). Set the new + names to keep native writes. - **The native Iceberg scan reads two files at once.** `spark.comet.scan.icebergNative.dataFileConcurrencyLimit` now defaults to 2. A task can use more off-heap memory, and a query without `ORDER BY` can return rows in a different order. Set it to `1` to restore the old behavior. -- **Spark-to-Arrow conversions have their own settings.** Converting ranges, cached tables, RDDs, queries without a - `FROM` clause, Data Source V1 relations and Text scans through - `spark.comet.sparkToColumnar.supportedOperatorList` is deprecated. Use the matching `spark.comet.convert.*` - setting. The old setting still works and logs a warning. -- **Three known differences from Spark are new in 1.2.0**, because the operators involved now run natively where - 1.1.0 fell back: `collect_set` over `FLOAT` or `DOUBLE` next to `count(DISTINCT ...)` on Spark before 4.2 - ([#7203]), `RANGE` window frames over a `DATE` column with an offset of roughly 92 million days or more - ([#5022]), and memory accounting for `min` and `max` over strings in a hash aggregate ([#6936]). The upgrade - guide gives a setting that avoids each one. -- **Native libraries are extracted into private directories** ([#6987], thanks to [@manuzhang]), and the packaging - of the published artifacts changed: `comet-common`, the parent pom and `comet-spark-integration` are no longer +- **Spark-to-Arrow conversions have their own settings.** Using + `spark.comet.sparkToColumnar.supportedOperatorList` to convert ranges, cached tables, RDDs, queries without a + `FROM` clause, Data Source V1 relations or Text scans is deprecated. Use the matching `spark.comet.convert.*` + setting instead ([#6602], [#7123]). The old setting still works and logs a warning. +- **Comet settings reach every stage.** A job launched from a query's RDD, such as through `Dataset.rdd`, ran its + map stages with every `spark.comet.*` setting at its default, including the memory pool. Those stages now use + the session's settings, so their behavior can change after upgrading ([#7302], thanks to [@comphead]). +- **Three known differences from Spark are new in 1.2.0.** The operators involved now run natively where 1.1.0 + fell back. The upgrade guide gives a setting that avoids each one: + - `collect_set` over `FLOAT` or `DOUBLE` next to `count(DISTINCT ...)` on Spark before 4.2 ([#7203]) + - `RANGE` window frames over a `DATE` column with an offset of roughly 92 million days or more ([#5022]) + - memory accounting for `min` and `max` over strings in a hash aggregate ([#6936]) +- **Native libraries are extracted into private directories** ([#6987], thanks to [@manuzhang]). +- **Fewer published artifacts.** `comet-common`, the parent pom and `comet-spark-integration` are no longer published ([#6888]). ## Compatibility From fcfd12c1fe5f178ada1095c86a39d28b965c2a9f Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Sat, 31 Oct 2026 08:58:55 -0600 Subject: [PATCH 06/26] Clarify that the direct columnar-to-row converter is opt-in --- content/blog/2026-11-03-datafusion-comet-1.2.0.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/content/blog/2026-11-03-datafusion-comet-1.2.0.md b/content/blog/2026-11-03-datafusion-comet-1.2.0.md index 567ded77..b622865b 100644 --- a/content/blog/2026-11-03-datafusion-comet-1.2.0.md +++ b/content/blog/2026-11-03-datafusion-comet-1.2.0.md @@ -295,8 +295,10 @@ run against it nightly ([#4950], [#6417]). ([#6969]). - **Spark-to-Arrow conversion**: columns are written in bulk ([#6566]), unsafe rows are converted straight from their memory ([#6730]), and JVM columnar shuffle converts strings, maps and arrays faster ([#6604]). -- **Columnar to row**: a direct path is up to 1.9x faster on decimal-heavy schemas, though the PR measured it 29% - slower on a plain long, int, double and string schema ([#5496], thanks to [@peterxcli]). +- **Columnar to row**: a new experimental converter writes Arrow values straight into Spark rows without + allocating an object per value, and is up to 1.9x faster on decimal-heavy schemas. It is opt-in + (`spark.comet.exec.columnarToRow.direct.enabled`), since the existing path is already fast for schemas without + decimals ([#5496], thanks to [@peterxcli]). - **Scans**: the next Parquet file's footer and page index load while the current file is read, which the PR reports as up to 1.9x faster on high-latency storage ([#7094]). Opt-in TopK threshold pushdown reaches the Parquet readers ([#6181], [#6263], thanks to [@pingzh]). From 68c0ff856eca9518fd20b84835009d811cba12ca Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Sat, 31 Oct 2026 09:00:30 -0600 Subject: [PATCH 07/26] Correct reason the direct columnar-to-row converter is opt-in --- content/blog/2026-11-03-datafusion-comet-1.2.0.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/blog/2026-11-03-datafusion-comet-1.2.0.md b/content/blog/2026-11-03-datafusion-comet-1.2.0.md index b622865b..015ae646 100644 --- a/content/blog/2026-11-03-datafusion-comet-1.2.0.md +++ b/content/blog/2026-11-03-datafusion-comet-1.2.0.md @@ -297,8 +297,8 @@ run against it nightly ([#4950], [#6417]). their memory ([#6730]), and JVM columnar shuffle converts strings, maps and arrays faster ([#6604]). - **Columnar to row**: a new experimental converter writes Arrow values straight into Spark rows without allocating an object per value, and is up to 1.9x faster on decimal-heavy schemas. It is opt-in - (`spark.comet.exec.columnarToRow.direct.enabled`), since the existing path is already fast for schemas without - decimals ([#5496], thanks to [@peterxcli]). + (`spark.comet.exec.columnarToRow.direct.enabled`) because the existing path is still faster for some schemas, + such as primitives mixed with strings ([#5496], thanks to [@peterxcli]). - **Scans**: the next Parquet file's footer and page index load while the current file is read, which the PR reports as up to 1.9x faster on high-latency storage ([#7094]). Opt-in TopK threshold pushdown reaches the Parquet readers ([#6181], [#6263], thanks to [@pingzh]). From 9bd2d8a06fe0b83c257290c71a893e3631d2f823 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Mon, 2 Nov 2026 11:48:14 -0700 Subject: [PATCH 08/26] Link the nested TPC-H benchmark post from the Comet 1.2.0 blog post --- content/blog/2026-11-03-datafusion-comet-1.2.0.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/content/blog/2026-11-03-datafusion-comet-1.2.0.md b/content/blog/2026-11-03-datafusion-comet-1.2.0.md index 015ae646..0571327c 100644 --- a/content/blog/2026-11-03-datafusion-comet-1.2.0.md +++ b/content/blog/2026-11-03-datafusion-comet-1.2.0.md @@ -290,6 +290,13 @@ run against it nightly ([#4950], [#6417]). ## Performance +[Nested TPC-H] is a new TPC-H derived benchmark that stores most columns in arrays of structs. At scale factor +1000 on Iceberg tables, Comet 1.2.0 is faster than Spark on all 22 queries and takes 9% less time than 1.1.0 +overall. Spill compression ([#6698]) and nested-field pruning in the native Iceberg scan ([#6725]), both described +above, account for part of that. + +[Nested TPC-H]: https://datafusion.apache.org/blog/2026/10/25/comet-nested-tpch/ + - **Casts and conditionals**: `CASE WHEN` and `IF` are up to 12x faster ([#6350]), string-to-timestamp casts up to 5.6x faster ([#5130]), and casts from integers, decimals, floats and booleans to timestamp are vectorized ([#6969]). From a482e1aca38c1d7ce789c5c7c244e118cc4f624a Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Mon, 2 Nov 2026 12:57:17 -0700 Subject: [PATCH 09/26] Update content/blog/2026-11-03-datafusion-comet-1.2.0.md Co-authored-by: Matt Butrovich --- content/blog/2026-11-03-datafusion-comet-1.2.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/blog/2026-11-03-datafusion-comet-1.2.0.md b/content/blog/2026-11-03-datafusion-comet-1.2.0.md index 0571327c..38084c22 100644 --- a/content/blog/2026-11-03-datafusion-comet-1.2.0.md +++ b/content/blog/2026-11-03-datafusion-comet-1.2.0.md @@ -32,7 +32,7 @@ The Apache DataFusion PMC is pleased to announce version 1.2.0 of the [Comet](ht Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for improved performance and efficiency without requiring any code changes. -This release covers roughly five weeks of development since 1.1.0: 400 commits from 40 contributors. See the +This release covers roughly five weeks of development since 1.1.0: 399 commits from 40 contributors. See the [change log] for the full list of changes. [change log]: https://github.com/apache/datafusion-comet/blob/branch-1.2/docs/source/changelog/1.2.0.md From 16dfe068c2de16d1e81436813461586b05c6ac8a Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Mon, 2 Nov 2026 12:57:34 -0700 Subject: [PATCH 10/26] Update content/blog/2026-11-03-datafusion-comet-1.2.0.md Co-authored-by: Matt Butrovich --- content/blog/2026-11-03-datafusion-comet-1.2.0.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/content/blog/2026-11-03-datafusion-comet-1.2.0.md b/content/blog/2026-11-03-datafusion-comet-1.2.0.md index 38084c22..17de10c2 100644 --- a/content/blog/2026-11-03-datafusion-comet-1.2.0.md +++ b/content/blog/2026-11-03-datafusion-comet-1.2.0.md @@ -43,9 +43,9 @@ and adding tests that keep them fixed. The highlights: -- **Results that match Spark**: native timestamp results carry the right timezone, `-0.0`, `0.0` and `NaN` - follow Spark in hashing, comparison, sorting, aggregation and array functions, and fixes to decimal `AVG`, - nested types, broadcasts and join order. +- **Results that match Spark**: native timestamp results carry the right timezone label. `-0.0`, `0.0` and + `NaN` follow Spark in hashing, comparison, sorting, aggregation and array functions. Decimal `AVG`, nested + types, broadcasts and join order get fixes too. - **Memory, spill and shuffle at scale**: memory returned to Spark sooner, fewer memory-related task failures, much smaller JVM shuffle memory use with many partitions, and smaller shuffle files. - **Iceberg**: native writes are a documented, opt-in feature with support for format-version 3 tables, and the From 21a2df09039c73afbdcf26a4e33c2605ea75effd Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Mon, 2 Nov 2026 12:58:03 -0700 Subject: [PATCH 11/26] Update content/blog/2026-11-03-datafusion-comet-1.2.0.md Co-authored-by: Matt Butrovich --- content/blog/2026-11-03-datafusion-comet-1.2.0.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/blog/2026-11-03-datafusion-comet-1.2.0.md b/content/blog/2026-11-03-datafusion-comet-1.2.0.md index 17de10c2..1d6c0434 100644 --- a/content/blog/2026-11-03-datafusion-comet-1.2.0.md +++ b/content/blog/2026-11-03-datafusion-comet-1.2.0.md @@ -159,8 +159,8 @@ bytes ([#5302], [#6206], [#6471], [#6480]). Thanks to [@stantheman0128], [@LinSi - **Container sizing.** The native memory limit follows how YARN and Kubernetes size executor containers ([#6375]). - **Spill files are compressed with lz4 by default.** The codec is set by - `spark.comet.exec.spill.compression.codec` ([#6698]). The PR reports TPC-H q21 going from 380 s to 250 s, - against 367 s for Spark. + `spark.comet.exec.spill.compression.codec` ([#6698]). In the investigation behind it, [Nested TPC-H] q21 at + scale factor 1000 went from 380 s to 250 s, against 367 s for Spark. - **The memory usage log is in the Spark event log.** It can be read after the application ends ([#6401], [#6250], [#6271]). From 673202ab1cae1e9f243d28d44036f457165889bc Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Mon, 2 Nov 2026 12:58:30 -0700 Subject: [PATCH 12/26] Update content/blog/2026-11-03-datafusion-comet-1.2.0.md Co-authored-by: Matt Butrovich --- content/blog/2026-11-03-datafusion-comet-1.2.0.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/content/blog/2026-11-03-datafusion-comet-1.2.0.md b/content/blog/2026-11-03-datafusion-comet-1.2.0.md index 1d6c0434..fbd17ddc 100644 --- a/content/blog/2026-11-03-datafusion-comet-1.2.0.md +++ b/content/blog/2026-11-03-datafusion-comet-1.2.0.md @@ -270,8 +270,10 @@ and are enabled with `spark.comet.exec.nativeArrowPythonUDF.enabled`. The PR rep `pyarrow.compute.negate` over 50 million rows running 3.6x faster than Spark, and a UDF that loops in Python running 3.4x slower, so try it on your own workload before relying on it. Thanks to [@viirya]. -Custom Rust UDFs through Arrow FFI ([#4459]) remain experimental. A new setting disables or restricts native -library loading ([#6775]). +Custom Rust UDFs are also new and experimental. Comet can load scalar UDFs written in Rust from a shared +library and call them through Arrow FFI ([#4459]). See the +[Rust UDF guide](https://datafusion.apache.org/comet/user-guide/latest/rust_udfs.html). A new setting disables +or restricts native library loading ([#6775]). ### Parquet writes to S3 with the magic committer From 75dcc1c25d4ccaebf9993a0fde77644f71605120 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Mon, 2 Nov 2026 12:58:57 -0700 Subject: [PATCH 13/26] Update content/blog/2026-11-03-datafusion-comet-1.2.0.md Co-authored-by: Matt Butrovich --- content/blog/2026-11-03-datafusion-comet-1.2.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/blog/2026-11-03-datafusion-comet-1.2.0.md b/content/blog/2026-11-03-datafusion-comet-1.2.0.md index fbd17ddc..429b79d0 100644 --- a/content/blog/2026-11-03-datafusion-comet-1.2.0.md +++ b/content/blog/2026-11-03-datafusion-comet-1.2.0.md @@ -329,7 +329,7 @@ A few changes in 1.2.0 can affect a deployment. The [Comet Upgrade Guide] has th different order. Set it to `1` to restore the old behavior. - **Spark-to-Arrow conversions have their own settings.** Using `spark.comet.sparkToColumnar.supportedOperatorList` to convert ranges, cached tables, RDDs, queries without a - `FROM` clause, Data Source V1 relations or Text scans is deprecated. Use the matching `spark.comet.convert.*` + `FROM` clause, Data Source V1 relations that are not file-based, or Text scans is deprecated. Use the matching `spark.comet.convert.*` setting instead ([#6602], [#7123]). The old setting still works and logs a warning. - **Comet settings reach every stage.** A job launched from a query's RDD, such as through `Dataset.rdd`, ran its map stages with every `spark.comet.*` setting at its default, including the memory pool. Those stages now use From 891d6a611353e057572cccdc6a3f583ee0a9f9fa Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Mon, 2 Nov 2026 12:59:44 -0700 Subject: [PATCH 14/26] Update content/blog/2026-11-03-datafusion-comet-1.2.0.md Co-authored-by: Matt Butrovich --- content/blog/2026-11-03-datafusion-comet-1.2.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/blog/2026-11-03-datafusion-comet-1.2.0.md b/content/blog/2026-11-03-datafusion-comet-1.2.0.md index 429b79d0..150334df 100644 --- a/content/blog/2026-11-03-datafusion-comet-1.2.0.md +++ b/content/blog/2026-11-03-datafusion-comet-1.2.0.md @@ -295,7 +295,7 @@ run against it nightly ([#4950], [#6417]). [Nested TPC-H] is a new TPC-H derived benchmark that stores most columns in arrays of structs. At scale factor 1000 on Iceberg tables, Comet 1.2.0 is faster than Spark on all 22 queries and takes 9% less time than 1.1.0 overall. Spill compression ([#6698]) and nested-field pruning in the native Iceberg scan ([#6725]), both described -above, account for part of that. +above, came out of the work on that benchmark. [Nested TPC-H]: https://datafusion.apache.org/blog/2026/10/25/comet-nested-tpch/ From 4c00c9d1c2eaf694175a962232d389ee72e684ae Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Mon, 2 Nov 2026 13:00:14 -0700 Subject: [PATCH 15/26] Update content/blog/2026-11-03-datafusion-comet-1.2.0.md Co-authored-by: Matt Butrovich --- content/blog/2026-11-03-datafusion-comet-1.2.0.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/content/blog/2026-11-03-datafusion-comet-1.2.0.md b/content/blog/2026-11-03-datafusion-comet-1.2.0.md index 150334df..579f825d 100644 --- a/content/blog/2026-11-03-datafusion-comet-1.2.0.md +++ b/content/blog/2026-11-03-datafusion-comet-1.2.0.md @@ -264,9 +264,11 @@ deletion vectors, row-group pruning and page pruning ([#5365]). It is opt-in: bu ### Python UDFs (experimental) -Scalar Arrow Python UDFs can run in native Comet, passing Arrow data to Python without going through rows -([#6130]). They need a native library built with the `python-udf` Cargo feature and PyArrow on every executor, -and are enabled with `spark.comet.exec.nativeArrowPythonUDF.enabled`. The PR reports a vectorized +On Spark 4.1 and later, scalar `@arrow_udf` functions can run in native Comet, passing Arrow data to Python +without going through rows ([#6130]). The released jars do not include this. It needs a native library built +from source with the `python-udf` Cargo feature, PyArrow on every executor, and +`spark.comet.exec.nativeArrowPythonUDF.enabled`. See the +[PyArrow UDF guide](https://datafusion.apache.org/comet/user-guide/latest/pyarrow-udfs.html). The PR reports a vectorized `pyarrow.compute.negate` over 50 million rows running 3.6x faster than Spark, and a UDF that loops in Python running 3.4x slower, so try it on your own workload before relying on it. Thanks to [@viirya]. From 106c7390b161459e94c6c70c4f393224089167ca Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Mon, 2 Nov 2026 13:00:55 -0700 Subject: [PATCH 16/26] Update content/blog/2026-11-03-datafusion-comet-1.2.0.md Co-authored-by: Matt Butrovich --- content/blog/2026-11-03-datafusion-comet-1.2.0.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/content/blog/2026-11-03-datafusion-comet-1.2.0.md b/content/blog/2026-11-03-datafusion-comet-1.2.0.md index 579f825d..194a9a90 100644 --- a/content/blog/2026-11-03-datafusion-comet-1.2.0.md +++ b/content/blog/2026-11-03-datafusion-comet-1.2.0.md @@ -93,8 +93,9 @@ choices. The differences are easy to miss in testing and show up as wrong joins, - **Array functions**: `array_remove`, `sort_array`, `array_distinct` and `array_union` ([#6518], [#6563]), and `arrays_overlap` on the Spark versions where `-0.0` and `0.0` are one value ([#6989]). - **Arithmetic**: `signum(-0.0)` ([#6556]) and the zero check in `Remainder` ([#6896]). -- **Maps**: Comet no longer builds a map natively when its key type holds a `FLOAT` or `DOUBLE`, because Spark 4.0+ - normalizes those keys and rejects duplicates in a way native code did not ([#6992]). +- **Maps**: Comet no longer builds a map natively when its key type holds a `FLOAT` or `DOUBLE` ([#6992]). + Spark finds duplicate keys by value, so `NaN`s with different bits are one key, and from 4.0 it also + normalizes `-0.0` keys to `0.0`. Native code told keys apart by their bits. The contributor guide now has a [floating point page] that says which Spark rule each expression follows and how to test for it. A sweep suite crosses edge values with every operator context, so a DataFusion upgrade that From d2f4146be2adb73b346f5ad327e8b6a595ace044 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Mon, 2 Nov 2026 13:19:43 -0700 Subject: [PATCH 17/26] Address review feedback on Comet 1.2.0 blog post --- .../blog/2026-11-03-datafusion-comet-1.2.0.md | 98 ++++++++++--------- 1 file changed, 53 insertions(+), 45 deletions(-) diff --git a/content/blog/2026-11-03-datafusion-comet-1.2.0.md b/content/blog/2026-11-03-datafusion-comet-1.2.0.md index 194a9a90..17778687 100644 --- a/content/blog/2026-11-03-datafusion-comet-1.2.0.md +++ b/content/blog/2026-11-03-datafusion-comet-1.2.0.md @@ -43,14 +43,14 @@ and adding tests that keep them fixed. The highlights: -- **Results that match Spark**: native timestamp results carry the right timezone label. `-0.0`, `0.0` and - `NaN` follow Spark in hashing, comparison, sorting, aggregation and array functions. Decimal `AVG`, nested - types, broadcasts and join order get fixes too. -- **Memory, spill and shuffle at scale**: memory returned to Spark sooner, fewer memory-related task failures, +- **Results that match Spark**: native timestamp results carry the right timezone label. `-0.0`, `0.0`, and + `NaN` follow Spark in hashing, comparison, sorting, aggregation, and array functions. Decimal `AVG`, nested + types, broadcasts, and join order get fixes too. +- **Memory, spill, and shuffle at scale**: memory returned to Spark sooner, fewer memory-related task failures, much smaller JVM shuffle memory use with many partitions, and smaller shuffle files. - **Iceberg**: native writes are a documented, opt-in feature with support for format-version 3 tables, and the native scan reads fewer nested fields and more files at once. -- **More of the plan runs natively**: existence joins, `SortAggregateExec`, the `PIVOT` fast path and +- **More of the plan runs natively**: existence joins, `SortAggregateExec`, the `PIVOT` fast path, and HyperLogLog sketch functions, among others. - **New places to run**: a native Delta Lake scan (experimental), Arrow Python UDFs (experimental), and Spark 4.2 as a supported version. @@ -80,17 +80,17 @@ Thanks to [@andygrove] for the timezone sweep and [@LinSimon-901101] for the lab [timezone handling page]: https://datafusion.apache.org/comet/contributor-guide/timezones.html -### Floating point: `-0.0`, `0.0` and `NaN` +### Floating point: `-0.0`, `0.0`, and `NaN` In some operations Spark treats `-0.0` and `0.0` as equal and every `NaN` as one value; in others it treats them as distinct. Which rule applies depends on the operation and the Spark version, and DataFusion makes different -choices. The differences are easy to miss in testing and show up as wrong joins, groups and sort orders. In +choices. The differences are easy to miss in testing and show up as wrong joins, groups, and sort orders. In 1.2.0, these follow Spark: -- **Hashing**: `hash`, `xxhash64` and native shuffle hash `NaN` as Spark does ([#6413]). -- **Comparison and ordering**: `min`, `max`, `greatest` and `least` ([#6457]), sort and window keys nested in +- **Hashing**: `hash`, `xxhash64`, and native shuffle hash `NaN` as Spark does ([#6413]). +- **Comparison and ordering**: `min`, `max`, `greatest`, and `least` ([#6457]), sort and window keys nested in arrays and structs ([#6475]), and `percentile` for negative `NaN`s ([#6829]). -- **Array functions**: `array_remove`, `sort_array`, `array_distinct` and `array_union` ([#6518], [#6563]), and +- **Array functions**: `array_remove`, `sort_array`, `array_distinct`, and `array_union` ([#6518], [#6563]), and `arrays_overlap` on the Spark versions where `-0.0` and `0.0` are one value ([#6989]). - **Arithmetic**: `signum(-0.0)` ([#6556]) and the zero check in `Remainder` ([#6896]). - **Maps**: Comet no longer builds a map natively when its key type holds a `FLOAT` or `DOUBLE` ([#6992]). @@ -101,15 +101,15 @@ The contributor guide now has a [floating point page] that says which Spark rule to test for it. A sweep suite crosses edge values with every operator context, so a DataFusion upgrade that shifts float behavior will fail a test before it reaches users. -Thanks to [@andygrove] for the float sweep, [@rich7420] for `percentile` and [@zhangfengcdt] for `signum`. +Thanks to [@andygrove] for the float sweep, [@rich7420] for `percentile`, and [@zhangfengcdt] for `signum`. [floating point page]: https://datafusion.apache.org/comet/contributor-guide/floating_point.html ### Collations -Strings with a non-default collation compare differently from ordinary strings. For collated casts, sort keys and +Strings with a non-default collation compare differently from ordinary strings. For collated casts, sort keys, and comparisons, Comet now falls back to Spark or runs the expression through [codegen dispatch] instead of comparing -bytes ([#5302], [#6206], [#6471], [#6480]). Thanks to [@stantheman0128], [@LinSimon-901101] and [@viirya]. +bytes ([#5302], [#6206], [#6471], [#6480]). Thanks to [@stantheman0128], [@LinSimon-901101], and [@viirya]. [codegen dispatch]: https://datafusion.apache.org/blog/2026/06/20/datafusion-comet-0.17.0/ @@ -118,33 +118,33 @@ bytes ([#5302], [#6206], [#6471], [#6480]). Thanks to [@stantheman0128], [@LinSi - **Decimal `AVG`**: results were `NULL` or wrong for sums that overflowed an intermediate `i128`, and empty and overflowing partial states were mixed up. 1.2.0 repairs the division and the buffers ([#5420], thanks to [@sunchao]) and falls back to Spark for the cases it cannot yet do correctly ([#7017]). -- **Variance, covariance and correlation**: these now update and merge in Spark's order, so values that are large +- **Variance, covariance, and correlation**: these now update and merge in Spark's order, so values that are large and close together give Spark's result, and `regr_r2` returns `NULL` or `DIVIDE_BY_ZERO` where Spark does ([#6076], thanks to [@rich7420]). - **Nested values in comparisons and sorts**: `greatest` and `least` compare struct arguments by position and order nulls inside nested values first ([#6760], [#6761], thanks to [@viirya]), and native sort keeps Spark's null ordering inside nested values ([#6968], thanks to [@Weijun-H]). - **Missing nested Parquet fields**: when every nested field a query asks for is missing from a Parquet file, - Comet could return a non-null struct, array or map where Spark returns `NULL`, which changed `IS NULL`, filters + Comet could return a non-null struct, array, or map where Spark returns `NULL`, which changed `IS NULL`, filters, and `count`. Comet now follows Spark's version-dependent rules, and falls back for nested reads of name-mapped Iceberg tables ([#7085], thanks to [@rich7420]). - **Nested binary read as strings**: a nested Parquet `BINARY` value read as a string that held invalid UTF-8 became `NULL`. It is now decoded the way Comet's `CAST(binary AS STRING)` decodes it, replacing ill-formed bytes with U+FFFD ([#7106], thanks to [@manuzhang]). -- **Sliced broadcasts**: broadcasts of sliced string, binary, array or map batches could return wrong values or +- **Sliced broadcasts**: broadcasts of sliced string, binary, array, or map batches could return wrong values or fail with `OversizedAllocationException` ([#6900], thanks to [@comphead]). - **Identity broadcasts**: a Spark operator that keeps every row of a Comet broadcast could see the last row of each batch repeated in place of the others. Comet now copies each row ([#7260], thanks to [@pingzh]). - **Join output order**: Spark drops a sort above a hash join because the join keeps its streamed side's order. Comet's native hash join did not keep it for some inputs, so the query could return unsorted partitions ([#6835], [#6810], thanks to [@dwsmith1983]). -- **Spark patch releases**: Spark 4.0.5, 4.1.4 and 4.2.1 change what `sequence` returns when `stop - start` +- **Spark patch releases**: Spark 4.0.5, 4.1.4, and 4.2.1 change what `sequence` returns when `stop - start` overflows a long, and the result of casting a zero with a large exponent, such as `'0e79'`, to decimal. Comet follows the behavior of the Spark version it runs on ([#7171], thanks to [@snmvaughan]). - **ANSI mode**: `array_append` now evaluates its item on rows whose array is `NULL`, so it raises the errors Spark raises, and `corr` raises `DIVIDE_BY_ZERO` ([#7025], [#6830], thanks to [@kazuyukitanimura]). -## Memory, Spill and Shuffle +## Memory, Spill, and Shuffle 1.1.0 made Comet's memory use visible. 1.2.0 fixes more of the ways it went wrong, and makes shuffle cheaper. @@ -173,8 +173,9 @@ Thanks to [@andygrove] for most of this work, with [@dwsmith1983] and [@viirya] - **JVM shuffle with many partitions.** The bypass-merge writer gave every partition a page of Spark's page size, 32 to 64 MB, as soon as it buffered a row. When a task's memory could not hold one page per partition, every new page forced a spill. Pages now start small and grow ([#7059]). The PR reports `REPARTITION(64)` on TPC-H - `orders` going from 266 s and 2.48 GB to 1.1 s and 156 MB, and on `lineitem` from 15.5 minutes to 3.3 s. -- **Smaller shuffle files.** Shuffle files with string, binary, array and map columns are 9% to 40% smaller in + SF2 `orders` going from 266 s and 2.48 GB of shuffle output to 1.1 s and 156 MB, and on `lineitem` from + 15.5 minutes to 3.3 s. +- **Smaller shuffle files.** Shuffle files with string, binary, array, and map columns are 9% to 40% smaller in the PR's benchmarks, because offsets are now stored as lengths and compressed with zstd ([#6943]). - **Faster spills.** A JVM shuffle spill writes every partition in one native call ([#7065]). The PR reports this as 21% to 28% faster with 1,000 partitions, and 45% to 49% faster when the writer spills every 100,000 rows. @@ -184,7 +185,7 @@ Thanks to [@andygrove] for most of this work, with [@dwsmith1983] and [@viirya] executor now share one decoded copy ([#7016]). In the PR's benchmark, a 1M-row build side read by 128 tasks on `local[8]` went from 1,321 ms to 684 ms. -Thanks to [@viirya] for the spill pointer sort, [@comphead] for the block reads and [@andygrove] for the rest. +Thanks to [@viirya] for the spill pointer sort, [@comphead] for the block reads, and [@andygrove] for the rest. ## Iceberg @@ -225,41 +226,45 @@ The [Iceberg guide] lists the supported operations, every fallback, and every kn write transactions ([#6587]). Thanks to [@andygrove] for the writer work, [@snmvaughan] for per-location credentials, [@unikdahal] for the -row-level operations, [@comphead] for nested-field pruning, and [@sam-1112], [@0lai0] and [@parthchandra] for -fallback gates. +row-level operations, [@mbutrovich] for float and double identity partitions, [@comphead] for nested-field +pruning, and [@sam-1112], [@0lai0], and [@parthchandra] for fallback gates. ## More of the Plan Runs Natively - **On by default**: existence joins ([#4587], thanks to [@coderfender]), queries without a `FROM` clause (`OneRowRelation`, [#6666]), aggregates with neither grouping keys nor aggregate functions ([#6899]), and `min` - and `max` over strings and binary ([#6927]). Broadcast joins whose build side holds an unsupported scan now run - natively too ([#6786], thanks to [@parthchandra]), except that a Text scan on the build side stays in Spark - unless `spark.comet.convert.text.enabled` is set, because converting it replaces bytes that are not valid UTF-8 - with U+FFFD ([#7123]). + and `max` over strings and binary ([#6927], thanks to [@comphead]). Broadcast joins whose build side holds an + unsupported scan now run natively too ([#6786], thanks to [@parthchandra]), except that a Text scan on the build + side stays in Spark unless `spark.comet.convert.text.enabled` is set, because converting it replaces bytes that + are not valid UTF-8 with U+FFFD ([#7123]). - **New operators and expressions**: `SortAggregateExec` ([#4565]), the `PivotFirst` fast path for `PIVOT` ([#4798]), HyperLogLog sketch functions ([#4802]), distinct-combined `collect_list` ([#4727]), `AtLeastNNonNulls` ([#6180]), `replace` ([#6680]), hashing of wide decimals ([#6548]), `RANGE` window frames - with an offset on a `DATE` key ([#4974]), and interval multiply and divide ([#4715], [#4901]). + with an offset on a `DATE` key ([#4974]), and interval multiply and divide ([#4715], [#4901]). Thanks to + [@peterxcli] for `collect_list` and the interval operations, [@rich7420] for `AtLeastNNonNulls` and wide + decimal hashing, [@sam-1112] for `replace`, and [@0lai0] for `RANGE` frames. - **Opt-in conversions**: a native `RangeExec` ([#6553]), operators above typed Dataset operations ([#6564]), - native shuffle over row input ([#6607]), and join filter pushdown through deterministic residuals ([#6431]). + native shuffle over row input ([#6607]), and join filter pushdown through deterministic residuals ([#6431], + thanks to [@sunchao]). ### The in-memory cache stays opt-in Comet can store Spark's in-memory cache in Arrow format and read it natively. We planned to turn this on by default in 1.2.0 and hardened it first: serializer and Kryo gating ([#6360], [#6537]), EXPLAIN and SQL-tab -fidelity ([#6453], [#6574], [#6577]), planner statistics ([#6412]) and a reader class generated once per executor +fidelity ([#6453], [#6574], [#6577]), planner statistics ([#6412]), and a reader class generated once per executor ([#6860]). Auditing the feature turned up cases where default settings could return rows in the wrong order, so it stays opt-in (`spark.comet.exec.inMemoryCache.enabled`) for 1.2.0. One of those cases, the join output order -fix above, is in this release ([#6835]); the rest are tracked in [#6834]. +fix above, is in this release ([#6835]); the rest are tracked in [#6834]. Thanks to [@comphead] for the EXPLAIN +and SQL-tab work and [@andygrove] for the rest. ## Reaching New Places ### Delta Lake (experimental) A new contrib module plans Delta Lake scans on the JVM and runs them in Comet's native Parquet scan, including -deletion vectors, row-group pruning and page pruning ([#5365]). It is opt-in: build the separate -`comet-contrib-delta` jar and set `spark.comet.scan.delta.enabled=true`. See the [Delta Lake guide]. Thanks to -[@dwsmith1983]. +deletion vectors, row-group pruning, and page pruning ([#5365]). It is opt-in: build the +`comet-contrib-delta-spark` jar from source, since no release publishes it yet, and set +`spark.comet.scan.delta.enabled=true`. See the [Delta Lake guide]. Thanks to [@dwsmith1983] and [@sunchao]. [Delta Lake guide]: https://datafusion.apache.org/comet/user-guide/latest/delta.html @@ -282,7 +287,7 @@ or restricts native library loading ([#6775]). The native Parquet writer can now write to S3 through the S3A magic committer ([#6940]), which Spark 4.1 configures by default when spark-hadoop-cloud is on the classpath. S3A creates each file, so its write settings, -such as server-side encryption, ACLs and storage class, apply as they do for Spark. Other S3 writes fall back to +such as server-side encryption, ACLs, and storage class, apply as they do for Spark. Other S3 writes fall back to Spark. The native Parquet writer is still for testing only and is off by default (`spark.comet.write.parquet.enabled`). See the [Parquet guide]. Thanks to [@andygrove]. @@ -303,10 +308,10 @@ above, came out of the work on that benchmark. [Nested TPC-H]: https://datafusion.apache.org/blog/2026/10/25/comet-nested-tpch/ - **Casts and conditionals**: `CASE WHEN` and `IF` are up to 12x faster ([#6350]), string-to-timestamp casts up - to 5.6x faster ([#5130]), and casts from integers, decimals, floats and booleans to timestamp are vectorized - ([#6969]). + to 5.6x faster ([#5130]), and casts from integers, decimals, floats, and booleans to timestamp are vectorized + ([#6969], thanks to [@Ersa-tech]). - **Spark-to-Arrow conversion**: columns are written in bulk ([#6566]), unsafe rows are converted straight from - their memory ([#6730]), and JVM columnar shuffle converts strings, maps and arrays faster ([#6604]). + their memory ([#6730]), and JVM columnar shuffle converts strings, maps, and arrays faster ([#6604]). - **Columnar to row**: a new experimental converter writes Arrow values straight into Spark rows without allocating an object per value, and is up to 1.9x faster on decimal-heavy schemas. It is opt-in (`spark.comet.exec.columnarToRow.direct.enabled`) because the existing path is still faster for some schemas, @@ -325,15 +330,16 @@ A few changes in 1.2.0 can affect a deployment. The [Comet Upgrade Guide] has th - **Native Iceberg writes use one setting.** In 1.1.0, native Iceberg writes were a testing feature that needed both `spark.comet.iceberg.write.enabled` and `spark.comet.write.iceberg.splitOperator.enabled`. The setting is now `spark.comet.write.iceberg.enabled`, and Comet ignores the old name. Parquet writes were renamed the same - way, from `spark.comet.parquet.write.enabled` to `spark.comet.write.parquet.enabled` ([#6494]). Set the new - names to keep native writes. + way, from `spark.comet.parquet.write.enabled` to `spark.comet.write.parquet.enabled` ([#6494], thanks to + [@zhangfengcdt]). Set the new names to keep native writes. `spark.comet.write.iceberg.splitOperator.enabled` + is no longer needed, because the new setting plans the split on its own. Remove it if you set it for 1.1.0. - **The native Iceberg scan reads two files at once.** `spark.comet.scan.icebergNative.dataFileConcurrencyLimit` now defaults to 2. A task can use more off-heap memory, and a query without `ORDER BY` can return rows in a different order. Set it to `1` to restore the old behavior. -- **Spark-to-Arrow conversions have their own settings.** Using - `spark.comet.sparkToColumnar.supportedOperatorList` to convert ranges, cached tables, RDDs, queries without a - `FROM` clause, Data Source V1 relations that are not file-based, or Text scans is deprecated. Use the matching `spark.comet.convert.*` - setting instead ([#6602], [#7123]). The old setting still works and logs a warning. +- **Spark-to-Arrow conversions have their own settings.** Using `spark.comet.sparkToColumnar.supportedOperatorList` + to convert ranges, cached tables, RDDs, queries without a `FROM` clause, Data Source V1 relations that are not + file-based, or Text scans is deprecated. Use the matching `spark.comet.convert.*` setting instead ([#6602], + [#7123]). The old setting still works and logs a warning. - **Comet settings reach every stage.** A job launched from a query's RDD, such as through `Dataset.rdd`, ran its map stages with every `spark.comet.*` setting at its default, including the memory pool. Those stages now use the session's settings, so their behavior can change after upgrading ([#7302], thanks to [@comphead]). @@ -343,7 +349,7 @@ A few changes in 1.2.0 can affect a deployment. The [Comet Upgrade Guide] has th - `RANGE` window frames over a `DATE` column with an offset of roughly 92 million days or more ([#5022]) - memory accounting for `min` and `max` over strings in a hash aggregate ([#6936]) - **Native libraries are extracted into private directories** ([#6987], thanks to [@manuzhang]). -- **Fewer published artifacts.** `comet-common`, the parent pom and `comet-spark-integration` are no longer +- **Fewer published artifacts.** `comet-common`, the parent pom, and `comet-spark-integration` are no longer published ([#6888]). ## Compatibility @@ -385,6 +391,8 @@ to get up and running, then point Comet at your existing Spark workloads. [@parthchandra]: https://github.com/parthchandra [@coderfender]: https://github.com/coderfender [@peterxcli]: https://github.com/peterxcli +[@Ersa-tech]: https://github.com/Ersa-tech +[@mbutrovich]: https://github.com/mbutrovich [@pingzh]: https://github.com/pingzh [@manuzhang]: https://github.com/manuzhang From ea3e93c215eadeb0f04eb0db3b58acb3786eeee3 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Mon, 2 Nov 2026 13:29:46 -0700 Subject: [PATCH 18/26] Credit co-authors of the Delta scan and ANSI corr fix --- content/blog/2026-11-03-datafusion-comet-1.2.0.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/content/blog/2026-11-03-datafusion-comet-1.2.0.md b/content/blog/2026-11-03-datafusion-comet-1.2.0.md index 17778687..9e9053f6 100644 --- a/content/blog/2026-11-03-datafusion-comet-1.2.0.md +++ b/content/blog/2026-11-03-datafusion-comet-1.2.0.md @@ -142,7 +142,8 @@ bytes ([#5302], [#6206], [#6471], [#6480]). Thanks to [@stantheman0128], [@LinSi overflows a long, and the result of casting a zero with a large exponent, such as `'0e79'`, to decimal. Comet follows the behavior of the Spark version it runs on ([#7171], thanks to [@snmvaughan]). - **ANSI mode**: `array_append` now evaluates its item on rows whose array is `NULL`, so it raises the errors - Spark raises, and `corr` raises `DIVIDE_BY_ZERO` ([#7025], [#6830], thanks to [@kazuyukitanimura]). + Spark raises, and `corr` raises `DIVIDE_BY_ZERO` ([#7025], [#6830], thanks to [@kazuyukitanimura], with + [@rich7420] on `corr`). ## Memory, Spill, and Shuffle @@ -264,7 +265,8 @@ and SQL-tab work and [@andygrove] for the rest. A new contrib module plans Delta Lake scans on the JVM and runs them in Comet's native Parquet scan, including deletion vectors, row-group pruning, and page pruning ([#5365]). It is opt-in: build the `comet-contrib-delta-spark` jar from source, since no release publishes it yet, and set -`spark.comet.scan.delta.enabled=true`. See the [Delta Lake guide]. Thanks to [@dwsmith1983] and [@sunchao]. +`spark.comet.scan.delta.enabled=true`. See the [Delta Lake guide]. Thanks to [@dwsmith1983], with [@schenksj], +[@adivaish], and [@sunchao] as co-authors. [Delta Lake guide]: https://datafusion.apache.org/comet/user-guide/latest/delta.html @@ -381,6 +383,8 @@ to get up and running, then point Comet at your existing Spark workloads. [@Weijun-H]: https://github.com/Weijun-H [@stantheman0128]: https://github.com/stantheman0128 [@sunchao]: https://github.com/sunchao +[@schenksj]: https://github.com/schenksj +[@adivaish]: https://github.com/adivaish [@comphead]: https://github.com/comphead [@dwsmith1983]: https://github.com/dwsmith1983 [@kazuyukitanimura]: https://github.com/kazuyukitanimura From 10e3267c1c139fe59afbde29d2d361ffc7722f6e Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Mon, 2 Nov 2026 13:48:15 -0700 Subject: [PATCH 19/26] Apply readability suggestions and fix Aditya Vaish's handle --- content/blog/2026-11-03-datafusion-comet-1.2.0.md | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/content/blog/2026-11-03-datafusion-comet-1.2.0.md b/content/blog/2026-11-03-datafusion-comet-1.2.0.md index 9e9053f6..cedd588d 100644 --- a/content/blog/2026-11-03-datafusion-comet-1.2.0.md +++ b/content/blog/2026-11-03-datafusion-comet-1.2.0.md @@ -192,9 +192,8 @@ Thanks to [@viirya] for the spill pointer sort, [@comphead] for the block reads, ### Native writes -Native Iceberg writes stay **off by default** in 1.2.0, as they were in 1.1.0 when they were experimental. We -considered turning them on, but this is their first release as a documented, user-facing feature, and we would -rather users opt in and report what they find first. We plan to revisit the default in a later release. +Native Iceberg writes stay **off by default** in 1.2.0. This is their first release as a documented feature, and +we want users to opt in and report what they find before we revisit the default. To use them, set `spark.comet.write.iceberg.enabled=true`. This replaces the pair of settings used in 1.1.0, so see [Upgrading to 1.2.0](#upgrading-to-120) if you enabled the feature then. What changed in the writer: @@ -235,9 +234,9 @@ pruning, and [@sam-1112], [@0lai0], and [@parthchandra] for fallback gates. - **On by default**: existence joins ([#4587], thanks to [@coderfender]), queries without a `FROM` clause (`OneRowRelation`, [#6666]), aggregates with neither grouping keys nor aggregate functions ([#6899]), and `min` and `max` over strings and binary ([#6927], thanks to [@comphead]). Broadcast joins whose build side holds an - unsupported scan now run natively too ([#6786], thanks to [@parthchandra]), except that a Text scan on the build - side stays in Spark unless `spark.comet.convert.text.enabled` is set, because converting it replaces bytes that - are not valid UTF-8 with U+FFFD ([#7123]). + unsupported scan now run natively too ([#6786], thanks to [@parthchandra]). A Text scan on the build side + stays in Spark unless `spark.comet.convert.text.enabled` is set, because converting it replaces invalid UTF-8 + with U+FFFD ([#7123]). - **New operators and expressions**: `SortAggregateExec` ([#4565]), the `PivotFirst` fast path for `PIVOT` ([#4798]), HyperLogLog sketch functions ([#4802]), distinct-combined `collect_list` ([#4727]), `AtLeastNNonNulls` ([#6180]), `replace` ([#6680]), hashing of wide decimals ([#6548]), `RANGE` window frames @@ -266,7 +265,7 @@ A new contrib module plans Delta Lake scans on the JVM and runs them in Comet's deletion vectors, row-group pruning, and page pruning ([#5365]). It is opt-in: build the `comet-contrib-delta-spark` jar from source, since no release publishes it yet, and set `spark.comet.scan.delta.enabled=true`. See the [Delta Lake guide]. Thanks to [@dwsmith1983], with [@schenksj], -[@adivaish], and [@sunchao] as co-authors. +[@adityavaish], and [@sunchao] as co-authors. [Delta Lake guide]: https://datafusion.apache.org/comet/user-guide/latest/delta.html @@ -384,7 +383,7 @@ to get up and running, then point Comet at your existing Spark workloads. [@stantheman0128]: https://github.com/stantheman0128 [@sunchao]: https://github.com/sunchao [@schenksj]: https://github.com/schenksj -[@adivaish]: https://github.com/adivaish +[@adityavaish]: https://github.com/adityavaish [@comphead]: https://github.com/comphead [@dwsmith1983]: https://github.com/dwsmith1983 [@kazuyukitanimura]: https://github.com/kazuyukitanimura From f15be8dbf4887bb18ef81adf215fc97e7955d743 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Mon, 2 Nov 2026 13:57:55 -0700 Subject: [PATCH 20/26] Apply remaining readability suggestions from review --- .../blog/2026-11-03-datafusion-comet-1.2.0.md | 57 +++++++++---------- 1 file changed, 26 insertions(+), 31 deletions(-) diff --git a/content/blog/2026-11-03-datafusion-comet-1.2.0.md b/content/blog/2026-11-03-datafusion-comet-1.2.0.md index cedd588d..38de9fa7 100644 --- a/content/blog/2026-11-03-datafusion-comet-1.2.0.md +++ b/content/blog/2026-11-03-datafusion-comet-1.2.0.md @@ -33,7 +33,8 @@ Comet is an accelerator for Apache Spark that translates Spark physical plans to improved performance and efficiency without requiring any code changes. This release covers roughly five weeks of development since 1.1.0: 399 commits from 40 contributors. See the -[change log] for the full list of changes. +[change log] for the full list of changes. Unless stated otherwise, performance figures come from the +benchmarks in the linked PRs. [change log]: https://github.com/apache/datafusion-comet/blob/branch-1.2/docs/source/changelog/1.2.0.md @@ -43,9 +44,8 @@ and adding tests that keep them fixed. The highlights: -- **Results that match Spark**: native timestamp results carry the right timezone label. `-0.0`, `0.0`, and - `NaN` follow Spark in hashing, comparison, sorting, aggregation, and array functions. Decimal `AVG`, nested - types, broadcasts, and join order get fixes too. +- **Results that match Spark**: timezone labels, `-0.0` and `NaN`, collations, decimal `AVG`, nested types, + broadcasts, and join order. - **Memory, spill, and shuffle at scale**: memory returned to Spark sooner, fewer memory-related task failures, much smaller JVM shuffle memory use with many partitions, and smaller shuffle files. - **Iceberg**: native writes are a documented, opt-in feature with support for format-version 3 tables, and the @@ -57,17 +57,13 @@ The highlights: ## Results That Match Spark -Timestamps and floating point produced the most reports, so we started there, followed by collations. Individual -fixes in other areas come after that. - ### Timestamps and timezones Spark labels a timestamp with the session timezone, and Comet's native code has to produce the same label, not only the same instant. Several native expressions got this wrong. 1.2.0 fixes them: -- Native timestamp results carry the correct timezone label: `timestamp_seconds` results are labeled as UTC - timestamps ([#6341]), `date_trunc` keeps its input's label ([#6345]), and `CASE` and `COALESCE` relabel - timestamp branches instead of panicking ([#6347]). +- `timestamp_seconds` results are labeled as UTC timestamps ([#6341]), `date_trunc` keeps its input's label + ([#6345]), and `CASE` and `COALESCE` relabel timestamp branches instead of panicking ([#6347]). - Session timezone IDs are normalized before they reach native code ([#6351]). - `date_trunc` with a format that differs from row to row uses the same kernel as a literal format ([#6354]). - The driver warns when the native and JVM timezone databases differ ([#6353]), because Comet cannot correct @@ -82,10 +78,10 @@ Thanks to [@andygrove] for the timezone sweep and [@LinSimon-901101] for the lab ### Floating point: `-0.0`, `0.0`, and `NaN` -In some operations Spark treats `-0.0` and `0.0` as equal and every `NaN` as one value; in others it treats them -as distinct. Which rule applies depends on the operation and the Spark version, and DataFusion makes different -choices. The differences are easy to miss in testing and show up as wrong joins, groups, and sort orders. In -1.2.0, these follow Spark: +Spark has two rules for these values. Some operations treat `-0.0` and `0.0` as equal and every `NaN` as one value, +and others keep them distinct. Which rule an operation follows can change between Spark versions, and DataFusion +makes its own choices. The differences show up as wrong joins, groups, and sort orders. In 1.2.0, these follow +Spark: - **Hashing**: `hash`, `xxhash64`, and native shuffle hash `NaN` as Spark does ([#6413]). - **Comparison and ordering**: `min`, `max`, `greatest`, and `least` ([#6457]), sort and window keys nested in @@ -171,20 +167,19 @@ Thanks to [@andygrove] for most of this work, with [@dwsmith1983] and [@viirya] ### Shuffle and broadcast -- **JVM shuffle with many partitions.** The bypass-merge writer gave every partition a page of Spark's page size, - 32 to 64 MB, as soon as it buffered a row. When a task's memory could not hold one page per partition, every - new page forced a spill. Pages now start small and grow ([#7059]). The PR reports `REPARTITION(64)` on TPC-H - SF2 `orders` going from 266 s and 2.48 GB of shuffle output to 1.1 s and 156 MB, and on `lineitem` from - 15.5 minutes to 3.3 s. -- **Smaller shuffle files.** Shuffle files with string, binary, array, and map columns are 9% to 40% smaller in - the PR's benchmarks, because offsets are now stored as lengths and compressed with zstd ([#6943]). -- **Faster spills.** A JVM shuffle spill writes every partition in one native call ([#7065]). The PR reports this - as 21% to 28% faster with 1,000 partitions, and 45% to 49% faster when the writer spills every 100,000 rows. - Spill pointers are now sorted in the sorter's scratch space ([#7125]). +- **JVM shuffle with many partitions.** The bypass-merge writer gave every partition a page of Spark's page size, 32 + to 64 MB, as soon as it buffered a row. When a task's memory could not hold one page per partition, every new page + forced a spill. Pages now start small and grow ([#7059]). `REPARTITION(64)` on TPC-H SF2 `orders` went from 266 s + and 2.48 GB of shuffle output to 1.1 s and 156 MB, and on `lineitem` from 15.5 minutes to 3.3 s. +- **Smaller shuffle files.** Shuffle files with string, binary, array, and map columns are 9% to 40% smaller, + because offsets are now stored as lengths and compressed with zstd ([#6943]). +- **Faster spills.** A JVM shuffle spill writes every partition in one native call ([#7065]), which is 21% to 28% + faster with 1,000 partitions, and 45% to 49% faster when the writer spills every 100,000 rows. Spill pointers are + now sorted in the sorter's scratch space ([#7125]). - **Shuffle reads.** Shuffle blocks are read in 64 KiB pieces ([#6805]). - **Broadcast.** Every task that read a Comet broadcast decoded the whole broadcast for itself. Tasks on an - executor now share one decoded copy ([#7016]). In the PR's benchmark, a 1M-row build side read by 128 tasks on - `local[8]` went from 1,321 ms to 684 ms. + executor now share one decoded copy ([#7016]). A 1M-row build side read by 128 tasks on `local[8]` went from + 1,321 ms to 684 ms. Thanks to [@viirya] for the spill pointer sort, [@comphead] for the block reads, and [@andygrove] for the rest. @@ -275,9 +270,9 @@ On Spark 4.1 and later, scalar `@arrow_udf` functions can run in native Comet, p without going through rows ([#6130]). The released jars do not include this. It needs a native library built from source with the `python-udf` Cargo feature, PyArrow on every executor, and `spark.comet.exec.nativeArrowPythonUDF.enabled`. See the -[PyArrow UDF guide](https://datafusion.apache.org/comet/user-guide/latest/pyarrow-udfs.html). The PR reports a vectorized -`pyarrow.compute.negate` over 50 million rows running 3.6x faster than Spark, and a UDF that loops in Python -running 3.4x slower, so try it on your own workload before relying on it. Thanks to [@viirya]. +[PyArrow UDF guide](https://datafusion.apache.org/comet/user-guide/latest/pyarrow-udfs.html). A vectorized +`pyarrow.compute.negate` over 50 million rows runs 3.6x faster than Spark, and a UDF that loops in Python +runs 3.4x slower, so try it on your own workload before relying on it. Thanks to [@viirya]. Custom Rust UDFs are also new and experimental. Comet can load scalar UDFs written in Rust from a shared library and call them through Arrow FFI ([#4459]). See the @@ -317,8 +312,8 @@ above, came out of the work on that benchmark. allocating an object per value, and is up to 1.9x faster on decimal-heavy schemas. It is opt-in (`spark.comet.exec.columnarToRow.direct.enabled`) because the existing path is still faster for some schemas, such as primitives mixed with strings ([#5496], thanks to [@peterxcli]). -- **Scans**: the next Parquet file's footer and page index load while the current file is read, which the PR - reports as up to 1.9x faster on high-latency storage ([#7094]). Opt-in TopK threshold pushdown reaches the +- **Scans**: the next Parquet file's footer and page index load while the current file is read, which is up to + 1.9x faster on high-latency storage ([#7094]). Opt-in TopK threshold pushdown reaches the Parquet readers ([#6181], [#6263], thanks to [@pingzh]). - **Joins**: decoded probe batches are filtered before intermediate joins ([#6433], thanks to [@sunchao]). From 8d8e4ebfb9f22d6f4287b36543bfb77c667f5ac6 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Mon, 2 Nov 2026 14:34:50 -0700 Subject: [PATCH 21/26] Address comphead's review comments on Comet 1.2.0 blog post --- content/blog/2026-11-03-datafusion-comet-1.2.0.md | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/content/blog/2026-11-03-datafusion-comet-1.2.0.md b/content/blog/2026-11-03-datafusion-comet-1.2.0.md index 38de9fa7..2d3aa148 100644 --- a/content/blog/2026-11-03-datafusion-comet-1.2.0.md +++ b/content/blog/2026-11-03-datafusion-comet-1.2.0.md @@ -133,10 +133,12 @@ bytes ([#5302], [#6206], [#6471], [#6480]). Thanks to [@stantheman0128], [@LinSi each batch repeated in place of the others. Comet now copies each row ([#7260], thanks to [@pingzh]). - **Join output order**: Spark drops a sort above a hash join because the join keeps its streamed side's order. Comet's native hash join did not keep it for some inputs, so the query could return unsorted partitions - ([#6835], [#6810], thanks to [@dwsmith1983]). + ([#6835], thanks to [@dwsmith1983], and [#6810], thanks to [@andygrove]). - **Spark patch releases**: Spark 4.0.5, 4.1.4, and 4.2.1 change what `sequence` returns when `stop - start` overflows a long, and the result of casting a zero with a large exponent, such as `'0e79'`, to decimal. Comet - follows the behavior of the Spark version it runs on ([#7171], thanks to [@snmvaughan]). + follows the behavior of the Spark version it runs on ([#7171], thanks to [@snmvaughan]). These patch releases + are newer than the versions listed under [Compatibility](#compatibility), which are the ones Comet 1.2.0 is + tested against. - **ANSI mode**: `array_append` now evaluates its item on rows whose array is `NULL`, so it raises the errors Spark raises, and `corr` raises `DIVIDE_BY_ZERO` ([#7025], [#6830], thanks to [@kazuyukitanimura], with [@rich7420] on `corr`). @@ -252,7 +254,7 @@ it stays opt-in (`spark.comet.exec.inMemoryCache.enabled`) for 1.2.0. One of tho fix above, is in this release ([#6835]); the rest are tracked in [#6834]. Thanks to [@comphead] for the EXPLAIN and SQL-tab work and [@andygrove] for the rest. -## Reaching New Places +## New Places to Run ### Delta Lake (experimental) @@ -298,7 +300,7 @@ run against it nightly ([#4950], [#6417]). [Nested TPC-H] is a new TPC-H derived benchmark that stores most columns in arrays of structs. At scale factor 1000 on Iceberg tables, Comet 1.2.0 is faster than Spark on all 22 queries and takes 9% less time than 1.1.0 -overall. Spill compression ([#6698]) and nested-field pruning in the native Iceberg scan ([#6725]), both described +overall. q02 is 15% slower than on 1.1.0, and we are investigating it. Spill compression ([#6698]) and nested-field pruning in the native Iceberg scan ([#6725]), both described above, came out of the work on that benchmark. [Nested TPC-H]: https://datafusion.apache.org/blog/2026/10/25/comet-nested-tpch/ @@ -319,11 +321,12 @@ above, came out of the work on that benchmark. ## Upgrading to 1.2.0 -A few changes in 1.2.0 can affect a deployment. The [Comet Upgrade Guide] has the details. +A few changes in 1.2.0 can affect a deployment. The [Comet Upgrade Guide] has more detail on the setting changes +and the new known differences. [Comet Upgrade Guide]: https://datafusion.apache.org/comet/user-guide/latest/migration-guide.html -- **Native Iceberg writes use one setting.** In 1.1.0, native Iceberg writes were a testing feature that needed +- **Native Iceberg writes use one setting.** In 1.1.0, native Iceberg writes were experimental and needed both `spark.comet.iceberg.write.enabled` and `spark.comet.write.iceberg.splitOperator.enabled`. The setting is now `spark.comet.write.iceberg.enabled`, and Comet ignores the old name. Parquet writes were renamed the same way, from `spark.comet.parquet.write.enabled` to `spark.comet.write.parquet.enabled` ([#6494], thanks to From ee1d2d0013884189df4273a1a42b71b4cfaab498 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Mon, 2 Nov 2026 14:35:03 -0700 Subject: [PATCH 22/26] Rewrap nested TPC-H paragraph --- content/blog/2026-11-03-datafusion-comet-1.2.0.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/content/blog/2026-11-03-datafusion-comet-1.2.0.md b/content/blog/2026-11-03-datafusion-comet-1.2.0.md index 2d3aa148..feeec92f 100644 --- a/content/blog/2026-11-03-datafusion-comet-1.2.0.md +++ b/content/blog/2026-11-03-datafusion-comet-1.2.0.md @@ -298,10 +298,10 @@ run against it nightly ([#4950], [#6417]). ## Performance -[Nested TPC-H] is a new TPC-H derived benchmark that stores most columns in arrays of structs. At scale factor -1000 on Iceberg tables, Comet 1.2.0 is faster than Spark on all 22 queries and takes 9% less time than 1.1.0 -overall. q02 is 15% slower than on 1.1.0, and we are investigating it. Spill compression ([#6698]) and nested-field pruning in the native Iceberg scan ([#6725]), both described -above, came out of the work on that benchmark. +[Nested TPC-H] is a new TPC-H derived benchmark that stores most columns in arrays of structs. At scale factor 1000 +on Iceberg tables, Comet 1.2.0 is faster than Spark on all 22 queries and takes 9% less time than 1.1.0 overall. q02 +is 15% slower than on 1.1.0, and we are investigating it. Spill compression ([#6698]) and nested-field pruning in +the native Iceberg scan ([#6725]), both described above, came out of the work on that benchmark. [Nested TPC-H]: https://datafusion.apache.org/blog/2026/10/25/comet-nested-tpch/ From a0cdc30e29a194ae1fea486f3c13321348ad2cc3 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Mon, 2 Nov 2026 14:36:49 -0700 Subject: [PATCH 23/26] Remove q02 note from Comet 1.2.0 blog post --- content/blog/2026-11-03-datafusion-comet-1.2.0.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/content/blog/2026-11-03-datafusion-comet-1.2.0.md b/content/blog/2026-11-03-datafusion-comet-1.2.0.md index feeec92f..f040d838 100644 --- a/content/blog/2026-11-03-datafusion-comet-1.2.0.md +++ b/content/blog/2026-11-03-datafusion-comet-1.2.0.md @@ -299,9 +299,9 @@ run against it nightly ([#4950], [#6417]). ## Performance [Nested TPC-H] is a new TPC-H derived benchmark that stores most columns in arrays of structs. At scale factor 1000 -on Iceberg tables, Comet 1.2.0 is faster than Spark on all 22 queries and takes 9% less time than 1.1.0 overall. q02 -is 15% slower than on 1.1.0, and we are investigating it. Spill compression ([#6698]) and nested-field pruning in -the native Iceberg scan ([#6725]), both described above, came out of the work on that benchmark. +on Iceberg tables, Comet 1.2.0 is faster than Spark on all 22 queries and takes 9% less time than 1.1.0 overall. +Spill compression ([#6698]) and nested-field pruning in the native Iceberg scan ([#6725]), both described above, +came out of the work on that benchmark. [Nested TPC-H]: https://datafusion.apache.org/blog/2026/10/25/comet-nested-tpch/ From d95df1b0e782a77f697485f52e97016d9ae8c55c Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Mon, 2 Nov 2026 14:37:16 -0700 Subject: [PATCH 24/26] Drop nested TPC-H result claims until official benchmarks are run --- content/blog/2026-11-03-datafusion-comet-1.2.0.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/content/blog/2026-11-03-datafusion-comet-1.2.0.md b/content/blog/2026-11-03-datafusion-comet-1.2.0.md index f040d838..1c9f0b7f 100644 --- a/content/blog/2026-11-03-datafusion-comet-1.2.0.md +++ b/content/blog/2026-11-03-datafusion-comet-1.2.0.md @@ -298,10 +298,9 @@ run against it nightly ([#4950], [#6417]). ## Performance -[Nested TPC-H] is a new TPC-H derived benchmark that stores most columns in arrays of structs. At scale factor 1000 -on Iceberg tables, Comet 1.2.0 is faster than Spark on all 22 queries and takes 9% less time than 1.1.0 overall. -Spill compression ([#6698]) and nested-field pruning in the native Iceberg scan ([#6725]), both described above, -came out of the work on that benchmark. +[Nested TPC-H] is a new TPC-H derived benchmark that stores most columns in arrays of structs. Spill compression +([#6698]) and nested-field pruning in the native Iceberg scan ([#6725]), both described above, came out of the work +on that benchmark. [Nested TPC-H]: https://datafusion.apache.org/blog/2026/10/25/comet-nested-tpch/ From 580face409e344eb1ab0785edadcbe25ad7a0869 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Mon, 2 Nov 2026 14:44:57 -0700 Subject: [PATCH 25/26] Revert "Drop nested TPC-H result claims until official benchmarks are run" This reverts commit d95df1b0e782a77f697485f52e97016d9ae8c55c. --- content/blog/2026-11-03-datafusion-comet-1.2.0.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/content/blog/2026-11-03-datafusion-comet-1.2.0.md b/content/blog/2026-11-03-datafusion-comet-1.2.0.md index 1c9f0b7f..f040d838 100644 --- a/content/blog/2026-11-03-datafusion-comet-1.2.0.md +++ b/content/blog/2026-11-03-datafusion-comet-1.2.0.md @@ -298,9 +298,10 @@ run against it nightly ([#4950], [#6417]). ## Performance -[Nested TPC-H] is a new TPC-H derived benchmark that stores most columns in arrays of structs. Spill compression -([#6698]) and nested-field pruning in the native Iceberg scan ([#6725]), both described above, came out of the work -on that benchmark. +[Nested TPC-H] is a new TPC-H derived benchmark that stores most columns in arrays of structs. At scale factor 1000 +on Iceberg tables, Comet 1.2.0 is faster than Spark on all 22 queries and takes 9% less time than 1.1.0 overall. +Spill compression ([#6698]) and nested-field pruning in the native Iceberg scan ([#6725]), both described above, +came out of the work on that benchmark. [Nested TPC-H]: https://datafusion.apache.org/blog/2026/10/25/comet-nested-tpch/ From 91306a0a948b99421ce9e82194bfd3091147d10b Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Mon, 2 Nov 2026 21:26:55 -0700 Subject: [PATCH 26/26] Change Comet 1.2.0 blog post date to 2026-11-02 --- ...sion-comet-1.2.0.md => 2026-11-02-datafusion-comet-1.2.0.md} | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename content/blog/{2026-11-03-datafusion-comet-1.2.0.md => 2026-11-02-datafusion-comet-1.2.0.md} (99%) diff --git a/content/blog/2026-11-03-datafusion-comet-1.2.0.md b/content/blog/2026-11-02-datafusion-comet-1.2.0.md similarity index 99% rename from content/blog/2026-11-03-datafusion-comet-1.2.0.md rename to content/blog/2026-11-02-datafusion-comet-1.2.0.md index f040d838..1ca74f98 100644 --- a/content/blog/2026-11-03-datafusion-comet-1.2.0.md +++ b/content/blog/2026-11-02-datafusion-comet-1.2.0.md @@ -1,7 +1,7 @@ --- layout: post title: Apache DataFusion Comet 1.2.0 Release -date: 2026-11-03 +date: 2026-11-02 author: pmc categories: [subprojects] ---