Add blog post: nested TPC-H benchmark for Comet by comphead · Pull Request #214 · apache/datafusion-site

comphead · 2026-10-26T21:02:43Z

Adds a blog post announcing nested TPC-H, a TPC-H derived benchmark for nested data in Apache DataFusion Comet (apache/datafusion-comet#7044).

Rendered version: content/blog/2026-10-25-comet-nested-tpch.md

The post covers:

Why flat TPC-H and TPC-DS miss problems that show up in real pipelines with nested data.
How the benchmark works. Keys and partition columns stay at the top level, and every other column moves into a single-element array<struct>. The queries return the same answers as TPC-H.
Results at SF1000 on Iceberg tables. Spark takes 1,515 s, unoptimized Comet 1,088 s and optimized Comet 777 s.
The issues the benchmark exposed in Comet (spill compression, nested column pruning, shuffle writes) and in iceberg-rust (nested name mapping, projection mask cost, list views, fixed-size lists).

Files:

content/blog/2026-10-25-comet-nested-tpch.md
content/images/comet-nested-tpch/: two SVG charts and an interactive dashboard.html

Before merge: the image paths are relative (../images/...) so that GitHub renders the post in preview. They need to change to /blog/images/comet-nested-tpch/... to match other posts.

timsaucer

This is a great article. Most of my comments are language nits, and feel free to ignore any of them. There is one real question about stating that only the shape of the tables changing.

I would definitely be interested in testing with struct data, but as you say that's a bigger change.

timsaucer · 2026-10-27T19:26:36Z

+alt="Total execution time at SF1000: Spark 1,515.0 s, Comet nested unoptimized 1,087.5 s, Comet nested optimized 777.2 s"
+/>
+
+Nested TPC-H is a TPC-H derived benchmark in the Comet repository ([#7044](https://github.com/apache/datafusion-comet/pull/7044)). It keeps the TPC-H data and questions and changes only the shape of the tables. The data and queries are engine neutral, so they can be used to compare Spark, Comet and other engines.


"It keeps the TPC-H data and questions and changes only the shape of the tables" was hard for me to read. Maybe I'd say something like "Only the shape of the tables are changed. The TPC-H data and questions are unmodified."

The thing I stumbled on reading was the " -- and -- and -- ".

On reading further down, it looks like we do change the queries by substituting in the explode, so maybe it's not just the shape of the tables?

timsaucer · 2026-10-27T19:29:09Z

+
+Comet hit the shuffle case in [#5397](https://github.com/apache/datafusion-comet/issues/5397), where native shuffle writes of wide nested fields were two to three times slower than Spark's. The issue notes that it was "invisible for TPC-* benchmarks because of historical flat schema".
+
+We needed a benchmark that fills the gap between academic benchmarking and real user workloads.


Suggested change

We needed a benchmark that fills the gap between academic benchmarking and real user workloads.

We needed a benchmark that filled the gap between academic benchmarking and real user workloads.

I think your version is technically right, but the tense switching felt off. Feel free to ignore.

timsaucer · 2026-10-27T19:33:41Z

+Unoptimized Comet takes `1,087.5` s against `1,515.0` s for Spark (1.39x) and is slower on q20 and q21. Optimized Comet takes `777.2` s (1.95x), 29% less than unoptimized, and is faster than Spark on all 22 queries.
+
+<img
+src="../images/comet-nested-tpch/intro-total.svg"


This is repeating the first graphic, right?

its repeating yes, the idea was to keep the first chart as the teaser, and results section replicates the same chart to keep all the results in a single section and avoid the user to scroll back to the beginning for checking the totals. WDYT? is it a good practice to repeat, or it should be structured some other way?

I'd lean toward dropping the second copy of the chart. The paragraph above already gives the totals and speedups. I'd keep the per-query table, though, because people will want to cite specific numbers.

timsaucer · 2026-10-27T19:34:24Z

+alt="Execution time by query for Spark, Comet nested unoptimized and Comet nested optimized"
+/>
+
+| Query | Spark (s) | Comet unoptimized (s) | Comet optimized (s) |


Since you have the chart I think you can probably cut the table, unless you think someone will use it for something.

comphead · 2026-10-27T20:11:21Z

I would definitely be interested in testing with struct data, but as you say that's a bigger change.

this is exactly testing struct/list data, no maps though. Maybe I need to mention it

Co-authored-by: Tim Saucer <timsaucer@gmail.com>

…preview-comet-nested-tpch

andygrove · 2026-10-29T13:27:32Z

@@ -0,0 +1,210 @@
+---
+layout: post
+title: Announcing a derived TPC-H, a benchmark for nested data in Apache DataFusion Comet


"Announcing a derived TPC-H, a benchmark" seems a little awkward to me. Maybe this would be better? Feel free to ignore.

Suggested change

title: Announcing a derived TPC-H, a benchmark for nested data in Apache DataFusion Comet

title: Nested TPC-H: A Benchmark for Nested Data in Apache DataFusion Comet

Thiis would also be more consistent with the opening paragraph, which says "Nested TPC-H is a TPC-H derived benchmark ..."

andygrove · 2026-10-29T13:31:06Z

+
+### In Comet
+
+- **q21 spills uncompressed ([#6467](https://github.com/apache/datafusion-comet/issues/6467), fixed by [#6698](https://github.com/apache/datafusion-comet/pull/6698)).** Spark pushes the `EXISTS` and `NOT EXISTS` predicates below the `explode`, so the whole unpruned `lineitem_data` column goes through the shuffle, sort and joins. The sort spilled several hundred GiB per iteration, and DataFusion writes spill files uncompressed by default. #6698 adds `spark.comet.exec.spill.compression.codec`, which defaults to `lz4`. This is behind q21's regression in the results above. The investigation in #6467 used a separate run on Spark 4, where q21 went from 380 s to 250 s against 367 s for Spark.


"used a separate run on Spark 4" is confusing. I thought that all the runs were using Spark 4.0?

andygrove · 2026-10-29T13:32:37Z

+- **Cluster**: Kubernetes with 16-core executors.
+- **Method**: one run per configuration, three iterations per query. Figures are the fastest iteration, and totals are sums of the per-query times.
+- **Spark**: Spark 4.0
+- **Comet nested unoptimized**: Comet 1.1.0 snapshot (revision `a50fe3aea863`)


I don't think a50fe3aea863 exists. I think it would be better to re-run the benchmarks with 1.0.0 or 1.1.0 versus 1.2.0. You can use latest from branch-1.2 since we do not plan on backporting any PRs that would affect this benchmark.

Also, why use 1.1.0 snapshot? 1.1.0 is already released.

andygrove · 2026-10-29T13:34:59Z

+- **Method**: one run per configuration, three iterations per query. Figures are the fastest iteration, and totals are sums of the per-query times.
+- **Spark**: Spark 4.0
+- **Comet nested unoptimized**: Comet 1.1.0 snapshot (revision `a50fe3aea863`)
+- **Comet nested optimized**: Comet 1.3.0 snapshot (revision `330b29cf`)


This commit is "chore: start 1.3.0 development (#7167)" so effectively it is the same as the 1.2.0 release. Again I think it is cleaner if we just refer to 1.1.0 and 1.2.0 releases. I plan on cutting 1.2.0 release candidate today.

andygrove · 2026-10-29T13:36:18Z

+Query engines get fast at what their benchmarks measure, and the most widely used analytical benchmarks contain no nested data. Apache DataFusion Comet now runs all 22 queries of a nested TPC-H at scale factor 1000 (1 TB) in 777 s, against 1,515 s for Spark.
+
+<img
+src="../images/comet-nested-tpch/intro-total.svg"


These paths will need fixing before we publish.

andygrove · 2026-10-29T13:39:23Z

+- **Method**: one run per configuration, three iterations per query. Figures are the fastest iteration, and totals are sums of the per-query times.
+- **Spark**: Spark 4.0
+- **Comet nested unoptimized**: Comet 1.1.0 snapshot (revision `a50fe3aea863`)
+- **Comet nested optimized**: Comet 1.3.0 snapshot (revision `330b29cf`)


When this is re-run against 1.2.0, could we add a sentence saying which fixes are included? From what I can tell, 1.2.0 has #6698 and #6725, plus iceberg-rust #3384 and #3396 through the pinned iceberg-rust revision, but not #3467 (still open). That would make it clear what the improvement between the two runs comes from.

andygrove · 2026-10-29T13:39:23Z

+
+### In Comet
+
+- **q21 spills uncompressed ([#6467](https://github.com/apache/datafusion-comet/issues/6467), fixed by [#6698](https://github.com/apache/datafusion-comet/pull/6698)).** Spark pushes the `EXISTS` and `NOT EXISTS` predicates below the `explode`, so the whole unpruned `lineitem_data` column goes through the shuffle, sort and joins. The sort spilled several hundred GiB per iteration, and DataFusion writes spill files uncompressed by default. #6698 adds `spark.comet.exec.spill.compression.codec`, which defaults to `lz4`. This is behind q21's regression in the results above. The investigation in #6467 used a separate run on Spark 4, where q21 went from 380 s to 250 s against 367 s for Spark.


Separately from the Spark version question, the numbers here (380 s → 250 s, against 367 s for Spark) don't match the results table (490.6 s → 256.6 s, against 373.5 s). Readers will probably try to reconcile them. I'd either drop them or explain why they're different.

andygrove · 2026-10-29T13:39:23Z

+
+- **q21 spills uncompressed ([#6467](https://github.com/apache/datafusion-comet/issues/6467), fixed by [#6698](https://github.com/apache/datafusion-comet/pull/6698)).** Spark pushes the `EXISTS` and `NOT EXISTS` predicates below the `explode`, so the whole unpruned `lineitem_data` column goes through the shuffle, sort and joins. The sort spilled several hundred GiB per iteration, and DataFusion writes spill files uncompressed by default. #6698 adds `spark.comet.exec.spill.compression.codec`, which defaults to `lz4`. This is behind q21's regression in the results above. The investigation in #6467 used a separate run on Spark 4, where q21 went from 380 s to 250 s against 367 s for Spark.
+- **Nested columns are read in full ([#6724](https://github.com/apache/datafusion-comet/issues/6724), fixed by [#6725](https://github.com/apache/datafusion-comet/pull/6725)).** Comet's native Iceberg scan decoded every field of a nested column even when Spark pruned it. In q20 the `partsupp` scan read 26.4 GiB for a query that needs one field, and #6724 found that scan to be the main cost of Comet's q20. This is why q20 was no faster than Spark in the unoptimized run. #6725 passes the pruned schema, and a microbenchmark of that read went from 624 ms to 62 ms, against 413 ms for Spark. The follow-up [#7019](https://github.com/apache/datafusion-comet/issues/7019) covers a partition column nested inside a pruned struct.
+- **Nested shuffle writes ([#5397](https://github.com/apache/datafusion-comet/issues/5397)).** The motivating issue above. Related work is tracked in [#6855](https://github.com/apache/datafusion-comet/issues/6855).


Shuffle is listed as one of the four groups of issues in the earlier section, but there's no fix linked here. #5397 is closed and #6855 is still open. Does the optimized run include any shuffle improvement? If not, it might be worth saying this is still open work.

andygrove · 2026-10-29T13:39:23Z

+- **Data**: nested TPC-H, scale factor 1000 (1 TB).
+- **Tables**: Apache Iceberg tables with zstd-compressed Parquet data files on S3, created with `nest_tpch.scala`. Comet reads them with its native Iceberg scan, which is built on [iceberg-rust](https://github.com/apache/iceberg-rust).
+- **Cluster**: Kubernetes with 16-core executors.
+- **Method**: one run per configuration, three iterations per query. Figures are the fastest iteration, and totals are sums of the per-query times.


With one run per configuration, a few queries come out slightly slower optimized than unoptimized (q05 47.4 s vs 45.7 s, q08 42.8 s vs 42.0 s). That's probably noise, but a short note on run-to-run variance would stop readers from reading too much into small differences.

andygrove · 2026-10-29T13:39:23Z

+- **List view columns read as NULL ([#3393](https://github.com/apache/iceberg-rust/issues/3393), fix in review in [#3467](https://github.com/apache/iceberg-rust/pull/3467)).** When a file's embedded Arrow schema asks for `ListView`, the projection treats the whole list as one leaf. The column reads as NULL, and a list view of structs shifts every later column to the wrong Parquet leaf. #3467 reads list views as lists.
+- **Fixed-size list columns fail to read ([#3471](https://github.com/apache/iceberg-rust/issues/3471), open).** This is the same kind of problem for `FixedSizeList`. The reader cannot convert it to Iceberg's `list` type.
+
+## Credits


This section doesn't credit anyone. Maybe list the people who worked on the fixes in Comet and iceberg-rust, or rename it to something like "Get involved"?

andygrove · 2026-10-29T13:39:23Z

+title: Announcing a derived TPC-H, a benchmark for nested data in Apache DataFusion Comet
+date: 2026-10-25
+author: pmc
+categories: [performance, benchmarks]


Nit: benchmarks would be a new category. Other posts use performance and subprojects, so maybe [performance, subprojects]?

andygrove · 2026-10-29T13:39:24Z

+
+## What nested benchmarks revealed
+
+The nested schema is: single-element arrays of structs, with no maps for now. Even that was enough to expose issues that were invisible on flat TPC-H. In the unoptimized run, Comet was slower than Spark on q20 (32.9 s against 31.5 s) and q21 (490.6 s against 373.5 s), and barely faster on q02 (34.0 s against 34.9 s). Tracing those queries surfaced a lot of technical debt in how nested data is read, shuffled and spilled, in Comet and in the iceberg-rust reader underneath it.


Nit: the colon after "The nested schema is" is stray.

andygrove · 2026-10-29T13:39:24Z

+
+## Why: TPC-H is flat, real data is not
+
+[TPC-H](https://www.tpc.org/tpch/) is one of the most cited analytical benchmarks, and its schema is flat: eight normalized tables with primitive columns. Its 22 queries exercise well-understood choke points such as aggregation, join ordering and correlated subqueries (Boncz, Neumann and Erling, "TPC-H Analyzed", TPCTC 2013).


Nit: could we link the Boncz et al. and Melnik et al. (Dremel) papers? (The Dremel citation is on line 49.)

andygrove · 2026-10-29T13:39:24Z

+
+[TOC]
+
+Query engines get fast at what their benchmarks measure, and the most widely used analytical benchmarks contain no nested data. Apache DataFusion Comet now runs all 22 queries of a nested TPC-H at scale factor 1000 (1 TB) in 777 s, against 1,515 s for Spark.


Nit: I think a more concrete opening would be stronger than the general statement about benchmarks, e.g. leading with the fact that Comet's TPC-H gains weren't showing up in users' nested pipelines. Feel free to ignore.

andygrove · 2026-10-29T13:39:32Z

A couple of housekeeping items before merge:

The post date (2026-10-25) needs updating to the publish date.
The PR description mentions an interactive dashboard.html, but it isn't in the PR. Either add it or update the description.

andygrove

Requesting changes so this doesn't get merged before the benchmarks are re-run against the 1.1.0 and 1.2.0 releases, the Spark version question is resolved, and the image paths are fixed.

comphead changed the title ~~Comet nested benchmark experiment~~ Add blog post: nested TPC-H benchmark for Comet Oct 26, 2026

Add blog post: nested TPC-H benchmark for Comet

3802cff

comphead force-pushed the preview-comet-nested-tpch branch from 90163db to 3802cff Compare October 26, 2026 21:14

comphead requested review from alamb and andygrove October 26, 2026 21:17

manuzhang reviewed Oct 27, 2026

View reviewed changes

Comment thread content/blog/2026-10-25-comet-nested-tpch.md

fixes

f45ee2d

timsaucer approved these changes Oct 27, 2026

View reviewed changes

comphead and others added 4 commits October 28, 2026 16:13

Update content/blog/2026-10-25-comet-nested-tpch.md

b769570

Co-authored-by: Tim Saucer <timsaucer@gmail.com>

fixes

eb87551

Merge remote-tracking branch 'origin/preview-comet-nested-tpch' into …

3c37ec2

…preview-comet-nested-tpch

fixes

229f8e9

andygrove reviewed Oct 29, 2026

View reviewed changes

andygrove requested changes Oct 29, 2026

View reviewed changes


		Comet hit the shuffle case in [#5397](https://github.com/apache/datafusion-comet/issues/5397), where native shuffle writes of wide nested fields were two to three times slower than Spark's. The issue notes that it was "invisible for TPC-* benchmarks because of historical flat schema".

		We needed a benchmark that fills the gap between academic benchmarking and real user workloads.

	We needed a benchmark that fills the gap between academic benchmarking and real user workloads.
	We needed a benchmark that filled the gap between academic benchmarking and real user workloads.

	title: Announcing a derived TPC-H, a benchmark for nested data in Apache DataFusion Comet
	title: Nested TPC-H: A Benchmark for Nested Data in Apache DataFusion Comet


		### In Comet

		- q21 spills uncompressed ([#6467](https://github.com/apache/datafusion-comet/issues/6467), fixed by [#6698](https://github.com/apache/datafusion-comet/pull/6698)). Spark pushes the `EXISTS` and `NOT EXISTS` predicates below the `explode`, so the whole unpruned `lineitem_data` column goes through the shuffle, sort and joins. The sort spilled several hundred GiB per iteration, and DataFusion writes spill files uncompressed by default. #6698 adds `spark.comet.exec.spill.compression.codec`, which defaults to `lz4`. This is behind q21's regression in the results above. The investigation in #6467 used a separate run on Spark 4, where q21 went from 380 s to 250 s against 367 s for Spark.


		## What nested benchmarks revealed

		The nested schema is: single-element arrays of structs, with no maps for now. Even that was enough to expose issues that were invisible on flat TPC-H. In the unoptimized run, Comet was slower than Spark on q20 (32.9 s against 31.5 s) and q21 (490.6 s against 373.5 s), and barely faster on q02 (34.0 s against 34.9 s). Tracing those queries surfaced a lot of technical debt in how nested data is read, shuffled and spilled, in Comet and in the iceberg-rust reader underneath it.


		## Why: TPC-H is flat, real data is not

		[TPC-H](https://www.tpc.org/tpch/) is one of the most cited analytical benchmarks, and its schema is flat: eight normalized tables with primitive columns. Its 22 queries exercise well-understood choke points such as aggregation, join ordering and correlated subqueries (Boncz, Neumann and Erling, "TPC-H Analyzed", TPCTC 2013).


		[TOC]

		Query engines get fast at what their benchmarks measure, and the most widely used analytical benchmarks contain no nested data. Apache DataFusion Comet now runs all 22 queries of a nested TPC-H at scale factor 1000 (1 TB) in 777 s, against 1,515 s for Spark.

Conversation

comphead commented Oct 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

timsaucer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

comphead commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andygrove commented Oct 29, 2026

Uh oh!

andygrove left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

comphead commented Oct 26, 2026 •

edited

Loading

comphead commented Oct 27, 2026 •

edited

Loading