Skip to content

[SPARK-56887][SQL][FOLLOWUP] Add spill support and extend ShuffledJoin in SortMergeAsOfJoinExec #56595

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sarutak wants to merge 2 commits into
base: master
Choose a base branch
from
Open

[SPARK-56887][SQL][FOLLOWUP] Add spill support and extend ShuffledJoin in SortMergeAsOfJoinExec #56595

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9d30286
[SPARK-56887][SQL][FOLLOW-UP] Extend ShuffledJoin + spill support + f…
sarutak Jun 15, 2026
2401517
Benchmark results for AsOfJoinBenchmark (JDK 17/21/25, AMD EPYC 7763)
sarutak Jun 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions sql/core/benchmarks/AsOfJoinBenchmark-jdk21-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,18 +2,18 @@
AS-OF Join Benchmark
================================================================================================

OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1010-azure
OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1018-azure
AMD EPYC 7763 64-Core Processor
AS-OF Join (left=10000, right=10000, groups=100): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
--------------------------------------------------------------------------------------------------------------------------------
Correlated subquery (baseline) 37494 37697 287 0.0 3749357.9 1.0X
Sort-merge AS-OF join 62 85 19 0.2 6236.4 601.2X
Correlated subquery (baseline) 38299 38496 279 0.0 3829864.5 1.0X
Sort-merge AS-OF join 68 92 13 0.1 6813.3 562.1X

OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1010-azure
OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1018-azure
AMD EPYC 7763 64-Core Processor
AS-OF Join no equi-key (left=10000, right=10000): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
--------------------------------------------------------------------------------------------------------------------------------
Correlated subquery (baseline) 23664 23684 27 0.0 2366425.3 1.0X
Sort-merge AS-OF join 1773 1795 32 0.0 177269.8 13.3X
Correlated subquery (baseline) 24733 24827 133 0.0 2473327.2 1.0X
Sort-merge AS-OF join 1570 1581 16 0.0 156990.6 15.8X


12 changes: 6 additions & 6 deletions sql/core/benchmarks/AsOfJoinBenchmark-jdk25-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,18 +2,18 @@
AS-OF Join Benchmark
================================================================================================

OpenJDK 64-Bit Server VM 25.0.3+9-LTS on Linux 6.17.0-1010-azure
OpenJDK 64-Bit Server VM 25.0.3+9-LTS on Linux 6.17.0-1018-azure
AMD EPYC 7763 64-Core Processor
AS-OF Join (left=10000, right=10000, groups=100): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
--------------------------------------------------------------------------------------------------------------------------------
Correlated subquery (baseline) 37549 37690 199 0.0 3754903.0 1.0X
Sort-merge AS-OF join 56 73 14 0.2 5550.4 676.5X
Correlated subquery (baseline) 40197 40231 47 0.0 4019717.4 1.0X
Sort-merge AS-OF join 61 89 15 0.2 6093.4 659.7X

OpenJDK 64-Bit Server VM 25.0.3+9-LTS on Linux 6.17.0-1010-azure
OpenJDK 64-Bit Server VM 25.0.3+9-LTS on Linux 6.17.0-1018-azure
AMD EPYC 7763 64-Core Processor
AS-OF Join no equi-key (left=10000, right=10000): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
--------------------------------------------------------------------------------------------------------------------------------
Correlated subquery (baseline) 22100 22128 40 0.0 2209969.1 1.0X
Sort-merge AS-OF join 1614 1624 13 0.0 161417.4 13.7X
Correlated subquery (baseline) 23803 23818 21 0.0 2380348.1 1.0X
Sort-merge AS-OF join 1585 1602 24 0.0 158493.5 15.0X


12 changes: 6 additions & 6 deletions sql/core/benchmarks/AsOfJoinBenchmark-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,18 +2,18 @@
AS-OF Join Benchmark
================================================================================================

OpenJDK 64-Bit Server VM 17.0.19+10-LTS on Linux 6.17.0-1010-azure
OpenJDK 64-Bit Server VM 17.0.19+10-LTS on Linux 6.17.0-1018-azure
AMD EPYC 7763 64-Core Processor
AS-OF Join (left=10000, right=10000, groups=100): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
--------------------------------------------------------------------------------------------------------------------------------
Correlated subquery (baseline) 37391 37469 110 0.0 3739110.1 1.0X
Sort-merge AS-OF join 59 70 9 0.2 5918.1 631.8X
Correlated subquery (baseline) 37910 37980 99 0.0 3790977.0 1.0X
Sort-merge AS-OF join 53 65 9 0.2 5264.7 720.1X

OpenJDK 64-Bit Server VM 17.0.19+10-LTS on Linux 6.17.0-1010-azure
OpenJDK 64-Bit Server VM 17.0.19+10-LTS on Linux 6.17.0-1018-azure
AMD EPYC 7763 64-Core Processor
AS-OF Join no equi-key (left=10000, right=10000): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
--------------------------------------------------------------------------------------------------------------------------------
Correlated subquery (baseline) 24013 24033 28 0.0 2401313.2 1.0X
Sort-merge AS-OF join 1713 1719 9 0.0 171343.3 14.0X
Correlated subquery (baseline) 23421 23429 11 0.0 2342088.5 1.0X
Sort-merge AS-OF join 1399 1402 4 0.0 139932.2 16.7X


Loading