improve: task serialization size is too large

[wForget]

Is your feature request related to a problem? Please describe.

https://github.com/wForget/benchmarks-spark-native/actions/runs/27939521326/job/82669454968

Running case: q64
......
WARN DAGScheduler: Broadcasting large task binary with size 140.0 MiB

Describe the solution you'd like

Describe alternatives you've considered

Additional context

[wForget]

The Broadcast object generated by NativeBroadcastExchangeBase.doExecuteBroadcast holds all rows data during serialization.

We should avoid serializing v.

auron/spark-extension/src/main/scala/org/apache/spark/sql/execution/auron/plan/NativeBroadcastExchangeBase.scala

Line 151 in ca3bece

override protected def getValue(): Any = v

[Sigma-Ma]

Hi @wForget, I’d like to work on this issue.

The large task binary comes from the anonymous Broadcast in doExecuteBroadcast, which captures the transformed relation in the task closure.

I plan to use a real Spark broadcast for that relation while keeping the native IPC path unchanged, and add regression tests for serialization size and mixed native/Spark execution. Please let me know if this approach sounds good. Thanks!

[wForget]

@Sigma-Ma Thanks, this solution sounds good, please feel free to submit a PR.

[Sigma-Ma]

Hi @wForget, I’ve opened #2506 with the fix.

It replaces the value-capturing wrapper with a real Spark broadcast while keeping the native IPC path unchanged. I also added regression coverage for mixed native/Spark execution and serialized broadcast size.