[AURON #1863] Support native Flink UNIX_TIMESTAMP: converter integration (reviewer preview) - #1
Closed
weiqingy wants to merge 30 commits into
Closed
[AURON #1863] Support native Flink UNIX_TIMESTAMP: converter integration (reviewer preview)#1weiqingy wants to merge 30 commits into
weiqingy wants to merge 30 commits into
Conversation
# Which issue does this PR close? Closes apache#2352 # Rationale for this change `try_new_spill` always uses `spills.len` as the `spill_id`, so we should always lock `spills` before `trying_new_spill`. # What changes are included in this PR? # Are there any user-facing changes? # How was this patch tested? --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
…pache#2399) ### Which issue does this PR close? Closes apache#2398 ### Rationale for this change The `auron-build.sh` script incorrectly handles Maven `-D` arguments. The script previously stored these arguments in a single string and added a leading space before the first argument. It then passed the complete string to Maven as one quoted argument. For example, Maven received: ```text " -DskipBuildNative" ``` instead of: ``` "-DskipBuildNative" ``` This caused Maven to report: ``` Unknown lifecycle phase " -DskipBuildNative" ``` ### What changes are included in this PR? - Replace the string-based `MVN_D_ARGS` variable with a Bash array. - Store every Maven `-D` option as an independent array element. - Pass Maven properties using quoted array expansion. - Preserve multiple Maven properties and values containing spaces. - Simplify the local Maven command execution path. ### Are there any user-facing changes? Yes. Users can now pass Maven `-D` properties through `auron-build.sh` as documented. For example: ``` ./auron-build.sh --pre --sparkver 4.1 --scalaver 2.13 -DskipBuildNative ``` Multiple Maven properties are also preserved as separate arguments. ### How was this patch tested? - Reproduced the original behavior and confirmed that the script produced: ``` < -DskipBuildNative> ``` - Verified that the fixed script produces: ``` <-DskipBuildNative> ``` ### Was this patch authored or co-authored using generative AI tooling? - [ ] Yes - [x] No If yes, include: `Generated-by: <tool name and version>` ASF guidance: https://www.apache.org/legal/generative-tooling.html Signed-off-by: slfan1989 <slfan1989@apache.org>
Bumps [futures-util](https://github.com/rust-lang/futures-rs) from 0.3.32 to 0.3.33. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/rust-lang/futures-rs/releases">futures-util's releases</a>.</em></p> <blockquote> <h2>0.3.33</h2> <ul> <li>Fix <code>ReadLine</code>'s soundness issue regarding to exception safety. (<a href="https://redirect.github.com/rust-lang/futures-rs/issues/3020">#3020</a>)</li> <li>Fix unsound <code>Send</code> impl for <code>IterPinRef</code> and <code>Iter</code>. (<a href="https://redirect.github.com/rust-lang/futures-rs/issues/3003">#3003</a>)</li> <li>Fix stacked borrows violation in <code>compat01as03</code> implementation. (<a href="https://redirect.github.com/rust-lang/futures-rs/issues/3012">#3012</a>)</li> <li>Fix memory leak in <code>FuturesUnordered::IntoIter</code>. (<a href="https://redirect.github.com/rust-lang/futures-rs/issues/3005">#3005</a>)</li> <li>Add <code>portable-atomic-alloc</code> feature and use it in <code>FuturesUnordered</code>. (<a href="https://redirect.github.com/rust-lang/futures-rs/issues/3007">#3007</a>)</li> <li>Re-export <code>alloc::task::Wake</code>. (<a href="https://redirect.github.com/rust-lang/futures-rs/issues/3010">#3010</a>)</li> <li>Update <code>spin</code> to 0.12. (<a href="https://redirect.github.com/rust-lang/futures-rs/issues/3014">#3014</a>)</li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/rust-lang/futures-rs/blob/main/CHANGELOG.md">futures-util's changelog</a>.</em></p> <blockquote> <h1>0.3.33 - 2026-07-18</h1> <ul> <li>Fix <code>ReadLine</code>'s soundness issue regarding to exception safety. (<a href="https://redirect.github.com/rust-lang/futures-rs/issues/3020">#3020</a>)</li> <li>Fix unsound <code>Send</code> impl for <code>IterPinRef</code> and <code>Iter</code>. (<a href="https://redirect.github.com/rust-lang/futures-rs/issues/3003">#3003</a>)</li> <li>Fix stacked borrows violation in <code>compat01as03</code> implementation. (<a href="https://redirect.github.com/rust-lang/futures-rs/issues/3012">#3012</a>)</li> <li>Fix memory leak in <code>FuturesUnordered::IntoIter</code>. (<a href="https://redirect.github.com/rust-lang/futures-rs/issues/3005">#3005</a>)</li> <li>Add <code>portable-atomic-alloc</code> feature and use it in <code>FuturesUnordered</code>. (<a href="https://redirect.github.com/rust-lang/futures-rs/issues/3007">#3007</a>)</li> <li>Re-export <code>alloc::task::Wake</code>. (<a href="https://redirect.github.com/rust-lang/futures-rs/issues/3010">#3010</a>)</li> <li>Update <code>spin</code> to 0.12. (<a href="https://redirect.github.com/rust-lang/futures-rs/issues/3014">#3014</a>)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/rust-lang/futures-rs/commit/89cc254cb8bfcc78e6a31e7711d0757f97bcb531"><code>89cc254</code></a> Release 0.3.33</li> <li><a href="https://github.com/rust-lang/futures-rs/commit/cd9f5befe699dab2e94cfd32708879b41be0960a"><code>cd9f5be</code></a> ci: Update release workflow</li> <li><a href="https://github.com/rust-lang/futures-rs/commit/d79a499c5f7b4427b8dfbee2216b0ff042bb38e4"><code>d79a499</code></a> Resolve rustdoc::broken_intra_doc_links warning</li> <li><a href="https://github.com/rust-lang/futures-rs/commit/95bbcf83be9e414f1e5b9e0f5e0c91f1d8e433e3"><code>95bbcf8</code></a> Resolve rustdoc ambiguous link error</li> <li><a href="https://github.com/rust-lang/futures-rs/commit/303c1658dc7bde42cf15d7a09621de8c6c857b37"><code>303c165</code></a> Resolve rustdoc::redundant_explicit_links warning</li> <li><a href="https://github.com/rust-lang/futures-rs/commit/f34e3f5b9daba3901a464a0694e448a7eacf0e19"><code>f34e3f5</code></a> ci: Cleanup</li> <li><a href="https://github.com/rust-lang/futures-rs/commit/66591a24275a79d9a77330dbb9106b79f623aa0d"><code>66591a2</code></a> Enable Miri for more tests</li> <li><a href="https://github.com/rust-lang/futures-rs/commit/ab1072fec105b27e5133f9f7dd01f44fba55629f"><code>ab1072f</code></a> Simplify target_has_atomic cfg in utility crates</li> <li><a href="https://github.com/rust-lang/futures-rs/commit/cf5d23b68bf7eafb926a2d485b1a1276a4a4e146"><code>cf5d23b</code></a> Fix unsound compat01as03 implementation (fixes <a href="https://redirect.github.com/rust-lang/futures-rs/issues/2514">#2514</a>) (<a href="https://redirect.github.com/rust-lang/futures-rs/issues/3012">#3012</a>)</li> <li><a href="https://github.com/rust-lang/futures-rs/commit/8ae794faefbbae7f0070d11f75ea7e56bc2698c5"><code>8ae794f</code></a> Add portable-atomic-alloc feature and use it in FuturesUnordered (<a href="https://redirect.github.com/rust-lang/futures-rs/issues/3007">#3007</a>)</li> <li>Additional commits viewable in <a href="https://github.com/rust-lang/futures-rs/compare/0.3.32...0.3.33">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [async-trait](https://github.com/dtolnay/async-trait) from 0.1.89 to 0.1.91. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/dtolnay/async-trait/releases">async-trait's releases</a>.</em></p> <blockquote> <h2>0.1.90</h2> <ul> <li>Update to syn 3</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/dtolnay/async-trait/commit/d049ee02a2d50b72e03d07f06311e23bf5b512a8"><code>d049ee0</code></a> Release 0.1.91</li> <li><a href="https://github.com/dtolnay/async-trait/commit/7a0961f275432c40cc5e7aa011362e4b50d763b1"><code>7a0961f</code></a> Merge pull request <a href="https://redirect.github.com/dtolnay/async-trait/issues/301">#301</a> from dtolnay/mutability</li> <li><a href="https://github.com/dtolnay/async-trait/commit/740f86f23d176229011f2389c8a206ef5ba547e7"><code>740f86f</code></a> Ignore mut_mut pedantic clippy lint in test</li> <li><a href="https://github.com/dtolnay/async-trait/commit/4699cd320a8aaaf06a2a369cb9e1f2964b14b71c"><code>4699cd3</code></a> Fix mutability for by-reference receivers</li> <li><a href="https://github.com/dtolnay/async-trait/commit/6dd3573df95878d34fcfc0ab9c242aeab3140f82"><code>6dd3573</code></a> Add regression test for issue 300</li> <li><a href="https://github.com/dtolnay/async-trait/commit/2371797a3938808bd7e1f4f9abd0eed51bd99634"><code>2371797</code></a> Release 0.1.90</li> <li><a href="https://github.com/dtolnay/async-trait/commit/d03f075ecc2b9fcbf6757f3654a7974a518a144e"><code>d03f075</code></a> Merge pull request <a href="https://redirect.github.com/dtolnay/async-trait/issues/299">#299</a> from dtolnay/syn3</li> <li><a href="https://github.com/dtolnay/async-trait/commit/6cf42c104d1c02aa97d4fc62ff117f8d6b05eacb"><code>6cf42c1</code></a> Update to syn 3</li> <li><a href="https://github.com/dtolnay/async-trait/commit/b9daabad756580d31bd2b9221ea599db51bf6cdd"><code>b9daaba</code></a> Ignore match_same_arms pedantic clippy lint</li> <li><a href="https://github.com/dtolnay/async-trait/commit/aa706d127114e57dc163238af947ba495b0b86d2"><code>aa706d1</code></a> Update actions/upload-artifact@v6 -> v7</li> <li>Additional commits viewable in <a href="https://github.com/dtolnay/async-trait/compare/0.1.89...0.1.91">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.52.3 to 1.53.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/tokio-rs/tokio/releases">tokio's releases</a>.</em></p> <blockquote> <h2>Tokio v1.53.0</h2> <h1>1.53.0 (July 17th, 2026)</h1> <h3>Added</h3> <ul> <li>fs: implement <code>From<OwnedFd></code> and <code>From<OwnedHandle></code> for <code>File</code> (<a href="https://redirect.github.com/tokio-rs/tokio/issues/8266">#8266</a>)</li> <li>metrics: add task schedule latency metric (<a href="https://redirect.github.com/tokio-rs/tokio/issues/7986">#7986</a>)</li> <li>net: add <code>SocketAddr</code> methods to Unix sockets (<a href="https://redirect.github.com/tokio-rs/tokio/issues/8144">#8144</a>)</li> </ul> <h3>Changed</h3> <ul> <li>io: add <code>#[inline]</code> to IO trait impls for in-memory types (<a href="https://redirect.github.com/tokio-rs/tokio/issues/8242">#8242</a>)</li> <li>net: implement UCred::pid on FreeBSD (<a href="https://redirect.github.com/tokio-rs/tokio/issues/8086">#8086</a>)</li> <li>net: support Nuttx target os (<a href="https://redirect.github.com/tokio-rs/tokio/issues/8259">#8259</a>)</li> <li>signal: refactor global variables on Windows (<a href="https://redirect.github.com/tokio-rs/tokio/issues/8231">#8231</a>)</li> <li>sync: <code>mpsc::{Receiver,UnboundedReceiver}</code> now drops waker on drop, even if there are still senders (<a href="https://redirect.github.com/tokio-rs/tokio/issues/8095">#8095</a>)</li> <li>taskdump: support taskdumps on s390x (<a href="https://redirect.github.com/tokio-rs/tokio/issues/8192">#8192</a>)</li> <li>time: add <code>#[track_caller]</code> to <code>timeout_at()</code> (<a href="https://redirect.github.com/tokio-rs/tokio/issues/8077">#8077</a>)</li> <li>time: consolidate mutex locks on spurious poll (<a href="https://redirect.github.com/tokio-rs/tokio/issues/8124">#8124</a>)</li> <li>time: defer waker clone on spurious poll (<a href="https://redirect.github.com/tokio-rs/tokio/issues/8107">#8107</a>)</li> <li>time: move lazy-registration state into <code>Sleep</code> (<a href="https://redirect.github.com/tokio-rs/tokio/issues/8132">#8132</a>)</li> <li>tracing: remove unnecessary span clone (<a href="https://redirect.github.com/tokio-rs/tokio/issues/8126">#8126</a>)</li> </ul> <h3>Fixed</h3> <ul> <li>io: do not treat zero-length reads as EOF in <code>Chain</code> (<a href="https://redirect.github.com/tokio-rs/tokio/issues/8251">#8251</a>)</li> <li>net: use getpeereid for QNX peer credentials (<a href="https://redirect.github.com/tokio-rs/tokio/issues/8270">#8270</a>)</li> <li>runtime: avoid illegal state in <code>FastRand</code> (<a href="https://redirect.github.com/tokio-rs/tokio/issues/8078">#8078</a>)</li> <li>sync: wake mpsc receiver when a queued <code>reserve[_many]</code> returns permits (<a href="https://redirect.github.com/tokio-rs/tokio/issues/8260">#8260</a>)</li> <li>taskdump: skip double wake on <code>Trace::capture</code>/<code>Trace::trace_with</code> (<a href="https://redirect.github.com/tokio-rs/tokio/issues/8043">#8043</a>)</li> <li>time: avoid stack overflow in runtime constructor (<a href="https://redirect.github.com/tokio-rs/tokio/issues/8093">#8093</a>)</li> <li>time (alt timer): ensure timers stay in the same runtime after <code>.reset()</code> (<a href="https://redirect.github.com/tokio-rs/tokio/issues/8169">#8169</a>)</li> </ul> <h3>IO uring (unstable)</h3> <ul> <li>fs: use io-uring for <code>fs::try_exists</code> (<a href="https://redirect.github.com/tokio-rs/tokio/issues/8080">#8080</a>)</li> <li>fs: use io-uring for renaming files (<a href="https://redirect.github.com/tokio-rs/tokio/issues/7800">#7800</a>)</li> <li>rt: flush io-uring CQE in case of CQE overflow (<a href="https://redirect.github.com/tokio-rs/tokio/issues/8277">#8277</a>)</li> </ul> <h3>Documented</h3> <ul> <li>docs: clarify cancel safety wording (<a href="https://redirect.github.com/tokio-rs/tokio/issues/8181">#8181</a>)</li> <li>fs: clarify <code>create_dir_all</code> succeeds if path exists (<a href="https://redirect.github.com/tokio-rs/tokio/issues/8149">#8149</a>)</li> <li>io: add warning about stdout reordering with multiple handles (<a href="https://redirect.github.com/tokio-rs/tokio/issues/8276">#8276</a>)</li> <li>net: document pipe <code>try_read*</code>/<code>try_write*</code> readiness behavior (<a href="https://redirect.github.com/tokio-rs/tokio/issues/8032">#8032</a>)</li> <li>runtime: document interaction with fork() (<a href="https://redirect.github.com/tokio-rs/tokio/issues/8202">#8202</a>)</li> <li>sync: clarify broadcast lagging semantics (<a href="https://redirect.github.com/tokio-rs/tokio/issues/8239">#8239</a>)</li> <li>sync: document memory ordering guarantees for Semaphore (<a href="https://redirect.github.com/tokio-rs/tokio/issues/8119">#8119</a>)</li> <li>task: explain why <code>yield_now</code> defers its waker (<a href="https://redirect.github.com/tokio-rs/tokio/issues/8254">#8254</a>)</li> <li>time: add panic docs to <code>timeout_at()</code> (<a href="https://redirect.github.com/tokio-rs/tokio/issues/8077">#8077</a>)</li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/tokio-rs/tokio/commit/be689a35f5ade5a39e507f79d3ec85cdab27806f"><code>be689a3</code></a> chore: prepare Tokio v1.53.0 (<a href="https://redirect.github.com/tokio-rs/tokio/issues/8294">#8294</a>)</li> <li><a href="https://github.com/tokio-rs/tokio/commit/50f76c71ec7203013f7f0cda59deaa9016e93939"><code>50f76c7</code></a> chore: prepare tokio-macros v2.7.1 (<a href="https://redirect.github.com/tokio-rs/tokio/issues/8295">#8295</a>)</li> <li><a href="https://github.com/tokio-rs/tokio/commit/f61fccad3cd598cce743fc511a983364b77af92a"><code>f61fcca</code></a> Merge 'tokio-1.52.4' into 'master' (<a href="https://redirect.github.com/tokio-rs/tokio/issues/8290">#8290</a>)</li> <li><a href="https://github.com/tokio-rs/tokio/commit/efdba5fcf02c4b93d379114df136b994c3b21445"><code>efdba5f</code></a> chore: prepare Tokio v1.52.4 (<a href="https://redirect.github.com/tokio-rs/tokio/issues/8289">#8289</a>)</li> <li><a href="https://github.com/tokio-rs/tokio/commit/b0ba02e75507518baed6718b0c37105e430f3a93"><code>b0ba02e</code></a> Merge 'tokio-1.51.4' into 'tokio-1.52.x' (<a href="https://redirect.github.com/tokio-rs/tokio/issues/8288">#8288</a>)</li> <li><a href="https://github.com/tokio-rs/tokio/commit/7bcd2d343d997a658be5bc165520070c997fefbc"><code>7bcd2d3</code></a> taskdump: remove crate disambiguators from output (<a href="https://redirect.github.com/tokio-rs/tokio/issues/8288">#8288</a>)</li> <li><a href="https://github.com/tokio-rs/tokio/commit/f84b209126a9b0f66dae1025e30e44d217205513"><code>f84b209</code></a> chore: prepare Tokio v1.51.4 (<a href="https://redirect.github.com/tokio-rs/tokio/issues/8286">#8286</a>)</li> <li><a href="https://github.com/tokio-rs/tokio/commit/eacb98e18901fcbe011bfceddbc31853f1bf0c27"><code>eacb98e</code></a> runtime: don't skip the driver when <code>before_park</code> schedules work (<a href="https://redirect.github.com/tokio-rs/tokio/issues/8222">#8222</a>)</li> <li><a href="https://github.com/tokio-rs/tokio/commit/5e16ee00faae5b81a58f70b734577b800e0b203b"><code>5e16ee0</code></a> task: avoid replacing the JoinQueue waker in <code>try_join_next</code> (<a href="https://redirect.github.com/tokio-rs/tokio/issues/8279">#8279</a>)</li> <li><a href="https://github.com/tokio-rs/tokio/commit/88212ab64afd2b2f87deeb073f9eef6aaaad3701"><code>88212ab</code></a> sync: document memory ordering guarantees for Semaphore (<a href="https://redirect.github.com/tokio-rs/tokio/issues/8119">#8119</a>)</li> <li>Additional commits viewable in <a href="https://github.com/tokio-rs/tokio/compare/tokio-1.52.3...tokio-1.53.0">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
# Which issue does this PR close? Closes apache#2372 # Rationale for this change Docker build mode needs a way to specify the Docker platform from `auron-build.sh`, especially on ARM hosts where release builds may need to run as `linux/amd64`. # What changes are included in this PR? as title # Are there any user-facing changes? no # How was this patch tested? rebuild auron with docker mode
This was referenced Jul 21, 2026
weiqingy
force-pushed
the
AURON-1863-converter
branch
from
July 21, 2026 19:33
774e1dd to
e309ae1
Compare
…pache#2414) # Which issue does this PR close? Closes apache#2413. # Rationale for this change apache#2410 bumped `actions/labeler` from v6 to v7, which brought `js-yaml` `^5.1.0`. js-yaml 5.x enforces stricter indentation on multi-line flow sequences and rejects the current `.github/labeler.yml` with `YAMLException: deficient indentation (27:7)`, so the `triage` check now fails on PRs that run after that bump. # What changes are included in this PR? Rewrite the glob lists in `.github/labeler.yml` from multi-line flow sequences (`[ ... ]`) to block sequences, for all 17 label entries. The label names and glob patterns are unchanged; only the YAML list style changes. Verified with `js-yaml` 5.2.1 (the version labeler v7 resolves): the current file throws `deficient indentation (27:7)`, and the updated file parses cleanly into the same 17 labels and 36 glob patterns. # Are there any user-facing changes? No. This is a CI configuration fix. # How was this patch tested? Loaded both the old and new `.github/labeler.yml` with `js-yaml` 5.2.1. The old file reproduces the CI error; the new file parses without error and yields identical label-to-glob mappings. # Note The `triage` check on this PR is expected to fail until this is merged. The labeler workflow runs on `pull_request_target` and reads `.github/labeler.yml` from the base branch (master), not from the PR, so it still sees the current file until the fix lands on master.
…erformance (apache#2339) # Which issue does this PR close? Closes apache#2320 # Rationale for this change - Boolean type was not given a default value, causing incorrect data results. - Complex nested PB structures can encounter parsing errors. # What changes are included in this PR? - Fix boolean type was not given a default value, causing incorrect data results. - FIx complex nested PB structures can encounter parsing errors. - improve PB parsing performance # Are there any user-facing changes? - No # How was this patch tested? - No
# Which issue does this PR close? Closes apache#2417 # Rationale for this change Now when shuffle is written, there is a shuffle size after log output compression, but there is no output compression size when using rss. https://github.com/apache/auron/blob/2889b563139e6495495b61d0902e3f93f0ebe8bb/native-engine/datafusion-ext-plans/src/shuffle/buffered_data.rs#L156 # What changes are included in this PR? Output the compressed size when writing RSS. # Are there any user-facing changes? No # How was this patch tested? Production environment verification ``` 2026-07-22 20:10:07.090 (+80.707s) [INFO] [datafusion_ext_plans::shuffle::buffered_data:196] (stage: 0, partition: 1694, tid: 0) - all buffered data drained to rss, pushed_compressed=74.2 MiB, uncompressed_mem=528.1 MiB 2026-07-22 20:10:07.110 (+80.727s) [INFO] [datafusion_ext_plans::shuffle::buffered_data:196] (stage: 0, partition: 385, tid: 2) - all buffered data drained to rss, pushed_compressed=81.3 MiB, uncompressed_mem=531.6 MiB 2026-07-22 20:10:09.505 (+83.122s) [INFO] [datafusion_ext_plans::shuffle::buffered_data:196] (stage: 0, partition: 3824, tid: 1) - all buffered data drained to rss, pushed_compressed=80.0 MiB, uncompressed_mem=528.4 MiB ``` # Was this patch authored or co-authored using generative AI tooling? - [ ] Yes - [ ] No
<!-- - Start the PR title with the related issue ID, e.g. '[AURON #XXXX] Short summary...'. --> # Which issue does this PR close? Closes apache#1891 # Rationale for this change This improves function coverage in Auron by creating a native randn implementation. # What changes are included in this PR? Adds a native randn implementation. # Are there any user-facing changes? Yes, it adds the randn function. # How was this patch tested? Added unit tests and manually tested in spark-shell. ```scala import org.apache.spark.sql.functions.randn val df = spark.range(5) val outputPath = "/tmp/spark_range_output.parquet" df.write.mode("overwrite").parquet(outputPath) val readDf = spark.read.parquet(outputPath) val resultDf = readDf.withColumn("random_normal", randn(18)) resultDf.collect ``` Output: ``` 26/01/30 15:41:22 WARN NativeHelper: memory total: 1408.0 MiB, onheap: 1024.0 MiB, offheap: 384.0 MiB 26/01/30 15:41:24 WARN AuronCallNativeWrapper: Start executing native plan 26/01/30 15:41:24 WARN AuronCallNativeWrapper: Start executing native plan 26/01/30 15:41:24 WARN AuronCallNativeWrapper: Start executing native plan 26/01/30 15:41:24 WARN AuronCallNativeWrapper: Start executing native plan 26/01/30 15:41:24 WARN AuronCallNativeWrapper: Start executing native plan 26/01/30 15:41:24 WARN AuronCallNativeWrapper: Start executing native plan ------ initializing auron native environment ------ initializing logging with level: info 2026-01-30 15:41:24.368 (+0.000s) [INFO] [auron::exec:73] (stage: 0, partition: 0, tid: 0) - initializing JNI bridge 2026-01-30 15:41:24.369 (+0.001s) [INFO] [auron_jni_bridge::jni_bridge:473] (stage: 0, partition: 0, tid: 0) - Initializing JavaClasses... 2026-01-30 15:41:24.375 (+0.007s) [INFO] [auron_jni_bridge::jni_bridge:529] (stage: 0, partition: 0, tid: 0) - Initializing JavaClasses finished 2026-01-30 15:41:24.375 (+0.007s) [INFO] [auron::exec:77] (stage: 0, partition: 0, tid: 0) - initializing datafusion session 2026-01-30 15:41:24.375 (+0.007s) [INFO] [auron_memmgr:48] (stage: 0, partition: 0, tid: 0) - mem manager initialized with total memory: 230.4 MiB 2026-01-30 15:41:24.385 (+0.017s) [INFO] [auron::rt:146] (stage: 2, partition: 1, tid: 12) - start executing plan: ProjectExec [apache#3@0 AS apache#3, Randn(seed=18, partition=1) AS apache#5], schema=[apache#3:Int64;N, apache#5:Float64] RenameColumnsExec: ["apache#3"], schema=[apache#3:Int64;N] ParquetExec: limit=None, file_group=[FileGroup { files: [], statistics: None }, FileGroup { files: [PartitionedFile { object_meta: ObjectMeta { location: Path { raw: "ZmlsZTovLy90bXAvc3BhcmtfcmFuZ2Vfb3V0cHV0LnBhcnF1ZXQvcGFydC0wMDAwMS04ZTkwNmRiYS0zZDg3LTRkZWMtYjM0NC1hYjdiZWUyODEwZWQtYzAwMC5zbmFwcHkucGFycXVldA" }, last_modified: 1970-01-01T00:00:00Z, size: 472, e_tag: None, version: None }, partition_values: [], range: Some(FileRange { start: 0, end: 472 }), statistics: None, extensions: None, metadata_size_hint: None }], statistics: Some(Statistics { num_rows: Exact(0), total_byte_size: Exact(0), column_statistics: [ColumnStatistics { null_count: Absent, max_value: Absent, min_value: Absent, sum_value: Absent, distinct_count: Absent }] }) }, FileGroup { files: [], statistics: None }, FileGroup { files: [], statistics: None }, FileGroup { files: [], statistics: None }, FileGroup { files: [], statistics: None }], predicate=Some(Literal { value: Boolean(true), field: Field { name: "lit", data_type: Boolean, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} } }), schema=[id:Int64;N] 2026-01-30 15:41:24.385 (+0.017s) [INFO] [auron::rt:146] (stage: 2, partition: 5, tid: 16) - start executing plan: ProjectExec [apache#3@0 AS apache#3, Randn(seed=18, partition=5) AS apache#5], schema=[apache#3:Int64;N, apache#5:Float64] RenameColumnsExec: ["apache#3"], schema=[apache#3:Int64;N] ParquetExec: limit=None, file_group=[FileGroup { files: [], statistics: None }, FileGroup { files: [], statistics: None }, FileGroup { files: [], statistics: None }, FileGroup { files: [], statistics: None }, FileGroup { files: [], statistics: None }, FileGroup { files: [PartitionedFile { object_meta: ObjectMeta { location: Path { raw: "ZmlsZTovLy90bXAvc3BhcmtfcmFuZ2Vfb3V0cHV0LnBhcnF1ZXQvcGFydC0wMDAwMC04ZTkwNmRiYS0zZDg3LTRkZWMtYjM0NC1hYjdiZWUyODEwZWQtYzAwMC5zbmFwcHkucGFycXVldA" }, last_modified: 1970-01-01T00:00:00Z, size: 297, e_tag: None, version: None }, partition_values: [], range: Some(FileRange { start: 0, end: 297 }), statistics: None, extensions: None, metadata_size_hint: None }], statistics: Some(Statistics { num_rows: Exact(0), total_byte_size: Exact(0), column_statistics: [ColumnStatistics { null_count: Absent, max_value: Absent, min_value: Absent, sum_value: Absent, distinct_count: Absent }] }) }], predicate=Some(Literal { value: Boolean(true), field: Field { name: "lit", data_type: Boolean, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} } }), schema=[id:Int64;N] 2026-01-30 15:41:24.385 (+0.017s) [INFO] [auron::rt:146] (stage: 2, partition: 2, tid: 13) - start executing plan: ProjectExec [apache#3@0 AS apache#3, Randn(seed=18, partition=2) AS apache#5], schema=[apache#3:Int64;N, apache#5:Float64] RenameColumnsExec: ["apache#3"], schema=[apache#3:Int64;N] ParquetExec: limit=None, file_group=[FileGroup { files: [], statistics: None }, FileGroup { files: [], statistics: None }, FileGroup { files: [PartitionedFile { object_meta: ObjectMeta { location: Path { raw: "ZmlsZTovLy90bXAvc3BhcmtfcmFuZ2Vfb3V0cHV0LnBhcnF1ZXQvcGFydC0wMDAwMy04ZTkwNmRiYS0zZDg3LTRkZWMtYjM0NC1hYjdiZWUyODEwZWQtYzAwMC5zbmFwcHkucGFycXVldA" }, last_modified: 1970-01-01T00:00:00Z, size: 472, e_tag: None, version: None }, partition_values: [], range: Some(FileRange { start: 0, end: 472 }), statistics: None, extensions: None, metadata_size_hint: None }], statistics: Some(Statistics { num_rows: Exact(0), total_byte_size: Exact(0), column_statistics: [ColumnStatistics { null_count: Absent, max_value: Absent, min_value: Absent, sum_value: Absent, distinct_count: Absent }] }) }, FileGroup { files: [], statistics: None }, FileGroup { files: [], statistics: None }, FileGroup { files: [], statistics: None }], predicate=Some(Literal { value: Boolean(true), field: Field { name: "lit", data_type: Boolean, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} } }), schema=[id:Int64;N] 2026-01-30 15:41:24.385 (+0.017s) [INFO] [auron::rt:146] (stage: 2, partition: 4, tid: 15) - start executing plan: ProjectExec [apache#3@0 AS apache#3, Randn(seed=18, partition=4) AS apache#5], schema=[apache#3:Int64;N, apache#5:Float64] RenameColumnsExec: ["apache#3"], schema=[apache#3:Int64;N] ParquetExec: limit=None, file_group=[FileGroup { files: [], statistics: None }, FileGroup { files: [], statistics: None }, FileGroup { files: [], statistics: None }, FileGroup { files: [], statistics: None }, FileGroup { files: [PartitionedFile { object_meta: ObjectMeta { location: Path { raw: "ZmlsZTovLy90bXAvc3BhcmtfcmFuZ2Vfb3V0cHV0LnBhcnF1ZXQvcGFydC0wMDAwNS04ZTkwNmRiYS0zZDg3LTRkZWMtYjM0NC1hYjdiZWUyODEwZWQtYzAwMC5zbmFwcHkucGFycXVldA" }, last_modified: 1970-01-01T00:00:00Z, size: 471, e_tag: None, version: None }, partition_values: [], range: Some(FileRange { start: 0, end: 471 }), statistics: None, extensions: None, metadata_size_hint: None }], statistics: Some(Statistics { num_rows: Exact(0), total_byte_size: Exact(0), column_statistics: [ColumnStatistics { null_count: Absent, max_value: Absent, min_value: Absent, sum_value: Absent, distinct_count: Absent }] }) }, FileGroup { files: [], statistics: None }], predicate=Some(Literal { value: Boolean(true), field: Field { name: "lit", data_type: Boolean, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} } }), schema=[id:Int64;N] 2026-01-30 15:41:24.385 (+0.017s) [INFO] [auron::rt:146] (stage: 2, partition: 3, tid: 14) - start executing plan: ProjectExec [apache#3@0 AS apache#3, Randn(seed=18, partition=3) AS apache#5], schema=[apache#3:Int64;N, apache#5:Float64] RenameColumnsExec: ["apache#3"], schema=[apache#3:Int64;N] ParquetExec: limit=None, file_group=[FileGroup { files: [], statistics: None }, FileGroup { files: [], statistics: None }, FileGroup { files: [], statistics: None }, FileGroup { files: [PartitionedFile { object_meta: ObjectMeta { location: Path { raw: "ZmlsZTovLy90bXAvc3BhcmtfcmFuZ2Vfb3V0cHV0LnBhcnF1ZXQvcGFydC0wMDAwOS04ZTkwNmRiYS0zZDg3LTRkZWMtYjM0NC1hYjdiZWUyODEwZWQtYzAwMC5zbmFwcHkucGFycXVldA" }, last_modified: 1970-01-01T00:00:00Z, size: 472, e_tag: None, version: None }, partition_values: [], range: Some(FileRange { start: 0, end: 472 }), statistics: None, extensions: None, metadata_size_hint: None }], statistics: Some(Statistics { num_rows: Exact(0), total_byte_size: Exact(0), column_statistics: [ColumnStatistics { null_count: Absent, max_value: Absent, min_value: Absent, sum_value: Absent, distinct_count: Absent }] }) }, FileGroup { files: [], statistics: None }, FileGroup { files: [], statistics: None }], predicate=Some(Literal { value: Boolean(true), field: Field { name: "lit", data_type: Boolean, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} } }), schema=[id:Int64;N] 2026-01-30 15:41:24.385 (+0.017s) [INFO] [auron::rt:146] (stage: 2, partition: 0, tid: 11) - start executing plan: ProjectExec [apache#3@0 AS apache#3, Randn(seed=18, partition=0) AS apache#5], schema=[apache#3:Int64;N, apache#5:Float64] RenameColumnsExec: ["apache#3"], schema=[apache#3:Int64;N] ParquetExec: limit=None, file_group=[FileGroup { files: [PartitionedFile { object_meta: ObjectMeta { location: Path { raw: "ZmlsZTovLy90bXAvc3BhcmtfcmFuZ2Vfb3V0cHV0LnBhcnF1ZXQvcGFydC0wMDAwNy04ZTkwNmRiYS0zZDg3LTRkZWMtYjM0NC1hYjdiZWUyODEwZWQtYzAwMC5zbmFwcHkucGFycXVldA" }, last_modified: 1970-01-01T00:00:00Z, size: 472, e_tag: None, version: None }, partition_values: [], range: Some(FileRange { start: 0, end: 472 }), statistics: None, extensions: None, metadata_size_hint: None }], statistics: Some(Statistics { num_rows: Exact(0), total_byte_size: Exact(0), column_statistics: [ColumnStatistics { null_count: Absent, max_value: Absent, min_value: Absent, sum_value: Absent, distinct_count: Absent }] }) }, FileGroup { files: [], statistics: None }, FileGroup { files: [], statistics: None }, FileGroup { files: [], statistics: None }, FileGroup { files: [], statistics: None }, FileGroup { files: [], statistics: None }], predicate=Some(Literal { value: Boolean(true), field: Field { name: "lit", data_type: Boolean, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} } }), schema=[id:Int64;N] 2026-01-30 15:41:24.394 (+0.026s) [INFO] [datafusion_datasource_parquet::opener:421] (stage: 2, partition: 4, tid: 15) - executing parquet scan with adaptive batch size: 10000 2026-01-30 15:41:24.394 (+0.026s) [INFO] [datafusion_datasource_parquet::opener:421] (stage: 2, partition: 0, tid: 11) - executing parquet scan with adaptive batch size: 10000 2026-01-30 15:41:24.394 (+0.026s) [INFO] [datafusion_datasource_parquet::opener:421] (stage: 2, partition: 3, tid: 14) - executing parquet scan with adaptive batch size: 10000 2026-01-30 15:41:24.394 (+0.026s) [INFO] [datafusion_datasource_parquet::opener:421] (stage: 2, partition: 1, tid: 12) - executing parquet scan with adaptive batch size: 10000 2026-01-30 15:41:24.394 (+0.026s) [INFO] [datafusion_datasource_parquet::opener:421] (stage: 2, partition: 2, tid: 13) - executing parquet scan with adaptive batch size: 10000 2026-01-30 15:41:24.394 (+0.026s) [INFO] [datafusion_datasource_parquet::opener:421] (stage: 2, partition: 5, tid: 16) - executing parquet scan with adaptive batch size: 1 2026-01-30 15:41:24.488 (+0.120s) [INFO] [auron::rt:183] (stage: 2, partition: 5, tid: 16) - task finished 2026-01-30 15:41:24.488 (+0.120s) [INFO] [auron::rt:183] (stage: 2, partition: 0, tid: 11) - task finished 2026-01-30 15:41:24.488 (+0.120s) [INFO] [auron::rt:183] (stage: 2, partition: 4, tid: 15) - task finished 2026-01-30 15:41:24.488 (+0.120s) [INFO] [auron::rt:183] (stage: 2, partition: 3, tid: 14) - task finished 2026-01-30 15:41:24.488 (+0.120s) [INFO] [auron::rt:266] (stage: 0, partition: 0, tid: 0) - (partition=5) native execution finalizing 2026-01-30 15:41:24.488 (+0.120s) [INFO] [auron::rt:183] (stage: 2, partition: 2, tid: 13) - task finished 2026-01-30 15:41:24.488 (+0.120s) [INFO] [auron::rt:183] (stage: 2, partition: 1, tid: 12) - task finished 2026-01-30 15:41:24.488 (+0.120s) [INFO] [auron::rt:274] (stage: 0, partition: 0, tid: 0) - (partition=5) native execution finalized 2026-01-30 15:41:24.511 (+0.143s) [INFO] [auron::rt:266] (stage: 0, partition: 0, tid: 0) - (partition=3) native execution finalizing 2026-01-30 15:41:24.511 (+0.143s) [INFO] [auron::rt:266] (stage: 0, partition: 0, tid: 0) - (partition=4) native execution finalizing 2026-01-30 15:41:24.511 (+0.143s) [INFO] [auron::rt:266] (stage: 0, partition: 0, tid: 0) - (partition=0) native execution finalizing 2026-01-30 15:41:24.511 (+0.143s) [INFO] [auron::rt:266] (stage: 0, partition: 0, tid: 0) - (partition=2) native execution finalizing 2026-01-30 15:41:24.511 (+0.143s) [INFO] [auron::rt:266] (stage: 0, partition: 0, tid: 0) - (partition=1) native execution finalizing 2026-01-30 15:41:24.512 (+0.144s) [INFO] [auron::rt:274] (stage: 0, partition: 0, tid: 0) - (partition=0) native execution finalized 2026-01-30 15:41:24.512 (+0.144s) [INFO] [auron::rt:274] (stage: 0, partition: 0, tid: 0) - (partition=4) native execution finalized 2026-01-30 15:41:24.512 (+0.144s) [INFO] [auron::rt:274] (stage: 0, partition: 0, tid: 0) - (partition=1) native execution finalized 2026-01-30 15:41:24.512 (+0.144s) [INFO] [auron::rt:274] (stage: 0, partition: 0, tid: 0) - (partition=2) native execution finalized 2026-01-30 15:41:24.512 (+0.144s) [INFO] [auron::rt:274] (stage: 0, partition: 0, tid: 0) - (partition=3) native execution finalized import org.apache.spark.sql.functions.randn df: org.apache.spark.sql.Dataset[Long] = [id: bigint] outputPath: String = /tmp/spark_range_output.parquet readDf: org.apache.spark.sql.DataFrame = [id: bigint] resultDf: org.apache.spark.sql.DataFrame = [id: bigint, random_normal: double] res0: Array[org.apache.spark.sql.Row] = Array([3,1.4607292672705405], [0,-0.3268302897860617], [1,-0.09087682847007866], [4,-1.2271197538792842], [2,-0.546398027932835]) ``` --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…an data types (apache#2428) # Which issue does this PR close? Closes apache#2427 # Rationale for this change Auron Iceberg native scan can fall back when the Iceberg scan schema contains data types that are not supported by native scan conversion. Before this change, unsupported metadata columns already had a clear fallback reason, but unsupported data columns did not. This made it harder to diagnose why an Iceberg scan stayed on Spark execution. # What changes are included in this PR? Adds an Iceberg fallback reason for unsupported scan schema data types. Reports unsupported field names and Spark SQL types in the never-convert reason. Extends the existing unsupported decimal Iceberg scan test to check the fallback reason. # Are there any user-facing changes? No API changes. Users may see a clearer fallback reason when an Iceberg native scan is skipped because of unsupported data types. # How was this patch tested? UT. # Was this patch authored or co-authored using generative AI tooling? - [ ] Yes - [x] No Co-authored-by: Shilun Fan <slfan1989@apache.org> Signed-off-by: weimingdiit <weimingdiit@gmail.com>
apache#2426) **Which issue does this PR close?** Closes apache#2425 **Rationale for this change** Auron Iceberg native scan already collects Iceberg residual filters and converts supported filters into native scan pruning predicates. Iceberg can produce `STARTS_WITH` filters for prefix string predicates, such as `LIKE 'a%'`, but Auron currently does not convert this Iceberg operation. As a result, the query can still use a native post-scan filter, but the starts-with predicate is not used for native scan pruning. **What changes are included in this PR?** Adds Iceberg `STARTS_WITH` conversion for string columns in `IcebergScanSupport`. Reuses the existing Spark `StartsWith` native pruning conversion. Keeps other string and binary pruning behavior unchanged. Adds an Iceberg integration test for prefix LIKE pruning. **Are there any user-facing changes?** No user-facing API changes. More Iceberg prefix string filters can now be passed into native scan pruning. **How was this patch tested?** UT. Co-authored-by: Shilun Fan <slfan1989@apache.org> Signed-off-by: weimingdiit <weimingdiit@gmail.com>
…ample. (apache#2408) ### Which issue does this PR close? Closes apache#2407 ### Rationale for this change The Docker build example generated by `auron-build.sh --help` used Spark 4.1 together with the `centos7` image. The `centos7` image only provides JDK 8, while Spark 4.x requires JDK 17 or later. The example also enabled Iceberg and Hudi for Spark versions with which those integrations are not compatible. As a result, users following the documented example encountered Java version or integration compatibility errors before completing the build. ### What changes are included in this PR? - Replace the single Docker build example with explicit examples for Spark 3.5, Spark 4.0, and Spark 4.1. - Use the `rockylinux8` image for these examples because it supports the JDK selection required by Spark 4.x. - Use Scala 2.12 for Spark 3.5 and Scala 2.13 for Spark 4.x. - Include only compatible third-party integrations in each example: - Spark 3.5 includes Iceberg 1.10.1 and Hudi 0.15. - Spark 4.0 includes Iceberg 1.10.1 but excludes Hudi. - Spark 4.1 excludes both Iceberg and Hudi. - Add a build compatibility matrix to `CONTRIBUTING.md`. - Document the Spark, Scala, JDK, Docker image, Flink, Celeborn, Uniffle, Paimon, Iceberg, and Hudi combinations. - Update the existing Docker build example in `CONTRIBUTING.md` to use `rockylinux8`. - Document that the compatibility matrix and Shell help examples should be updated together. ### Are there any user-facing changes? Yes. The help output now provides separate Docker build examples for Spark 3.5, Spark 4.0, and Spark 4.1. Each example uses an appropriate Scala, JDK, Docker image, and third-party integration combination. Contributors can also consult the compatibility matrix in `CONTRIBUTING.md` before selecting build arguments. ### How was this patch tested? > ./auron-build.sh --help ``` Usage: ./auron-build.sh [OPTIONS] <maven build options> Build Auron project with specified Maven profiles Options: --pre Activate pre-release profile --release Activate release profile --clean <true|false> Clean before build (default: true) --skiptests <true|false> Skip unit tests (default: true) --sparktests <true|false> Run spark tests (default: false) --docker <true|false> Build in Docker environment (default: false) --threads <N|NC> Maven build threads (e.g. 1, 4, 1C). Default: local unset, docker 8 --image <NAME> Docker image to use (e.g. centos7,ubuntu24,rockylinux8,debian11,azurelinux3, default: centos7) --sparkver <VERSION> Specify Spark version (e.g. 3.0,3.1,3.2,3.3,3.4,3.5,4.0,4.1) --flinkver <VERSION> Specify Flink version (e.g. 1.18) --scalaver <VERSION> Specify Scala version (e.g. 2.12,2.13) --celeborn <VERSION> Specify Celeborn version (e.g. 0.5,0.6) --uniffle <VERSION> Specify Uniffle version (e.g. 0.10) --paimon <VERSION> Specify Paimon version (e.g. 1.2) --iceberg <VERSION> Specify Iceberg version (e.g. 1.10.1) --hudi <VERSION> Specify Hudi version (e.g. 0.15) -h, --help Show this help message Examples: ./auron-build.sh --pre --sparkver 4.1 --scalaver 2.13 -DskipBuildNative # Spark 3.5 with all supported third-party integrations ./auron-build.sh --docker true --image rockylinux8 --clean true --skiptests true --release --sparkver 3.5 --scalaver 2.12 --flinkver 1.18 --celeborn 0.6 --uniffle 0.10 --paimon 1.2 --iceberg 1.10.1 --hudi 0.15 # Spark 4.0 (Hudi is not supported) ./auron-build.sh --docker true --image rockylinux8 --clean true --skiptests true --release --sparkver 4.0 --scalaver 2.13 --flinkver 1.18 --celeborn 0.6 --uniffle 0.10 --paimon 1.2 --iceberg 1.10.1 # Spark 4.1 (Iceberg and Hudi are not supported) ./auron-build.sh --docker true --image rockylinux8 --clean true --skiptests true --release --sparkver 4.1 --scalaver 2.13 --flinkver 1.18 --celeborn 0.6 --uniffle 0.10 --paimon 1.2 ``` ### Was this patch authored or co-authored using generative AI tooling? - [ ] Yes - [x] No If yes, include: `Generated-by: <tool name and version>` ASF guidance: https://www.apache.org/legal/generative-tooling.html Signed-off-by: slfan1989 <slfan1989@apache.org>
…out safety (apache#2415) # Which issue does this PR close? Closes apache#2329 # Rationale for this change apache#1865 moved Calc-source fusion to a graph-level pass (`AuronOperatorFusionProcessor`) that counts each source's consumers and fuses only sole-consumer sources. A source feeding more than one Calc (for example `UNION ALL` over one table under Flink's default source reuse) is declined at the sole-consumer gate, and each Calc runs as a standalone native operator over the shared source stream. That multi-consumer path is correct but was under-tested and under-documented. The existing shared-source test only covered the default config, where object reuse is effectively off so Flink deep-copies each row to heap per consumer. The interesting case is object reuse ON: Flink then hands the same `AuronColumnarRowData` reference to both Calc consumers with no defensive copy in between, which is exactly where a fan-out aliasing or use-after-free bug would surface. Nothing pinned that. And the reason a multi-consumer source declines fusion was only a one-line comment. # What changes are included in this PR? A new end-to-end test `testSharedSourceUnionAllFanOutSafeWithObjectReuse` in `AuronKafkaSourceMergeITCase`: the same shared-source `UNION ALL` as the existing test, but with object reuse enabled. It asserts the row set stays correct and both Calcs remain standalone operators (the multi-consumer source is not fused). An expanded doc-comment at the sole-consumer gate in `AuronOperatorFusionProcessor` explaining why a multi-consumer source is not fused (the native source runtime is single-plan and single-output, so N per-consumer plans cannot share one source) and why declining is safe (each consumer runs a standalone native Calc that copies every column out of the shared columnar view). No production logic changes. # Are there any user-facing changes? No. # How was this patch tested? `./build/mvn test -Pspark-3.5 -Pscala-2.12 -Pflink-1.18 -pl auron-flink-extension/auron-flink-planner -am -Dtest=AuronKafkaSourceMergeITCase` with the native library built. All 4 tests pass. # Was this patch authored or co-authored using generative AI tooling? - [x] Yes （Claude Code (Opus 4.8)） - [ ] No
weiqingy
force-pushed
the
AURON-1863-converter
branch
2 times, most recently
from
July 27, 2026 23:09
e59b2b0 to
ce8a590
Compare
….sh with patch series (apache#2433) Replace direct git/rev dependencies for datafusion, arrow-rs, orc-rust, and serde_json with a vendor+patch approach: - Add dev/vendors/setup-vendors.sh to clone upstream repos at pinned tags and apply local patch series - Add patch files under dev/vendors/patches/{datafusion,arrow-rs,orc-rust,serde_json}/ - Update Cargo.toml: use relative path patches to vendored packages, exclude vendor workspaces from the main workspace - Update .gitignore to ignore dev/vendors/packages/ - Update .rat-excludes for dev/vendors/** - Call setup-vendors.sh all from build-native.sh before cargo build # Which issue does this PR close? Closes apache#2432 # Rationale for this change # What changes are included in this PR? # Are there any user-facing changes? No. # How was this patch tested? Via github CI. # Was this patch authored or co-authored using generative AI tooling? - [*] Yes - [ ] No If yes, include: `Generated-by: GLM-5.2` ASF guidance: https://www.apache.org/legal/generative-tooling.html Co-authored-by: diliulou <wanglei05@kuaishou.com>
Bumps [actions/stale](https://github.com/actions/stale) from 10 to 11. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/actions/stale/releases">actions/stale's releases</a>.</em></p> <blockquote> <h2>v11.0.0</h2> <h2>What's Changed</h2> <h3>Enhancement</h3> <ul> <li>Migrate to ESM and update dependencies by <a href="https://github-grid.enterprise.slack.com/team/U08CVLQ4JKE"><code>@chiranjib-swain</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1350">actions/stale#1350</a></li> </ul> <h3>Dependency Update</h3> <ul> <li>Override brace-expansion to 5.0.8 to address 24 high-severity dependency vulnerabilities by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1351">actions/stale#1351</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/actions/stale/compare/v10...v11.0.0">https://github.com/actions/stale/compare/v10...v11.0.0</a></p> <h2>v10.4.0</h2> <h2>What's Changed</h2> <h3>Bug Fix</h3> <ul> <li>Fixed <code>only-issue-types</code> validation by <a href="https://github.com/trueberryless"><code>@trueberryless</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1338">actions/stale#1338</a></li> </ul> <h3>Dependency Updates</h3> <ul> <li>Bump undici to 6.27.0 via override, clean up stale license files, and version to 10.4.0. by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1342">actions/stale#1342</a></li> </ul> <h2>New Contributors</h2> <ul> <li><a href="https://github.com/trueberryless"><code>@trueberryless</code></a> made their first contribution in <a href="https://redirect.github.com/actions/stale/pull/1338">actions/stale#1338</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/actions/stale/compare/v10.3.0...v10.4.0">https://github.com/actions/stale/compare/v10.3.0...v10.4.0</a></p> <h2>v10.3.0</h2> <h2>What's Changed</h2> <h3>Bug Fix</h3> <ul> <li>Enhancement: ignore stale labeling events by <a href="https://github.com/shamoon"><code>@shamoon</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1311">actions/stale#1311</a></li> </ul> <h3>Dependency Updates</h3> <ul> <li>Upgrade dependencies (<code>@actions/core</code>, <code>@octokit/plugin-retry</code>, <a href="https://github.com/typescript-eslint"><code>@typescript-eslint</code></a>) by <a href="https://github.com/Copilot"><code>@Copilot</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1335">actions/stale#1335</a></li> </ul> <h2>New Contributors</h2> <ul> <li><a href="https://github.com/shamoon"><code>@shamoon</code></a> made their first contribution in <a href="https://redirect.github.com/actions/stale/pull/1311">actions/stale#1311</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/actions/stale/compare/v10...v10.3.0">https://github.com/actions/stale/compare/v10...v10.3.0</a></p> <h2>v10.2.0</h2> <h2>What's Changed</h2> <h3>Bug Fix</h3> <ul> <li>Fix checking state cache (fix <a href="https://redirect.github.com/actions/stale/issues/1136">#1136</a>) and switch to Octokit helper methods by <a href="https://github.com/itchyny"><code>@itchyny</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1152">actions/stale#1152</a></li> </ul> <h3>Dependency Updates</h3> <ul> <li>Upgrade js-yaml from 4.1.0 to 4.1.1 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1304">actions/stale#1304</a></li> <li>Upgrade lodash from 4.17.21 to 4.17.23 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1313">actions/stale#1313</a></li> <li>Upgrade actions/cache from 4.0.3 to 5.0.2 and actions/github from 5.1.1 to 7.0.0 by <a href="https://github.com/chiranjib-swain"><code>@chiranjib-swain</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1312">actions/stale#1312</a></li> </ul> <h2>New Contributors</h2> <ul> <li><a href="https://github.com/itchyny"><code>@itchyny</code></a> made their first contribution in <a href="https://redirect.github.com/actions/stale/pull/1152">actions/stale#1152</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/actions/stale/compare/v10...v10.2.0">https://github.com/actions/stale/compare/v10...v10.2.0</a></p> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/actions/stale/blob/main/CHANGELOG.md">actions/stale's changelog</a>.</em></p> <blockquote> <h1>Changelog</h1> <h1>[10.1.0]</h1> <h2>What's Changed</h2> <ul> <li>Add only-issue-types option to filter issues by type by <a href="https://github.com/Bibo-Joshi"><code>@Bibo-Joshi</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1255">actions/stale#1255</a></li> </ul> <h1>[10.0.0]</h1> <h2>What's Changed</h2> <h2>Breaking Changes</h2> <ul> <li>Upgrade to node 24 by <a href="https://github.com/salmanmkc"><code>@salmanmkc</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1279">actions/stale#1279</a> Make sure your runner is on version v2.327.1 or later to ensure compatibility with this release. <a href="https://github.com/actions/runner/releases/tag/v2.327.1">Release Notes</a></li> </ul> <h2>Enhancement</h2> <ul> <li>Introducing sort-by option by <a href="https://github.com/suyashgaonkar"><code>@suyashgaonkar</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1254">actions/stale#1254</a></li> </ul> <h2>Dependency Upgrades</h2> <ul> <li>Upgrade actions/publish-immutable-action from 0.0.3 to 0.0.4 by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/actions/stale/pull/1186">actions/stale#1186</a></li> <li>Upgrade undici from 5.28.4 to 5.28.5 by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/actions/stale/pull/1201">actions/stale#1201</a></li> <li>Upgrade <code>@action/cache</code> from 4.0.0 to 4.0.2 by <a href="https://github.com/aparnajyothi-y"><code>@aparnajyothi-y</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1226">actions/stale#1226</a></li> <li>Upgrade <code>@action/cache</code> from 4.0.2 to 4.0.3 by <a href="https://github.com/suyashgaonkar"><code>@suyashgaonkar</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1233">actions/stale#1233</a></li> <li>Upgrade undici from 5.28.5 to 5.29.0 by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/actions/stale/pull/1251">actions/stale#1251</a></li> <li>Upgrade form-data to bring in fix for critical vulnerability by <a href="https://github.com/gowridurgad"><code>@gowridurgad</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1277">actions/stale#1277</a></li> </ul> <h2>Documentation changes</h2> <ul> <li>Changelog update for recent releases by <a href="https://github.com/suyashgaonkar"><code>@suyashgaonkar</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1224">actions/stale#1224</a></li> <li>Permissions update in Readme by <a href="https://github.com/ghadimir"><code>@ghadimir</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1248">actions/stale#1248</a></li> </ul> <h1>[9.1.0]</h1> <h2>What's Changed</h2> <ul> <li>Documentation update by <a href="https://github.com/Marukome0743"><code>@Marukome0743</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1116">actions/stale#1116</a></li> <li>Add workflow file for publishing releases to immutable action package by <a href="https://github.com/Jcambass"><code>@Jcambass</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1179">actions/stale#1179</a></li> <li>Update undici from 5.28.2 to 5.28.4 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1150">actions/stale#1150</a></li> <li>Update actions/checkout from 3 to 4 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1091">actions/stale#1091</a></li> <li>Update actions/publish-action from 0.2.2 to 0.3.0 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1147">actions/stale#1147</a></li> <li>Update ts-jest from 29.1.1 to 29.2.5 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1175">actions/stale#1175</a></li> <li>Update <code>@actions/core</code> from 1.10.1 to 1.11.1 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1191">actions/stale#1191</a></li> <li>Update <code>@types/jest</code> from 29.5.11 to 29.5.14 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1193">actions/stale#1193</a></li> <li>Update <code>@actions/cache</code> from 3.2.2 to 4.0.0 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1194">actions/stale#1194</a></li> </ul> <h1>[9.0.0]</h1> <h2>Breaking Changes</h2> <ol> <li>Action is now stateful: If the action ends because of <a href="https://github.com/actions/stale#operations-per-run">operations-per-run</a> then the next run will start from the first unprocessed issue skipping the issues processed during the previous run(s). The state is reset when all the issues are processed. This should be considered for scheduling workflow runs.</li> <li>Version 9 of this action updated the runtime to Node.js 20. All scripts are now run with Node.js 20 instead of Node.js 16 and are affected by any breaking changes between Node.js 16 and 20.</li> </ol> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/actions/stale/commit/4391f3da665fdf50b6810c1a66712fb9ba21aa93"><code>4391f3d</code></a> Fix 24 high severity vulnerabilities by overriding brace-expansion to 5.0.8 (...</li> <li><a href="https://github.com/actions/stale/commit/eaf9131fae5eafd0c31a64ebe3a2e183266fec48"><code>eaf9131</code></a> refactor: update imports to use ES module syntax and improve test structure (...</li> <li>See full diff in <a href="https://github.com/actions/stale/compare/v10...v11">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…che#2441) # Which issue does this PR close? Closes apache#2440 # Rationale for this change `./dev/reformat` produced different formatting results depending on the JDK used to run it. `pom.xml` selected `scalafmtVersion` based on the active JDK profile: `3.0.0` for `jdk-8`/`jdk-11`, and `3.9.9` for `jdk-17`/`jdk-21`. However, `scalafmt.conf` declares `version = 3.9.9`, so only the JDK 17/21 profiles actually matched the intended formatting rules. As a result, running the formatter under JDK 8/11 vs JDK 17/21 produced different indentation for multi-line string interpolations (e.g. `.getOrElse(...)` continuation lines), causing spurious diffs in `git status` depending on which JDK happened to be active locally. # What changes are included in this PR? Unify `scalafmtVersion` to `3.9.9` across all `jdk-8`/`jdk-11`/`jdk-17`/`jdk-21` profiles, matching the version declared in `scalafmt.conf`. # Are there any user-facing changes? No # How was this patch tested? GHA # Was this patch authored or co-authored using generative AI tooling? - [ ] Yes - [ ] No
# Which issue does this PR close? Closes apache#2439 # Rationale for this change https://github.com/apache/auron/actions/runs/30325123247/job/90213501717 ``` build-release-1 | [ERROR] [mvn-builder-auron-core] ./dev/vendors/setup-vendors.sh: line 135: git: command not found ``` # What changes are included in this PR? 1. `dev/docker-build/centos7/Dockerfile`: install `git` alongside the other build tools (it was previously missing entirely, causing `setup-vendors.sh` to fail with `git: command not found`). 2. `dev/vendors/setup-vendors.sh`: replace `git -C "${target_dir}" apply --3way "${patch}"` with `(cd "${target_dir}" && git apply --3way "${patch}")`, so patch application no longer depends on a git version >= 1.8.5. 3. `.github/workflows/build-amd64-releases.yml`: add `dev/docker-build/centos7/Dockerfile` to the `paths` trigger list so changes to the build image are validated by this workflow on PRs. # Are there any user-facing changes? No # How was this patch tested? GHA # Was this patch authored or co-authored using generative AI tooling? - [ ] Yes - [ ] No
…complex Arrow row types (apache#2424) # Which issue does this PR close? Closes apache#2419 **Rationale for this change** Auron's sort merge join uses encoded Arrow rows for join key comparison and uses RowNullChecker to skip rows with null join keys. RowNullChecker previously inspected Arrow row bytes manually. This is fragile for complex row encodings such as struct, list, and dictionary, because their layout is controlled by Arrow RowConverter and should not be interpreted with fixed or estimated field lengths. This can cause RowNullChecker to mis-detect null join keys for complex key types. **What changes are included in this PR?** Adds an Arrow RowConverter to RowNullChecker. Updates RowNullChecker's `has_nulls` path to decode Rows through Arrow RowConverter before building the null-key mask. Preserves conservative join key null semantics: top-level null key values are treated as null join keys nested null fields or elements inside non-null struct/list keys do not make the key null Keeps the existing RowNullChecker public method signatures unchanged. Keeps the existing byte-level helper path unchanged. Adds unit coverage for complex struct, list, and dictionary key rows. **Are there any user-facing changes?** No user-facing API changes. **How was this patch tested?** UT. **Was this patch authored or co-authored using generative AI tooling?** - [x] Yes - [ ] No Generated-by: OpenAI Codex (GPT-5) --------- Co-authored-by: Shilun Fan <slfan1989@apache.org> Signed-off-by: weimingdiit <weimingdiit@gmail.com>
…angelog native scan (apache#2430) **Which issue does this PR close?** Closes apache#2429 **Rationale for this change** Auron native Iceberg changelog scan requires all planned changelog tasks to use a single supported file format. Without validating the planned tasks, a mixed-format fallback test could pass because of an unrelated changelog fallback and fail to exercise the intended file-format guard. **What changes are included in this PR?** Adds fallback coverage for an Iceberg changelog range containing both Parquet and ORC data files. The test: - verifies that all planned changelog tasks are `AddedRowsScanTask` - verifies that the task file formats are exactly Parquet and ORC - compares the result with Spark execution - verifies that `NativeIcebergTableScan` is not used No execution logic is changed. **Are there any user-facing changes?** No user-facing changes. This is a test-only change. **How was this patch tested?** UT. --------- Co-authored-by: Shilun Fan <slfan1989@apache.org> Signed-off-by: weimingdiit <weimingdiit@gmail.com>
# Which issue does this PR close? Closes apache#2446 this PR is to fix the incorrect flushing logic, that should be based on the row count rather than the bytes # Rationale for this change Without this fix, the flushing will be triggered too frequently. # What changes are included in this PR? # Are there any user-facing changes? # How was this patch tested? # Was this patch authored or co-authored using generative AI tooling? - [x] Yes - [ ] No Generated-by: OpenAI Codex (GPT-5)
weiqingy
force-pushed
the
AURON-1863-converter
branch
from
August 3, 2026 20:04
ce8a590 to
829ac34
Compare
…he#2423) # Which issue does this PR close? Closes apache#2422 # Rationale for this change Configurable spill trigger (rss_sort_repartitioner.rs). Replace the hardcoded `mem_used_percent() > 0.4` with two tunable thresholds; spill fires when either is exceeded (whichever first): - `spark.auron.rss.spill.memoryFraction` (relative, default 0.4 = previous behavior) - `spark.auron.rss.spill.memorySize` (absolute bytes, default 0 = disabled; acts as a deterministic cap on the per-spill burst regardless of executor memory) Both are read once per executor via OnceCell (no per-batch JNI conf reads). Adds `MemConsumer::mem_used()` (absolute bytes accessor) to support the absolute threshold. # What changes are included in this PR? Auron's rss shuffle buffers records in native memory and spills to celeborn in bursts. At a shuffle-stage peak, many tasks spill simultaneously; if each spill is large, the aggregate burst can overrun a celeborn worker's inbound capacity and push the worker into a push-pause: it stops reading, holds inbound buffers, and only recovers after the idle-connection timeout / connection close (observed as a multi-minute stall — tasks survive via celeborn's client-side revive but throughput degrades). The previous hardcoded 0.4 fraction made the per-spill burst size depend on executor memory and not directly tunable against this. # Are there any user-facing changes? No # How was this patch tested? Production environment verification # Was this patch authored or co-authored using generative AI tooling? - [x] Yes - [ ] No
…r function (apache#2409) # Which issue does this PR close? Closes apache#1863. The converter that emits this function follows in a separate PR, once this one merges. To show where this is going, the converter change is previewable here: #1 (a draft in my fork, diffed against this PR's branch so it shows only the converter side). That PR will be opened against this repo after this one lands. # Rationale for this change Flink's `UNIX_TIMESTAMP` converts a formatted date-time string to a Unix timestamp in seconds. AIP-1 lists it as a Phase 1 built-in function. Supporting it natively is the first step; this PR adds the native (Rust) evaluation, and a follow-up PR wires the Flink converter to emit it. The hard part is matching Flink's parsing exactly. Flink parses with `java.text.SimpleDateFormat` in its default lenient mode, so it accepts non-zero-padded fields, field rollover, and trailing input. A strict parser diverges from Flink on the default format `yyyy-MM-dd HH:mm:ss` as soon as a field is not zero-padded, and would return a wrong timestamp rather than an error. So this implements a lenient parser rather than delegating to a strict one. # What changes are included in this PR? A new `Flink_UnixTimestamp` ext scalar function in `datafusion-ext-functions`, registered through the existing ext-function path (no proto change). It takes three string arguments `[value, format, zoneId]` and returns an `Int64`. The format is a translated strftime-style pattern and the zone is resolved by the caller; the converter PR supplies both. The parser reproduces `SimpleDateFormat` lenient semantics for the supported fields: variable field widths, rollover normalization, tolerant of trailing input, a leading minus on numeric fields. It also handles the Julian/Gregorian hybrid calendar (cutover 1582-10-15), since `GregorianCalendar` is a hybrid while chrono is proleptic Gregorian, and resolves DST-ambiguous or nonexistent local times using the zone's standard offset, matching Flink. Failure semantics match Flink: an unparseable string yields `Long.MIN_VALUE`, a NULL input yields NULL, and the milliseconds-to-seconds step truncates toward zero. The function is registered but not yet emitted by any planner, so this PR is self-contained and dormant until the converter lands. # Are there any user-facing changes? No. The function is not reachable from SQL until the converter PR is merged. # How was this patch tested? 19 unit tests derived from a differential oracle that was validated against real `java.text.SimpleDateFormat` across a large randomized set of (input, format, timezone) inputs. Coverage includes the happy path, field rollover, non-padded fields, trailing garbage, NULL, unparseable-to-`Long.MIN_VALUE`, DST gap and overlap across several zones, pre-1970 negative timestamps, pre-1582 hybrid-calendar dates, the arity guard, an empty format matching nothing, and canonical field widths.
…pache#2435) **Which issue does this PR close?** Closes apache#2434 **Rationale for this change** Auron native Iceberg changelog scan currently supports insert-only changelog tasks. Full-data-file delete changelog tasks can be executed natively without row-level delete-file handling. Auron can scan the deleted data file directly and materialize changelog metadata from the Iceberg changelog task. Supporting this case improves native Iceberg changelog scan coverage while keeping more complex delete and update semantics on Spark's reader. **What changes are included in this PR?** Allows native Iceberg changelog scan to accept `DeletedDataFileScanTask` when: - the changelog operation is `DELETE` - `existingDeletes()` is empty Keeps existing native support for `AddedRowsScanTask` when: - the changelog operation is `INSERT` - `deletes()` is empty Keeps fallback behavior for unsupported changelog tasks, including row-level deletes, position/equality deletes, update changelog tasks, mixed file formats, and delete tasks with existing delete files. Adds coverage for: - full-data-file delete changelog native scan - a changelog range that contains both insert tasks and full-data-file delete tasks **Are there any user-facing changes?** No user-facing API changes. More Iceberg changelog scans can now be executed natively by Auron. **How was this patch tested?** UT. --------- Co-authored-by: Shilun Fan <slfan1989@apache.org> Signed-off-by: weimingdiit <weimingdiit@gmail.com>
…tegration Wire Flink's UNIX_TIMESTAMP (1-arg and 2-arg forms) to the native Flink_UnixTimestamp function. The converter matches the operator by reference identity (as TRY_CAST does), translates the supported subset of Java date-format patterns to the native format, resolves the session time zone at plan time, and builds the native scalar-function node. Formats outside the supported subset, non-literal format arguments, the 0-argument form, and format patterns whose lenient parse cannot be represented after translation all fall back to Flink's engine. The 0-argument form is rejected explicitly at both the gate and the builder, so a call that parses nothing and reads the wall clock per record can never reach a native function that expects a value operand to size its output against. Also propagate the effective node config (carrying table.local-time-zone) into the standalone-Calc converter path; the persisted config it previously used does not carry the session time zone, which would silently default the native evaluation to UTC.
weiqingy
force-pushed
the
AURON-1863-converter
branch
from
August 5, 2026 04:46
829ac34 to
2040a2b
Compare
github-actions
Bot
added
documentation
Owner
Author
|
Superseded by the real PR against apache/auron: apache#2448 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
10 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Part of apache#1863.
Depends on the native
Flink_UnixTimestampfunction (the companion PR). It should be merged after that one, otherwise the native build has no matching function.Rationale for this change
This completes native support for Flink's
UNIX_TIMESTAMPby wiring the Flink Calc converter to emit the native function added in the companion PR. Without it, any Calc containingUNIX_TIMESTAMPfalls back to Flink's engine for the whole Calc.What changes are included in this PR?
The converter recognizes
UNIX_TIMESTAMPand lowers the 1-argument and 2-argument forms to the nativeFlink_UnixTimestampnode.UNIX_TIMESTAMPresolves toSqlKind.OTHER_FUNCTION, so it is matched by reference identity on the operator before the supported-kinds switch, the same wayTRY_CASTis handled.A format scanner translates the supported subset of Java date-format letters (
yyyy MM dd HH mm ssand literals) to the native format. Anything outside that subset falls back: other pattern letters, unsupported run-lengths, a non-literal format argument, the 0-argument form, and a numeric field adjacent to another numeric field where the run-length would not survive translation (for exampleyyyyMd). Falling back keeps results correct rather than risking a silent divergence.The session time zone is resolved at plan time and passed into the node. This required completing the config threading in the standalone-Calc path, which passed a persisted config that does not carry
table.local-time-zone. That gap had no effect until now:UNIX_TIMESTAMPis the first time-zone-sensitive expression the converter supports, and the earlier ones (arithmetic, comparison, logical, cast) never read the session zone. The effective node config is threaded through instead, so the configured zone reaches the native evaluation.Are there any user-facing changes?
Yes.
UNIX_TIMESTAMP(string)andUNIX_TIMESTAMP(string, format)now execute on the native engine when the format is a supported literal pattern. Unsupported patterns and the 0-argument form continue to run on Flink's engine, with the same results as before.How was this patch tested?
Unit tests for the scanner (accept/reject, quote escaping, the adjacency rule), the converter (node shape, format translation, time-zone propagation), and the operator-identity invariant. Fallback tests assert that unsupported inputs actually fall back rather than silently producing a native plan.
An end-to-end ITCase runs
UNIX_TIMESTAMP(ts)with a non-UTC session zone and confirms the native result matches the expected epoch values. The executed native plan shows the function and the resolved zone, confirming the query runs natively rather than falling back.