Align native UDF builds with jar dependency pins

[nvliyuan]

Summary

• Resolve spark-rapids-jni and cuDF revisions from the selected rapids-4-spark jar, then fetch the matching cudf-pins and rapids-cmake entrypoint.
• Build native UDF examples against jar-matched cuDF headers, RMM, CCCL, and rapids-cmake pins instead of moving branches.
• Apply the downloaded versions.json through rapids_cpm_init(OVERRIDE ...), so RMM/CCCL pins are actually enforced.
• Fail clearly when prebuilt mode lacks resolved pin metadata and use safe defaults for source builds.

This prevents release-bump gaps such as CI build examples-udf-examples-native/628, where the runtime jar was cuDF/RMM 26.08 while rapids-cmake main had already moved to RMM 26.10.

Closes #630

Test plan

• bash -n for resolve-jni-cudf-pins.sh, clone-cudf-repo.sh, and extract-cudf-libs.sh.
• shellcheck for the native build scripts.
• xmllint --noout pom.xml and git diff --check.
• On spark-yuanli, built the native UDF package with a 26.08 snapshot jar. The resolver selected the jar-recorded revisions instead of current main:
  • cuDF 4ae9d07c30e41a77d6bc6046eeee58873ba12aac
  • spark-rapids-jni 547370edfbd2d5cde9b08c496315050950de9ae9
  • rapids-cmake 74a28545e739bc8341cd002c9dae98e03b28939b
  • RMM a66679d671694a1cd9b3965fbc9eccdd44efa2e4
• Compared all native UDF cuDF undefined symbols against the jar-extracted libcudf.so: 10 required, 0 missing; cudf::copy_bitmask matched exactly.
• Ran a JVM smoke test using the packaged UDF jar plus the same 26.08 plugin jar: native udf completed rows=3.

[nvliyuan]

verified worked in local

[greptile-apps]

Greptile Summary

This PR replaces the previous branch-based native dependency resolution with a jar-anchored pin system. Instead of tracking a moving branch (e.g. rapids-cmake main) that can race ahead of the runtime jar, the build now reads spark-rapids-jni and cudf-java revision metadata embedded in the selected rapids-4-spark jar, downloads matching cudf-pins/versions.json and rapids-cmake SHA, and feeds those pins to CMake via rapids_cpm_init(OVERRIDE ...).

• resolve-jni-cudf-pins.sh (new): extracts revision metadata from the jar using Python's zipfile, downloads the matching versions.json and RAPIDS.cmake entrypoint, validates the pins, and writes jar-native-deps.properties.
• clone-cudf-repo.sh: generalised from branch-only to arbitrary git ref (commit SHA, tag, branch) using a blobless partial clone followed by a targeted shallow fetch.
• pom.xml / CMakeLists.txt: wire the resolved jar.rapids.cmake.sha, jar.rapids.cmake.file, and jar.cudf.pins.file properties into the cmake invocation so CCCL and RMM are fetched at the jar-matched versions.

Confidence Score: 4/5

Safe to merge once the stale cudf.git.branch reference in the failure message is corrected; the core pin-resolution and override wiring are working and verified end-to-end.

The cudf.headers.exist fail message in pom.xml still references ${cudf.git.branch} and directs users to -Dcudf.git.branch=main, but the ref now comes from ${jar.cudf.revision}. A user who hits a clone failure gets incorrect diagnostic advice pointing to a property that no longer controls the ref.

pom.xml — the cudf.headers.exist failure message at line 444 needs its ${cudf.git.branch} references replaced with ${jar.cudf.revision}.

Important Files Changed

Filename	Overview
examples/UDF-Examples/RAPIDS-accelerated-UDFs/resolve-jni-cudf-pins.sh	New script that reads spark-rapids-jni and cudf-java revision metadata from the rapids-4-spark jar (via Python/zipfile), downloads matching cudf-pins versions.json and rapids-cmake sha, validates the pins JSON, and writes a jar-native-deps.properties file; logic is sound and python3 requirement is checked upfront
examples/UDF-Examples/RAPIDS-accelerated-UDFs/clone-cudf-repo.sh	Generalised from branch-only to arbitrary git-ref (commit SHA, tag, branch) using blobless partial clone + shallow fetch; update path always fetches from origin even when the local ref already matches, causing unnecessary network traffic on every rebuild
examples/UDF-Examples/RAPIDS-accelerated-UDFs/pom.xml	Adds resolve-jni-cudf-pins.sh execution, loads resolved properties, and passes jar-matched rapids-cmake SHA and versions.json override to cmake; the cudf.headers.exist failure message still references ${cudf.git.branch} and suggests -Dcudf.git.branch=main, which is incorrect now that the ref comes from ${jar.cudf.revision}
examples/UDF-Examples/RAPIDS-accelerated-UDFs/src/main/cpp/CMakeLists.txt	Adds RAPIDS_CMAKE_BRANCH/RAPIDS_CMAKE_FILE/RAPIDS_CMAKE_CPM_OVERRIDE_VERSION_FILE cache variables and applies OVERRIDE to rapids_cpm_init(); SHA-detection block (lines 27-33) sets three variables that are never read downstream, leaving dead code
examples/UDF-Examples/RAPIDS-accelerated-UDFs/extract-cudf-libs.sh	Adds RAPIDS_JAR_PATH override, TARGET_DIR env-var override, delegates to resolve-jni-cudf-pins.sh for pin resolution, and passes the resolved commit SHA to clone-cudf-repo.sh; changes are straightforward and correct
examples/UDF-Examples/RAPIDS-accelerated-UDFs/README.md	Documentation updated to reflect pin-resolution flow and new target directories to clean; accurate and consistent with the implementation

Sequence Diagram

sequenceDiagram
    participant M as Maven (pom.xml)
    participant R as resolve-jni-cudf-pins.sh
    participant J as rapids-4-spark.jar
    participant GH as GitHub (raw)
    participant C as clone-cudf-repo.sh
    participant CM as CMakeLists.txt

    M->>J: unzip libcudf.so / libnvcomp.so
    M->>R: resolve-jni-cudf-pins.sh jar pins-dir props
    R->>J: read spark-rapids-jni-version-info.properties
    R->>J: read cudf-java-version-info.properties
    R->>GH: "download cudf-pins/versions.json @ JNI revision"
    R->>GH: "download cudf-pins/rapids-cmake.sha @ JNI revision"
    R->>GH: "download RAPIDS.cmake @ rapids-cmake SHA"
    R-->>M: jar-native-deps.properties
    M->>C: clone-cudf-repo.sh cudf-dir jar.cudf.revision
    C->>GH: "git clone filter=blob:none + fetch depth 1 SHA"
    M->>CM: cmake -DRAPIDS_CMAKE_FILE -DRAPIDS_CMAKE_CPM_OVERRIDE_VERSION_FILE
    CM->>CM: rapids_cpm_init(OVERRIDE versions.json)
    CM->>CM: rapids_cpm_cccl() rapids_cpm_rmm() at pinned versions
    CM->>CM: link udfexamplesjni to cudf_imported

Loading

_{Reviews (2): Last reviewed commit: "Apply jar-matched native dependency pins" | Re-trigger Greptile}

[greptile-apps]

awk fallback truncates property values containing =

The unzip fallback uses -F= and prints only $2, so a property value containing = is silently truncated. The url property is extracted this way; a query-string = in the JNI repository URL would produce a wrong RAW_BASE and silently download from an incorrect path.

[nvliyuan]

Updated on top of latest main and addressed the prior review findings:

• RAPIDS_CMAKE_CPM_OVERRIDE_VERSION_FILE is now passed to rapids_cpm_init(OVERRIDE ...), so jar-matched RMM/CCCL pins are enforced rather than only validated.
• The resolver now consistently requires Python 3 for both jar property parsing and JSON validation, removing the inconsistent unzip fallback.
• Python's partition(\"=\") preserves property values containing additional = characters, eliminating the awk truncation path.
• Prebuilt builds now fail with a clear message if pin metadata is missing; source builds receive safe defaults.

Remote validation with a 26.08 snapshot jar completed successfully: native package built, all 10 required cuDF symbols were exported by the jar's libcudf.so (0 missing), copy_bitmask matched exactly, and the packaged native UDF smoke test returned native udf completed rows=3.

[nvliyuan]

@GaryShen2008 can you please help review? thx