feat: add Hudi contrib build gate

[wirybeaver]

Which issue does this PR close?

Part of #1241.

Rationale for this change

Ballista already supports replacing its logical extension codec, so this PR no longer adds another format-neutral codec seam. Instead, it follows the contrib build-gate pattern used by Comet: establish a narrow Hudi wire boundary without adding Hudi to Ballista's dependency or release graph.

What changes are included?

• Adds the opt-in contrib-hudi Cargo feature to the ballista crate.
• Defines the typed Hudi provider payload (base_uri and caller-supplied options).
• Defines HudiProviderCodec, implemented by an integration outside Ballista's release graph.
• Adds HudiLogicalExtensionCodec, which handles only the Hudi provider payload and delegates every other logical node, provider, and file format to Ballista's existing codec.
• Adds no hudi-rs dependency and changes no default runtime behavior.

The follow-up #2389 implements the boundary with hudi-rs in an independently built crate excluded from the main workspace.

How are these changes tested?

• cargo check -p ballista --no-default-features --locked
• cargo check -p ballista --no-default-features --features contrib-hudi --locked
• cargo clippy -p ballista --lib --features contrib-hudi --locked -- -D warnings
• cargo package -p ballista --allow-dirty --no-verify
• Verified with cargo tree that neither the default build nor the contrib-hudi gate contains a Hudi crate.
• cargo fmt --all -- --check
• git diff --check

No runtime Hudi test is added here because this PR deliberately contains no Hudi implementation.

[milenkovicm]

I'm not sure that i understand why is this needed in ballista main ?
ballista codec is fully pluggable at the moment, implementers can plug in its own implementation hence i dont see this needed in main, please correct me if i'm wrong

[wirybeaver]

Thanks — you are right that Ballista’s codec is already fully pluggable. I reworked the PR and removed the additional format-neutral codec seam. It now only adds an opt-in Hudi-specific build gate on top of the existing codec override: a typed provider payload plus an implementation hook. There is no hudi-rs dependency or default runtime change, and cargo package still succeeds. The hudi-rs proof remains in #2389 as an independently built crate outside the workspace and root lockfile.

[milenkovicm]

I'm still failing to understand why we need to bring this integration to ballista ?

PR depends on unmaintained, private fork of hudi, hence my comments are same like last time, i would be reluctant to merge this to ballista main

[wirybeaver]

Thank you for taking another look. I think I caused confusion by using Hudi as the first example, so I want to clarify the boundary precisely.

#2388 has no dependency on hudi-rs, my fork, or any Hudi commit SHA. The only new dependencies behind contrib-hudi are datafusion-proto and prost, both already part of the Ballista/DataFusion ecosystem. The SHA shown on the review is the commit SHA of this PR itself, not a dependency pin.

#2388: Ballista build/release graph

+------------------------------------------------------+
| ballista crate                                       |
|                                                      |
| default build                                        |
|   - no hudi-rs dependency                            |
|   - no Hudi runtime path                             |
|                                                      |
| optional contrib-hudi build gate                     |
|   - wire descriptor                                  |
|   - provider codec interface                         |
|   - still no hudi-rs dependency                      |
+--------------------------+---------------------------+
                           |
                           | implemented externally
                           | only when explicitly built
                           v
+------------------------------------------------------+
| Separate format integration crate                    |
|   - owns the format SDK dependency                   |
|   - excluded from Ballista's workspace/release graph |
+------------------------------------------------------+

The broader proposal is a convention rather than special treatment for Hudi:

Ballista main crate: stable gates/SPIs, no table-format SDKs
        |
        +-- Apache Iceberg integration crate  -> Iceberg dependencies
        +-- Apache Delta integration crate    -> Delta dependencies
        +-- Apache Lance integration crate    -> Lance dependencies
        +-- Apache Hudi integration crate     -> Hudi dependencies

Each format would own a separate crate under the Apache project umbrella and would not add its SDK to the main Ballista crate, root lockfile, or release graph. This is intended to limit the release and dependency impact of third-party open-table-format integrations.

I chose hudi-rs only as a small validation vehicle because Iceberg and Delta Lake integrations are already being explored, but those efforts do not currently demonstrate this isolation convention. #2388 itself does not require accepting #2389. If it would make the design easier to evaluate independently of Hudi, I would be happy to provide Lance as a second example.