How to automatically build a valid pipeline when users can insert arbitrary modules that create, rename, transform, or derive variables?

[RBhupi]

Currently, we bind modules to concrete variable names.

SegmentationModule
inputs:
    dbz
outputs:
    labels

works until a user inserts:

AttenuationCorrection
and produces:
dbz_corrected

Now segmentation breaks or still use dbz instead of dbz_corrected.

Introduce Data Contracts

Instead of:

outputs = {
    "dbz": ndarray
}

use:

outputs = {
    ReflectivityField: "dbz"
}

or:

Variable(
    semantic_type="reflectivity",
    variable_name="dbz"
)

Then a module can consume:

ReflectivityField

regardless of actual variable name.

[krotname]

I think this fits Adapt's current design, but I would add one resolution layer before GraphBuilder, not replace the graph executor itself.

Right now the graph is purely key-based: GraphBuilder maps each declared output key to one producer, then wires modules whose inputs contain that key. It also rejects duplicate output keys. That is good for a resolved DAG, but it is too late for semantic choices like “use corrected reflectivity if present, otherwise raw reflectivity”.

A practical shape would be:

@dataclass(frozen=True)
class DataSlot:
    semantic_type: str       # reflectivity, segmentation_mask, projection, etc.
    role: str = "primary"     # raw, corrected, filtered, derived
    dims: tuple[str, ...] = ()
    variable_name: str | None = None

Then modules declare something like:

requires = {"reflectivity": DataSlot("reflectivity", role="corrected")}
provides = {"dbz_corrected": DataSlot("reflectivity", role="corrected", variable_name="dbz_corrected")}

The planner/resolver would run before GraphBuilder and produce the concrete Adapt-style declarations:

inputs = ["dbz_corrected"]
outputs = ["segmented_ds"]

So the existing BaseModule.inputs, BaseModule.outputs, input_contracts, output_contracts, and GraphExecutor can stay simple. The new part is only responsible for choosing which concrete key satisfies each semantic slot.

For the dbz / dbz_corrected case, I would avoid making downstream modules know every possible variable name. Let AttenuationCorrection provide a semantic reflectivity:corrected slot, then let SegmentationModule ask for reflectivity with a preference order like:

corrected > raw

If corrected exists, the resolver binds segmentation to dbz_corrected; otherwise it binds to dbz. If both exist and no preference is declared, fail fast with an ambiguity error rather than silently choosing one.

That also matches the current contract system: contracts should validate the selected artifact after resolution, while semantic selection should happen before DAG construction.