keying subset splits to identify test set

[ethane4]

Summary

It would be helpful to have some method for keying collections of splits ex. subset.slices['target_slice'] currently returns a list with no information on which slice is the test slice

Motivation

When we are doing anything other than k-fold splitting, we currently have no method for enforcing which split index should be dedicated to train. To avoid downstream issues, it would also be convenient to just key the split of interest from subset.slices['target_slice'] to avoid ambiguity

Proposed solution

• First proposed solution: we add a name attribute to DatasetSlice that is either None in the case of k-fold CV splits, or train/val/test if a more principled split is being used. Users/downstream pipelines can check the names of all DatasetSlices in a given list of splits and act accordingly. (I think this would involve very minimal changes to pg-base, the generic pg-benchmark entrypoint, and dvc)

• Second proposed solution: we swap the datatype of subset.slices['target_slice'] from list to dict. The dict keys are either integers in the k-fold split case, or 'train' 'test' 'val' in any other case. This would probably be cleaner for validation ex. checking there aren't more than one of each split type etc. but may require some changes to the split assignment functions in pg-base

[karel-w]

Since we use fractions to control size of slices in all our non-kfold splitting methods, we could create a names argument and validate len(name) == len(fractions). This would allow us to name splitting styles and get the specific slices of that split style.

E.g.

subsets = Subsets(dataset=dataset)
random_classic = RandomSplitter(fractions=[0.8, 0.1, 0.1], names=["train", "valid", "test"])
subsets.update(classic=random_classic.split(dataset=dataset))
train_data = subsets["classic"].get_slice("train")
valid_data = subsets["classic"].get_slice("valid")
test_data = subsets["classic"].get_slice("test")

def split(self, dataset: Dataset, ...) -> Subsets:
    if self.names is not None:
        slices_dict = {name: [slc] for name, slc in zip(self.names, slices)}
        return Subsets(dataset=dataset, slices=slices_dict)
    else:
        return Subsets(dataset=dataset, slices=slices)

class Subsets:
    def get_slice(self, name: str) -> Dataset:
          if not isinstance(self.slices, dict):
              raise TypeError("Subsets slices are not stored as a dictionary.")
          if name not in self.slices:
              raise KeyError(f"Slice name '{name}' not found. Available: {list(self.slices.keys())}")
          slice_list = self.slices[name]
          if len(slice_list) != 1:
              raise ValueError(
                  f"Expected slice '{name}' to contain exactly 1 element, "
                  f"got {len(slice_list)}"
              )
          return self.dataset[slice_list[0]]

[ethane4]

@florisvdf @hredestig any opinions on Karel's response?

[florisvdf]

Since we use fractions to control size of slices in all our non-kfold splitting methods, we could create a names argument and validate len(name) == len(fractions). This would allow us to name splitting styles and get the specific slices of that split style.

This exact pattern won't work for splits without a fractions argument, like the QuantileSplitter, though such splits could have their own validation logic for named slices.

@karel-w it seems like you prefer Ethan's second proposed option? Why not option one? In that case we wouldn't even need the type of validation you propose. Option 1 seems the be the least intrusive option to me as we can still index the slices, which we can't anymore when we change the datatype to a dict. We could also optionally create an enum to validate that a slice's name is always either None, train, validation, or test.

[hredestig]

I prefer option 1 since

• It's common pattern to do

  train, test, val = get_splits(..)
  

  which wouldn't work with a mapping
• I don't like the idea to make the naming mandatory

But then also, if not mandatory, you can't expect the subset.slices['target_slice'] to actually work unless for the cases when you created the splits yourself, e.g. as part of your current project. For a saved dataset, you'll need to deal with missing keys. Which maybe is fine - but also pointer that the StrEnum referred to above maybe a good idea. And then perhaps change the "name" concept to "type" or "purpose".