[FEATURE] add a cross_val_predict copy that works with TimebaseCV

[joshdunnlime]

TimebasedCV is great. I use it a lot, however it doesn't play nicely with cross_val_predict and so I find myself rewriting for-loops, gathering data in DataFrames etc. See this common issue on SO: scikit-learn-cross-val-predict-only-works-for-partitions. This slows quick model implementation, checks etc.

I also find myself having similar issues when trying to run CV and get both train-and-test scores and train-and-test predictions. None of the cvp, cvs or cv in sklearn covers all of these use cases and so I am left with looping.

Propsals:

1. add a copy of cross_val_predict that allows non-partitions. This would allow other non-partition use cases, not just for timebasedcv.
2. add a differently named class that extends cvp to allow non-partitions AND returns train-and-test scores and train-and-test predictions.

[FBruzzesi]

Hey @joshdunnlime 👋🏼 I am the author of timebasedcv as well as a maintainer here - I think this feature request might be asked in such project directly rather than here in scikit-lego.

Regarding scikit-learn compatibility, do I assume correctly that you tried using timebasedcv.sklearn.TimeBasedCVSplitter?

Would you be interested in working on it directly or share what you already have implemented?

[joshdunnlime]

I know! And that was part of my thinking...

Yes. I imported from the sklearn module. I should have been a bit clearer in my explanation - let's assume I have the following:

ts_split = TimeBasedCVSplitter(
    frequency="days",
    train_size=int(52 * 7 * 1.5),
    forecast_horizon=int(52 * 2 * 7),
    stride=2 * 7,
    window='expanding',
    time_series=df.index,
).split(df)

ts_split = list(ts_split)

# one example of some additional cv tidying logic
ts_splitter = [i for i in ts_split if len(i[1]) > 2000]

Now when I want to use this list in a sklearn cvp I get the partitioning error. This could also happen in a non-timeseries related cv strategy where you have training data that is not included in the test set. An example of this might be using synthetic data you don't want to test on, excluding groups that don't need forecasting but that add information or context to the model etc. Therefore I see it very much as an extension of an sklearn class vs just for TSCV. I considered starting my own skduplo package but figured sklego is far more mature 😁

Very happy to contribute. I have a very rough sketch that suits my needs but would probably implement propsal 2. from scratch. I would add it to model_selection to match cvp and cvs in sklearn.

[FBruzzesi]

Hey @joshdunnlime thanks for all the details! And apologies for:

• my partially informed reply last time
• having to wait so long for this reply

Time and attention have been in short supply lately

I just switched job and I am relocating between countries.

---

Coming back to the main topic:

This could also happen in a non-timeseries related cv strategy where you have training data that is not included in the test set. [...] Therefore I see it very much as an extension of an sklearn class vs just for TSCV.

This is definitely a good point to have something in here rather than in tscv, thanks for pointing out another example for which this could happen. I am tempted to check in scikit-learn itself what they would suggest, considering that TimeSeriesSplit is not "allowed" in there.

I imagine you would like to have the same exact API, but "discarding" rows that never make it as test set. The main issue I see with something with is not an aggregation such as cross_val_predict, is that there might not be a straightforward way to re-align the predictions. What I mean by this is that in the tscv case you might have samples with no prediction at the start of the series and samples with multiple predictions because they appear in multiple test sets, if for example your stride difference from the forecast horizon. Do you have any thought on this?

[joshdunnlime]

Yes - near-identical - possibly with a superset of the args and kwargs.

Something like a train set with minimum train length, a gap, small strides and reducing test/predict period.

train        gap predict
tttttttttttt|---|ppppppp    |
tttttttttttttt|---|ppppppp  |
tttttttttttttttt|---|ppppppp|
tttttttttttttttttt|---|ppppp|

So, yes, having duplicate and/or missing predictions is precisely the issue. Cross_validate already sets a precedent of returning a dict of lots of optional things:

• test score (default)
• train score (optional)
• indices (optional)

It would probably make more sense to copy and extend the cross_validate API to include test_predictions and train_predictions. These would just be returned as more items in the dict. These items would be a list of arrays (nicer still would be list of pandas DataFrames).

Essentially, the aim is get train and test preds and scores all together in once place, without too much looping, even if there are missing or duplicate bits.

Details

Best of luck with the move and new role!!