Fix/969 MSTL cross validation with refit = False

[yoselalberto]

Hello Nixtla, I propose a solution to an issue that was first reported in #969

The problem is that cross validation with MSTL fails when the training windows are shorter than season_lenght; I added a new parametershort_train_behavior that defines the behaviour when training window is shorter than max season_length, to either return NaN forecasts (default) or to skip the window, this only apply during cross-validation.

In addition, I added a note to the MSTL model page, and updated the MSTL class documentation string, so it is reflected in the API reference.

Feasibility over filling early windows

Using only CV windows with sufficient data aligns with the best practice: it is better to have "less but real" data than "more but fake" data for statistical validity; filling early validation windows is considered a high-risk practice because it can lead to overfitting and model failure in production.

The core goal of cross-validation is to estimate the model’s generalization error, when filling data synthetically, we are no longer measuring how the model performs on the real-world process; we are measuring how well it can "learn" the specific generator used to create the synthetic data(e.g. recycling values to complete train window).

Why

Reproducing the example provided, MSTL with cross_validation(..., refit=False) raises:

ValueError: could not broadcast input array from shape (11,) into shape (12,)

This occurs when early CV windows have fewer training observations than season_length:

• 25 observations, h=12, n_windows=3, step_size=1 → test_size=14
• First window: 11 training obs (25 - 14) < season_length=12
• MSTL's STL decomposition yields a seasonal component shorter than expected, causing _predict_mstl_seas to fail in np.tile()

The bug is MSTL-specific: I verified that Naive, SeasonalNaive, AutoETS, Holt, AutoARIMA, and DynamicOptimizedTheta all succeed with the same CV settings; other models either handle short series differently or avoid this case.

Solution

I reviewed how other StatsForecasts models handle windows with insufficient data; MSTL's new short_train_behavior parameter handles this: using nan returns NaN predictions (default, such as in SN/SES/SWA), while with skip only windows with sufficient training data are evaluated and returned; this only applies during cross-validation with MSTL so regular forecasts aren’t altered by short training windows.

Impact

Aspect	Impact
Other models	None, short_train_behavior applies only when a model defines it (currently MSTL), for others no windows are skipped; in addition, this only applies in cross-validation, there is no change to non-CV forecasting behavior for MSTL.
Speed	None, no impact on performance, the changes are pure Python control-flow: a boolean flag check in MSTL and a temporary attribute toggle in StatsForecast.cross_validation.
Output	When windows are skipped, the returned DataFrame contains fewer rows, only those from valid windows.

Changes

python/statsforecast/core.py: temporarily toggles a CV-only flag (_cv_short_train_behavior) around cross-validation and restores it afterward.
python/statsforecast/models.py: guards short-train handling and min_train_size behind the CV-only flag; docstring clarifies CV-only scope.
tests/test_mstl.py: updates forecast expectations to confirm short_train_behavior is ignored outside CV.
nbs/docs/models/MultipleSeasonalTrend.ipynb / nbs/docs/tutorials/MultipleSeasonalities.ipynb: documentation notes CV-specific behavior.

Closes #969 issue.

[codspeed-hq]

Merging this PR will not alter performance

✅ 38 untouched benchmarks

---

_{Comparing yoselalberto:fix/969_MSTL_cross_validation (8512a30) with main (a0ef3f4)}

[yoselalberto]

I reviewed the CI/CD jobs that failed, all are with python 3.9; I specified versions of datasetsforecast for python 3.9 (1.0.0) and for newer versions >= 3.10 (1.0.1); with this changes, the local tests completed

[yoselalberto]

I noticed that statsforecast was updated, I will incorporate the latests changes, and update the PR so that it integrates smoothly

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[yoselalberto]

Hi, I reviewed how other models handle windows with insufficient data; MSTL's new short_train_behavior parameter handles this: using nan returns NaN predictions (default, such as in SN/SES/SWA), or skip, which skips those windows, this only applies during cross validation. In addition, I added a note to the MSTL model page, and updated the MSTL class documentation string, so it is reflected in the API reference.

[madhukrupamegalattifrancis0]

From a supply chain planning perspective, forecast accuracy should be evaluated not only using statistical error metrics but also based on downstream operational impact.

For example:

• Service level impact (stockouts vs overstock)
• Inventory carrying cost implications
• Demand variability sensitivity

It may be valuable to introduce business-oriented evaluation metrics that complement traditional statistical measures, enabling better alignment between forecasting models and real-world supply chain performance.

I would be happy to propose a structured framework for integrating such metrics into the evaluation pipeline.

[madhukrupamegalattifrancis0]

Adding a quick extension to this idea:

In supply chain applications, evaluation metrics could be extended to include:

• Cost-weighted error (penalizing under-forecast vs over-forecast differently)
• Service level attainment simulation
• Inventory turnover impact

This could help bridge the gap between model accuracy and operational decision quality.

Happy to collaborate on defining such evaluation extensions if useful.