Data pipeline improvements

[dominiquekleeven]

Replaces old PR #43

Draft Pull Request - Work in progress

---

This PR relates to the following issue #29

The goal of the PR is to improve the data processing, and also prepare for a future migration to Darts by implementing the necessary steps (resampling of target and covariates, deduplication, handling NaN values, etc.).

Summary

• Improves data processing by resampling all data to a fixed interval based on the forecast frequency.
• Aligns target and regressors (covariates), with covariates resampled to match the target datapoint interval.
• Resampled data dramatically improves training and forecasting times, since the dataset becomes smaller and de-duplicated.
• Adds additional validation for generated forecasts.
• Introduces an evaluate_model function to the model_provider protocol.
• Adds test_prophet_model_performance.py, which uses the evaluation method to validate forecast accuracy and performance.
• The test asserts accuracy and catches regressions when the model changes, ensuring we don’t accidentally cause forecasting to fail or regress in performance.
• evaluate_model also enables us to later display metrics in the Web UI. At that stage, results should be stored alongside the model.

---

Evaluation Metrics (NEW) – test_prophet_model_performance.py

Dataset: mock-datapoints-power-grid.json (~130k datapoints)
Using default model settings (no fine tuning)

Training Time: 0.88s  
Forecast Time: 0.04s  
RMSE: 7.0kW (typical forecast error, lower is better)  
MAE: 5.2kW (average absolute error, lower is better)  
MAPE: 15.5% (average percentage error, lower is better)  
MdAPE: 7.6% (median percentage error, lower is better)  
R²: 0.277 (variance explained, higher is better)  

Evaluation Metrics (OLD)

https://github.com/openremote/service-ml-forecast/tree/old-data-processing-metrics
Dataset: mock-datapoints-power-grid.json (~130k datapoints)
Using default model settings (no fine tuning)

Training Time: 522.53s  
Forecast Time: 0.29s  
RMSE: 11.3kW  (typical forecast error, lower is better)  
MAE: 8.6kW (average absolute error, lower is better)  
MAPE: 30.0% (average percentage error, lower is better)  
MdAPE: 17.1% (median percentage error, lower is better)  
R²: 0.113 (variance explained, higher is better)  

---

Side-by-Side Comparison

The metrics do not reflect production performance, they should be interpreted as a comparison between old and new. With old being the baseline.

Metric	Old	New	Improvement
Training Time	522.53s	0.88s	~590× faster
Forecast Time	0.29s	0.04s	~7× faster
RMSE	11.3kW	7.0kW	~38% lower (lower is better)
MAE	8.6kW	5.2kW	~40% lower (lower is better)
MAPE	30.0%	15.5%	~48% lower (lower is better)
MdAPE	17.1%	7.6%	~56% lower (lower is better)
R²	0.113	0.277	~145% higher (higher is better)

---

Conclusion

By de-duplicating datapoints, filling gaps/NaN values, and resampling data to a consistent interval, the dataset size is dramatically reduced and becomes more uniform.
This cleaner dataset makes it easier for Prophet to detect trends, resulting in significantly faster training and forecasting times and improved accuracy across all metrics.