A machine learning project forecasting German day-ahead electricity prices (EPEX SPOT DE-LU) using ENTSO-E grid fundamentals and TTF natural gas prices. The project evolved through two distinct models, an initial directional classifier that I realized used the wrong signals and wasn't well thought out, before switching to a price-level regression.
This project was motivated by Williams, R. (2024), The Microeconomic Challenge with Renewable Energy (SSRN Working Paper 4718822, February 2024). The paper develops a microeconomic framework for electricity balancing costs and empirically documents a non-linear relationship between wind/solar penetration and balancing costs across five European markets
Two ideas from Williams drive the feature engineering in this project:
-
The merit-order mechanism and the role of dispatchable plants. Grid operators construct a generation stack ordered from lowest to highest marginal cost. Renewables sit at the bottom (near-zero marginal cost), nuclear next, then coal, gas, and oil peakers. Demand is essentially perfectly inelastic in the short run, so the clearing price is set by the marginal and most expensive plant required to meet load. In Germany this is most often a gas plant. The economically meaningful quantity is therefore not total load, but residual load: the demand that must be met by priced, dispatchable plants after renewables have been deployed. Residual load determines how far up the merit-order stack the market must climb, and the gas price determines the height of the gas segment.
-
The non-linearity of the supply curve at high renewable penetration. Williams shows that balancing costs follow a cubic relationship with renewable penetration — flat at low penetration, then sharply escalating beyond roughly 30–35%. Figure 5 in the paper shows the underlying mechanism: the supply curve is sharply convex at the right tail, where oil peakers replace gas. Small errors in load or renewable forecasts can swing the clearing price by a large amount because the marginal unit can jump from a cheap gas plant to an expensive peaker. This non-linearity is the direct economic justification for using a random forest rather than linear regression, the price response to residual load and gas is not a single slope, and a linear model cannot capture the curvature.
This framing gives the project its two main features:
Residual load = load forecast − wind forecast − solar forecast: the demand that priced, dispatchable plants must serve TTF gas price: the marginal cost of gas plants, which frequently set the clearing price in Germany.
The directional classifier (Model 1) ignored this economic structure entirely, predicting only the sign of day-over-day moves from rolling weather averages. The price-level regression (Model 2) uses residual load and gas directly, mirroring the structure of the merit-order curve that Williams's framework describes.
The project uses three data sources: ENTSO-E (via entsoe-py): day-ahead prices, load forecasts, wind and solar generation forecasts yfinance (TTF=F): TTF natural gas front-month futures Open-Meteo (archive API): weather observations (first model only)
Model 1: Directional Classifier with Rolling Weather Averages
The first model framed the problem as binary classification: predict whether tomorrow's day-ahead price will be higher or lower than today's. A RandomForestClassifier was trained on 10-day rolling averages of temperature, wind speed, solar radiation, and gas price, plus day-of-week and month as calendar features. The target was:
y = 1 if tomorrow's price > today's price, else 0
Hyperparameters were tuned with GridSearchCV using TimeSeriesSplit (5 folds) to respect temporal ordering, finding n_estimators=200, max_depth=20, min_samples_leaf=10.
Results:
Period Test Accuracy
2022 64.1%
2023 63.3%
2024 71.0%
2025 67.1%
Full test set (Jun 2025–Jun 2026) 66.0%
The calendar-only baseline (day-of-week and month, no weather, no gas) achieved 64.1% on the same test set.
Confusion matrix (full test set)
Full model: [[160 36] Calendar only: [[148 48]
[ 90 79]] [ 82 87]]
The full model correctly identified more up-days (79 vs 87 true positives for down-days) but both models struggled symmetrically.
Why this model was rejected: McNemar's test
The accuracy gap between the full model (66.0%) and the calendar-only baseline (64.1%) is 1.9%. I had to verify whether that gap is a real signal from the weather and gas features, or noise. McNemar's test answered this directly: it looks at the days where the two models disagree, and asks whether the full model wins those disagreements significantly more often than chance.
I found a p-value of 0.83, meaning there is an 83% probability of seeing a disagreement pattern this unbalanced even if the two models were identical. The weather and gas features add no statistically significant predictive power beyond knowing the day of week and month. The model was rejected.
Why the rolling averages failed
I found the failure to lie in the structure of the model. A 10-day rolling average of temperature or wind speed is a slow-moving signal. By the time it responds to a weather event, 10 days of smoothing have already erased it. What remains is essentially a proxy for the time of year — cold average temperatures mean winter, high solar radiation means summer. But day-of-week and month already encode that information exactly. The weather features were collinear with the calendar features, so they added no independent information the model could use. The McNemar result confirmed this.
What I learned
Accuracy alone does not establish that a model has learned anything meaningful — a near-random baseline can achieve 64% on a time series with seasonal structure Statistical testing (McNemar) is necessary to distinguish signal from noise before concluding a model is useful Slow-moving features derived from high-frequency data aren't reliable. The averaging process that makes them stable also makes them redundant with simpler proxies
Model 2: Price-Level Regression with Residual Load and Gas
Why regression, and why residual load:
Starting over I had to rethink both the target and the features.
On the target: directional accuracy is a weak criterion. The model can be right about direction on easy, trend-following days and wrong on the economically important spike and reversal days, and the accuracy metric will never show it. Predicting the price level directly is harder but more informative: errors are in €/MWh and can be compared to a meaningful baseline.
On the features: rolling weather averages failed because they restated seasonality. The merit-order framing from the research paper points to a better feature: residual load, computed from the same ENTSO-E forecasts used to clear the day-ahead market.
residual load = load forecast − solar (PV) forecast − wind offshore forecast − wind onshore forecast
This is forward-looking by construction — it's what grid operators and market participants see before the auction clears. A day with high wind and solar has low residual load; the renewable energy pushes expensive gas plants down the dispatch queue, and prices fall. A cold, calm, cloudy day has high residual load; the market must dispatch gas and the price rises. Residual load captures the causal mechanism driving price formation, not a lagged statistical proxy for it.
I used RMSE (Root Mean Squared Error) which measures the typical size of the model's prediction error in €/MWh, so a value of say 16.95 means daily predictions are off by roughly that much on average. Squaring the residuals before averaging means large misses are penalised disproportionately (a €60 error counts 36× a €10 error, not 6×), which matches the economic reality that occasional large errors matter far more than consistent small ones. It is preferred over MAPE for this market because German day-ahead prices cross zero on high-renewable days, which would make percentage errors blow up or become undefined.
Baseline: persistence, the dummy model
The benchmark throughout is the persistence forecast: predict tomorrow's price as today's price. This is the hardest naive baseline to beat for energy prices because day-ahead prices are autocorrelated — yesterday's price is genuinely informative about today's. Any model that cannot beat persistence is not useful.
Step 1: Linear regression
A linear regression on residual load alone established the direction of effect and provided a first quantitative check.
price = β₀ + β₁ × residual_load x ... + ε
Coefficient on residual load: 0.00303 (€/MWh per MW)
The positive sign is correct: higher residual load = higher price. The magnitude means an additional 1,000 MW of residual demand increases the expected price by approximately €3/MWh, consistent with the slope of the gas portion of the merit-order curve.
Model Test RMSE (€/MWh) Persistence:
Persistence 28.41 baseline
Linear regression 21.90 −22.9%
Linear regression beats persistence by 22.9% using a single economically-grounded feature. This is meaningful, but leaves a large fraction of price variance unexplained, and it can't handle the 2022 gas crisis, where prices moved into ranges far outside anything in the training data.
Step 2: Adding gas prices, and why linear regression failed
TTF gas futures price was added as a second feature. Gas plants set the marginal price in the German market on a large fraction of hours, so the TTF price directly determines where the gas segment of the merit-order curve sits. Including it should improve the model's ability to track price level across different gas-price regimes.
It does, but not with a linear model. I found the problem to be the 2022 European energy crisis. TTF gas peaked near €339/MWh in August 2022, compared to a pre-crisis median around €20–30/MWh. The relationship between gas price and electricity price is not simply additive across that range: at extreme gas prices the market dynamics change, demand response activates, and the price-formation relationship shifts nonlinearly. A linear model cannot capture this as it fits one slope for the entire gas price range, which is very wrong at the extremes.
A RandomForestRegressor handled the nonlinearity well. Trees partition the feature space into regions and predict the average target within each region. For this data, the gas and residual-load interaction is nonlinear, and tree splits capture interactions of this kind automatically.
Features of the model were residual load, gas, day of week, month
model = RandomForestRegressor(n_estimators=200, random_state=1) \
Results:
Model Test RMSE (€/MWh) vs Persistence
Persistence 28.41 baseline
Linear regression 21.90 −22.9%
Random forest 16.95 −40.3%
With test-period price standard deviation of 30.57 €/MWh, R^2 was 0.69, leaving 31% of price variance unexplained.
Ablation testing where I removed features and measured RMSE increase gives a picture of each feature's necessity:
['residual load', 'gas', 'day of week', 'month'] -> 16.950721791402877
['residual load', 'day of week', 'month'] -> 37.783531276723934
['residual load', 'gas'] -> 20.126757092964286
Validation 1: walk-forward (expanding window)
A single train/test split produces one performance estimate that depends entirely on which period landed in the test set. Walk-forward validation retrains and re-evaluates across multiple years, always in chronological order.
model trained on 2019-2019 data tested on 2020 RMSE: 10.107427306728619
dummy model trained on 2019 tested on 2020 RMSE: 11.52300428302183
the model improved on the dummy RMSE by 12.28%
model trained on 2019-2020 data tested on 2021 RMSE: 78.89850461830677
dummy model trained on 2020 tested on 2021 RMSE: 34.38911230493773
the model improved on the dummy RMSE by -129.43%
model trained on 2019-2021 data tested on 2022 RMSE: 106.39118090899066
dummy model trained on 2021 tested on 2022 RMSE: 65.36888110698143
the model improved on the dummy RMSE by -62.76%
model trained on 2019-2022 data tested on 2023 RMSE: 25.639098494959455
dummy model trained on 2022 tested on 2023 RMSE: 30.253354249771476
the model improved on the dummy RMSE by 15.25%
model trained on 2019-2023 data tested on 2024 RMSE: 22.478338772887135
dummy model trained on 2023 tested on 2024 RMSE: 31.33618527092678
the model improved on the dummy RMSE by 28.27%
model trained on 2019-2024 data tested on 2025 RMSE: 14.659528470265288
dummy model trained on 2024 tested on 2025 RMSE: 28.26956881375219
the model improved on the dummy RMSE by 48.14%
model trained on 2019-2025 data tested on 2026 RMSE: 18.13321372993209
dummy model trained on 2025 tested on 2026 RMSE: 30.743861912817554
the model improved on the dummy RMSE by 41.02%
Note: 2026 isn't a full year, data only goes to June 1
The regime shift finding: The model loses badly in 2021 and in 2022. This wasn't a random bad year, it was a consequence of the tree-extrapolation constraint. A random forest predicts by averaging training observations; its predictions are bounded by the range of target values seen during training. A model trained through 2020 has never seen prices above ~€80/MWh. When 2021 prices climb into the €100–200 range and 2022 prices reach the high hundreds, the model's predictions are capped well below reality, while persistence tracks a sustained climb day by day. The losses are mechanically explained.
Once the 2022 crisis enters the training set (test year 2023 onward), the model recovers and improves monotonically: +15%, +28%, +48%. This confirmed the model works within regimes it has seen, and breaks specifically across regime boundaries it has not.
Summary: beats persistence in 5 of 7 folds (4/6 full years). The two losses are fully explained by the regime change failure. Within stable regimes the model is reliable and improving.
Validation 2: Diebold-Mariano significance test
Walk-forward shows whether the model beats persistence across folds. The Diebold-Mariano test answers within a single test period: is the RF's lower RMSE a real, systematic edge, or did it happen to get lucky on the specific days in this test set?
I found: Diebold-Mariano TS: 4.778600036557779, a DM statistic of ~4.78 is 4.78 standard errors above zer Diebold-Mariano p-val: 4.169213365212698e-06 Model is significantly better if the TS > 0 and p-val < 0.05: True
Shortcomings and Limitations:
The model cannot extrapolate beyond its training range. Random forests predict by averaging training observations. Predictions are bounded by the range of target values seen in training. The 2021–2022 walk-forward results quantify this failure directly.
Heavy dependence on gas prices
Gas carries the vast majority of feature importance and removing it produces RMSE worse than persistence. This is economically correct, gas sets the marginal price in Germany on most hours, but it means the model is sensitive to gas input quality. Delayed, revised, or erroneous gas data will affect predictions sharply.
~31% of price variance is unexplained
Forecast inputs are assumed accurate
The model uses ENTSO-E forecasts of load, wind, and solar as features. Those forecasts carry their own errors, which would compound into the price prediction in live use. The current evaluation does not propagate forecast uncertainty, so live performance would likely be somewhat worse than the backtest RMSE suggests.
Statistical accuracy is not tradeable value
RMSE measures forecast accuracy, not profitability. A tradeable backtest would require exogenous market prices to trade against and explicit transaction costs.