The German grid has a word for its worst weather: Dunkelflaute, literally the dark doldrums. No wind, no sun, sometimes for days. The renewables go quiet, the whole merit order falls back on thermal plants, and the price of electricity spikes at exactly the hours demand shows up. This build asks whether those hours can be flagged a day ahead from the fundamentals that cause them: the forecast lull meeting the demand the grid was already carrying. Fusing the weather forecast with strictly-past grid state beats the seasonal and price-persistence baselines by 0.23 PR-AUC on a held-out year, across 17 European bidding zones from northern Sweden to southern Italy.
doldrums_spike, a calibrated gradient-boosted classifier over 17 bidding zones
(Germany-Luxembourg, France, Netherlands, Poland, the four Swedish zones SE1-SE4,
the five Norwegian zones NO1-NO5, both Danish zones DK1/DK2, and Italy north and
south), hourly, 2019 to now. A spike is an hour whose price sits above that
zone's trailing 30-day 90th percentile (a 13% base rate, comparable across zones
that live at very different price levels). Features are the zone's forecast
weather (wind at 100m, irradiance, temperature, a wind-lull flag), strictly-past
demand lags (residual load and load at t-24 and t-168), price lags, and calendar.
Held out on the last 12 months, split by time, never shuffled.
| metric (holdout: last 12 months, 136k zone-hours) | value |
|---|---|
| PR-AUC, fused | 0.579 |
| ROC-AUC, fused | 0.890 |
| PR-AUC, seasonal climatology baseline | 0.352 |
| PR-AUC, price-persistence (AR) baseline | 0.344 |
| lift over the strongest baseline | +0.227 |
| base rate | 0.127 |
Ranked by the model, the flags are dense where it matters: of the top 1% of hours it is most sure about, 89% are real spikes; the top 5% are 72%, the top 10% are 60%.
The ablation is the point. Weather alone knows the lull is coming but not what the grid is carrying; price persistence knows yesterday spiked but not that the wind is about to die. Each alone clears the seasonal floor by a little. Together they clear it by a lot, which is the claim: the fusion carries signal no single feed contains.
| features | PR-AUC (holdout) |
|---|---|
| BASE (zone + hour + day + month) | 0.392 |
| BASE + price lags | 0.462 |
| BASE + weather | 0.523 |
| BASE + grid lags | 0.451 |
| FUSED (all) | 0.579 |
Per zone the spread is the finding. The model is strongest where thermal plants set the marginal price: DK2 0.70, DE-LU 0.70, DK1 0.69, NL 0.69, PL 0.65. It is weakest across the Norwegian hydro zones (NO3 0.43, NO1 0.53, NO5 0.53) and northern Sweden (SE1 0.48). That is the physics, not a bug: where reservoirs buffer a wind lull, short-term weather moves the price far less than it does in a thermal-dominated grid, so the doldrums signal is genuinely weaker there. Continental and Danish zones, where a lull forces expensive gas on at the margin, are exactly where the model earns its lift. Probabilities are isotonic-calibrated on the holdout, so 0.6 means 0.6.
European bidding zones are wired together by AC borders and HVDC cables, so a doldrums that lands over one zone shows up in its neighbours' prices too. The training job measures this directly: for each interconnected pair it correlates hourly price first differences (differencing strips the common EU-wide trend, so the number is genuine hour-to-hour co-movement, not shared drift) and reports how often the two spike in the same hour. The app draws it as edges between zones; hover a zone to light its links.
| interconnected pair | price corr | spike co-occurrence |
|---|---|---|
| SE1 - SE2 (northern Sweden) | 0.92 | 0.91 |
| NO1 - NO5 (southern Norway) | 0.92 | 0.92 |
| DE-LU - DK1 (the German-Danish link) | 0.86 | 0.75 |
| IT-North - IT-South | 0.82 | 0.59 |
| DE-LU - NL | 0.79 | 0.75 |
| DE-LU - DK2 | 0.77 | 0.71 |
| NO1 - NO3 (weakest link measured) | 0.29 | 0.37 |
The tightly-coupled pairs are adjacent zones inside one country or across a fat cable; the weak ones are zones separated by congested interconnectors, where prices decouple exactly when scarcity bites.
Read these before quoting the number anywhere.
- The label is a proxy. A spike is "price above the zone's own trailing 90th percentile", not a fixed euro threshold. That keeps zones comparable across a range from cheap northern-Sweden hydro to the spiky continental zones, but means "spike" is relative to recent history, not an absolute scarcity line.
- Grid state is national, weather and price are per zone. energy-charts publishes residual load per country, not per bidding zone, so same-country zones (SE1-SE4, NO1-NO5, DK1/DK2, both Italian zones) share one grid feed. The price series and the weather are genuinely per zone, and the price lags carry most of the autoregressive signal, so the sub-country resolution is real where it matters most.
- This is not a trading signal. The day-ahead auction already prices in the market's own weather forecast. The honest claim is only that fundamentals anticipate spike hours with stated lift over the seasonal and AR baselines. It is a fundamentals monitor, not a way to beat the market.
- Forecast horizon is 24 hours. Served only for the next 24 delivery hours, the window where all three price lags are genuinely known at issue time. Beyond that a price lag would be a future price the model cannot have.
- Training weather is ERA5 reanalysis. Serving uses the real forecast, which is less perfect than hindcast, so live skill sits a little below the holdout number.
A price-forecasting build lives or dies on not cheating, so the guards are explicit.
- Grid demand enters only as strictly-past lags (t-24, t-168). Realized residual load at the delivery hour is not known when forecasting it, so the contemporaneous value never reaches the features (it stays in the FG for the app's realized panel only).
- Weather enters at the delivery hour because a forecast for it genuinely exists at serving time. The renewable side is reconstructed from that forecast.
- Price and threshold are never features: they define the label.
- The split is temporal, last 12 months held out, never shuffled. The seasonal and AR baselines are recomputed on the exact same split, and the training job gates itself: if FUSED does not beat them, it prints DOG-HOUSE and the build goes to the dog house instead of this table.
An FTI (feature, training, inference) system on Hopsworks. Feature extraction is
one shared module (doldrums_features.py) imported by every pipeline and the
app, so training and serving cannot skew.
flowchart LR
OM([Open-Meteo: forecast + ERA5]):::ext --> WF[weather_pipeline] --> FGW[(doldrums_weather)]:::hops
EC([energy-charts: price + grid]):::ext --> GF[grid_pipeline] --> FGG[(doldrums_grid)]:::hops
EC --> PF[price_pipeline] --> FGP[(doldrums_price · label)]:::hops
FGW --> FV{{doldrums_fv}}:::hops
FGG --> FV
FGP --> FV
FV --> TR[train: calibrated GBM + ablation gate] --> MR[(Model Registry)]:::hops
MR --> INF[predict, hourly]
FGW --> INF
FGG --> INF
FGP --> INF
INF --> FGX[(doldrums_predictions)]:::hops
FGX --> APP[dark-doldrums app]
classDef hops fill:#10b98122,stroke:#34d399,color:#e5e7eb;
classDef ext fill:none,stroke:#6b7280,color:#9ca3af,stroke-dasharray:4 3;
The file-by-file map:
doldrums_features.py shared MITs: zonal weather, wind-lull, hourly lags,
spike label, calendar; the 17-zone config + the
interconnected-neighbour list. Imported everywhere.
collect/energy_charts.py price + grid mix, keyless, 429-aware backoff
collect/openmeteo.py ERA5 archive + live forecast, keyless
pipelines/weather_pipeline.py F -> doldrums_weather (backfill + hourly)
pipelines/grid_pipeline.py F -> doldrums_grid (one fetch per country, fan out)
pipelines/price_pipeline.py F -> doldrums_price (price, lags, threshold, spike)
pipelines/build_fv.py T leak-free feature view (grid lags, not realized)
pipelines/train.py T temporal split, ablation, self-gate, calibrate,
price-coupling, register doldrums_spike + plots
pipelines/predict.py I hourly: next-24h spike probability -> FG
app/server.py + static/ I FastAPI thin client: live EU map, zoom, coupling
tools/build_geojson.py real bidding-zone polygons (electricitymaps source)
tools/make_banner.py the series banner
eda-dark-doldrums.md profile, baselines, univariate signal, leakage guards
Makefile the whole build, one target per stage
Everything keyless end to end: energy-charts.info for price and grid mix, Open-Meteo for the weather forecast and the ERA5 archive. No account, no token, fully reproducible.
Clone into a Hopsworks project on the /hopsfs/... FUSE mount. Paths
self-derive.
make weather-job grid-job price-job # F: backfill the three FGs (2019 -> now)
make fv train-job # T: build the FV, train + register doldrums_spike
make predict-job app # I: hourly forecast + the live map
make schedule # hourly incremental features + forecastmake smoke-weather (and -grid, -price) dry-run one zone in the terminal
without touching a feature group.
Zoomed into the continental cluster, hovering DE-LU: its price-coupling edges light up with the measured correlation (the German-Danish cable at 0.85, NL 0.79, DK2 0.77), and the panel below carries all 17 per-zone holdout numbers.
A live map of the 17 zones over the animated European wind field, each zone colored by its peak spike probability for the next 24 delivery hours, with an hourly risk strip per zone. Scroll to zoom, drag to pan, double-click a zone to fit it, and hover any zone to light its price-coupled neighbours with the measured correlation. The time axis scrubs from 48 hours of realized spikes into the 24-hour forecast, so predictions sit next to what actually happened. The panel titled "how honest is this" shows the model's own holdout numbers, per-zone PR-AUC, and last week's realized spike counts: the map is never sold as an oracle. Server-rendered, content in the initial payload, the page polls for fresh state.