Kafka Grid Intelligence

Real-time GB grid-stress prediction over Kafka. Historical NESO demand and Elexon generation-mix data are replayed through a Kafka pipeline (grid.raw → grid.enriched → grid.predictions), enriched with rolling features, and scored by an XGBoost classifier served from MLflow. Predictions land in Postgres.

replay → grid.raw.v1 → feature_engineer → grid.enriched.v1 → scorer → grid.predictions.v1 → pg_sink → Postgres

• Data — scripts/download_neso_demand.py pulls live from the NESO CKAN and Elexon FUELHH APIs into data/neso_demand.parquet (10,174 half-hourly periods).
• Model — scripts/train_model.py builds features via the shared compute_features_batch, labels against a causal rolling-p90 threshold, trains XGBoost, and registers grid-stress to MLflow.
• Pipeline — make up then make feature-engineer, make scorer, make sink, make replay. Feature parity between batch training and streaming inference is enforced by tests/test_feature_parity.py.

---

Model audit: why is the ROC-AUC 0.983?

A 0.983 ROC-AUC is unusually high for this task, so this section audits where that performance comes from. Is it label leakage, a trivially easy task, or a real model? The result is a mix: there is no evidence of leakage, short-range autocorrelation provides an easy signal, and calendar effects provide a genuine longer-range forecasting signal. Moving the prediction target into the future separates those two signals.

Report PR-AUC, not ROC-AUC. The positive class (a stressed period) is only ~6–10% of the data. ROC-AUC's baseline is 0.5 regardless of imbalance, so it stays optimistic while false positives pile up against the huge negative class. PR-AUC's precision term feels every false positive and has a baseline equal to the positive rate, so it is the reliable read here. Both metrics are shown below; the gap between them is itself part of the finding.

The label, and the trap inside it

The target is demand_mw > trailing-30-day p90, with the threshold computed from strictly past data (.shift(1) before the rolling quantile). The trap: GB electricity demand is extraordinarily autocorrelated —

lag	autocorrelation
30 min	0.99
1 h	0.96
2 h	0.87

— so demand_rolling_mean_2h is very nearly a copy of the current demand the model is trying to classify. At horizon 0, the label is almost given away by the features.

The test: hold the features fixed, and move the target forward H half-hourly periods — predict whether demand will be stressed H periods from now. Thus, H=24 is 12 hours ahead and H=48 is 24 hours ahead. As H grows, the current-demand shortcut stops working, and any skill that survives has to come from somewhere real. This is a single flag on the training script; no feature code changes.

Horizon sweep

Same 9 features, same model, same time-ordered split — only the forecast horizon (measured in half-hourly periods) changes.

Horizon	Lead	ROC-AUC	PR-AUC
0	now	0.983	0.919
2	1 h	0.969	0.863
4	2 h	0.929	0.749
6	3 h	0.915	0.698
8	4 h	0.902	0.642
12	6 h	0.915	0.708
24	12 h	0.911	0.629
48	24 h	0.909	0.607

The score does not decay to noise. It falls, then plateaus around 0.91 ROC / 0.6 PR-AUC and holds there all the way to a full day ahead.

Two signals, not one

A curve that falls then flattens is the fingerprint of two mechanisms with different reach:

• Persistence (short range, H 0→8). Skill from raw autocorrelation — high now, high soon. Excellent but easy; it decays smoothly as the recent-demand features stop being informative about the target, and is essentially gone by ~4 hours ahead.
• Periodicity (long range, H ≥ 12). Persistence is dead, yet ~0.91 ROC survives — carried by the daily and weekly cycle. Horizon 48 is "the same time tomorrow," so diurnal structure makes it predictable again (H 12 even ticks back up). This is the genuinely useful, non-trivial signal, and it holds steady out to 24 hours.

Proving it: the calendar ablation

If periodicity really drives the plateau, then the three calendar features (hour_of_day, day_of_week, is_weekend) should carry almost all the long-horizon skill on their own, and removing them should collapse it. Retrained at horizon 24 (12 h ahead), test base rate 9.9%:

Feature set	ROC-AUC	PR-AUC
All 9 features	0.911	0.629
Calendar removed (6 rolling only)	0.733	0.246
Calendar only (3 feats)	0.886	0.515

Strong evidence. Three calendar features recover 0.515 of the 0.629 PR-AUC. Strip them out and the model loses more than half its ability to flag stress events — the rolling/persistence features, a full half-day out, are nearly spent.

Note how differently the two metrics register this. ROC falls a modest 0.911 → 0.733; PR-AUC more than halves. ROC understated the collapse — concrete evidence, from the experiment itself, that PR-AUC was the reliable metric all along.

Feature importance

Independent of the AUC numbers, XGBoost gain importance shows the model visibly pivoting. At the nowcast the top driver is recent demand; 12 hours ahead, hour_of_day and is_weekend rise to the top and the rolling-mean importance collapses from 0.25 to 0.07. (Gain importance can over-weight low-cardinality binary splits, so the ablation PR-AUC is the quantitative claim and this figure is the direction.)

Conclusion

0.983 = ~7 ROC points of cheap persistence, stacked on a genuine roughly 0.5–0.6 PR-AUC periodicity signal. Not leakage, not trivial — a real model whose short-range skill is easy (autocorrelation) and whose long-range skill is modest but legitimate (diurnal/weekly structure), useful for day-ahead planning.

No leakage, by construction: the rolling threshold uses .shift(1), every feature excludes the current period, and the split is time-ordered rather than shuffled — so "no leakage" is a property of the code, not a hope.

Reproduce

# nowcast — the original 0.983 model
python -m scripts.train_model --horizon 0

# sweep the forecast horizon; features never change
for H in 2 4 8 12 24 48; do python -m scripts.train_model --horizon $H --no-register; done