Skip to content

[codex] Add calibrated forecast intervals - #21

Merged
amarakramali merged 1 commit into
mainfrom
codex/add-forecast-intervals
Jul 28, 2026
Merged

[codex] Add calibrated forecast intervals#21
amarakramali merged 1 commit into
mainfrom
codex/add-forecast-intervals

Conversation

@amarakramali

Copy link
Copy Markdown
Owner

What changed

  • calibrate one prediction-interval width per forecast lead using recursive,
    non-overlapping rolling-origin errors
  • reserve a chronological calibration window, then refit the exported model on
    all observations after calibration
  • add lower and upper interval columns to forecast CSVs and persist calibration
    metadata with the model
  • render the interval band in forecast plots and validate paired bounds in the
    Streamlit data loader
  • expose coverage and calibration-window controls in both forecast and demo
    commands
  • document the method and its time-series coverage limitation

Why

Point forecasts do not communicate how quickly uncertainty changes across a
recursive horizon. A single global residual standard deviation would also hide
the horizon effect and risk training leakage.

This implementation trains the calibration estimator only on data before the
held-out window. It then forecasts each calibration block recursively and uses
the finite-sample split-conformal order statistic independently at every lead.
The production estimator is refitted on all data only after widths have been
measured.

User impact

aep-forecast and aep-demo now produce:

  • forecast_xgb_MW
  • forecast_xgb_lower_MW
  • forecast_xgb_upper_MW

The default target coverage is 90% over a trailing 30-day calibration window.
Coverage remains empirical: temporal dependence and distribution drift prevent
a formal distribution-free guarantee.

Validation

  • python -m ruff check src tests streamlit_app.py
  • python -m pytest -q — 87 passed
  • python -m compileall -q src tests streamlit_app.py
  • python -m build
  • end-to-end 13-day synthetic demo with 3-hour intervals
  • verified lower bound ≤ point forecast ≤ upper bound for every generated row
  • inspected saved interval coverage, calibration window, and horizon widths in
    the model artifact

@amarakramali
amarakramali marked this pull request as ready for review July 28, 2026 10:05
@amarakramali
amarakramali merged commit 986ccdb into main Jul 28, 2026
2 checks passed
@amarakramali
amarakramali deleted the codex/add-forecast-intervals branch July 28, 2026 10:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant