Skip to content

fix: request energy-charts prices on local day boundaries#35

Merged
b3nn0 merged 1 commit into
b3nn0:mainfrom
smartenergycontrol-be:fix/energycharts-local-day-boundary
Jul 25, 2026
Merged

fix: request energy-charts prices on local day boundaries#35
b3nn0 merged 1 commit into
b3nn0:mainfrom
smartenergycontrol-be:fix/energycharts-local-day-boundary

Conversation

@scruysberghs

@scruysberghs scruysberghs commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Rebased onto current master.

Problem

fetch_prices_energycharts snaps the requested range to UTC midnight:

start_of_day = rstart.replace(hour=0, minute=0, second=0, microsecond=0)

rstart is already UTC here, but the market day runs on local time. For a
bidding zone east of UTC this drops the first hours of the day from the request,
and energy-charts returns exactly the window it was asked for.

Reproducible for BE (CEST, UTC+2):

start=2026-07-24T00:00:00Z  ->  88 quarter hours, first one 02:00 local
start=2026-07-23T00:00:00Z  -> 184 quarter hours, includes 00:00 local

The eight quarter hours from 00:00 to 01:45 local were never fetched. They ended
up in the store as 0.0 and were served by /prices as real prices:

2026-07-23T23:45:00+02:00   162.40
2026-07-24T00:00:00+02:00     0.00
   ...
2026-07-24T01:45:00+02:00     0.00
2026-07-24T02:00:00+02:00   155.26

That is the cheapest slot of the whole horizon, so anything optimising on price
schedules load exactly there while the real price was around 155 EUR/MWh.

This usually repairs itself, because a later and wider fetch covers the same
range before those hours become the present. It only surfaces when the fetched
range starts on the new UTC day. The gap is the size of the UTC offset: two
hours in CEST, one hour in CET.

Relation to #30 / _is_invalid_zero_data

This is the same family of problem as #30 (and the zero data mentioned in #34),
but the guard added there only discards a fetch when the last ~20 hours are all
zero, i.e. a full-day blackout. The case here is a partial gap of a few quarter
hours at the day boundary, embedded in otherwise valid data, so the guard does
not catch it. That guard treats the symptom; this change removes the cause, so
those quarter hours are fetched with real prices in the first place. The two are
independent and both useful.

Fix

Convert the range to the bidding zone timezone before snapping to midnight, then
back to UTC for the query. region.get_timezone_info() is already used
elsewhere in this class. The window can only grow, so nothing that was covered
before stops being covered.

Observed on zone BE, 2026-07-23.

@scruysberghs

Copy link
Copy Markdown
Contributor Author

ah, seems like I was running an older version, I'll rebase the pr. Maybe db97100 already would have fixed the issue I saw today

The market day runs on local time, but the fetch window was snapped to UTC
midnight. For zones east of UTC that cuts the first hours of the day off the
request, and energy-charts returns only what was asked for.

For BE on 2026-07-23 the store requested start=2026-07-24T00:00:00Z and got
88 quarter hours starting at 02:00 local instead of 96 starting at 00:00.
The eight missing quarters were stored as 0.0 and served as real prices.
The gap is the size of the UTC offset, so two hours in CEST and one hour in
CET.

Convert the range to the bidding zone timezone before snapping to midnight.
The window can only grow, so nothing that was covered before stops being
covered.
@scruysberghs
scruysberghs force-pushed the fix/energycharts-local-day-boundary branch from 504a48f to 656d5d7 Compare July 23, 2026 18:26
@b3nn0

b3nn0 commented Jul 23, 2026

Copy link
Copy Markdown
Owner

Hmm, I do agree that it would probably be better to use local times here, but are you sure this actually fixes your problem?
For training, the data fetching might fall 1-2 hours short of "today's" end. But that doesn't really matter at all, except for training on slightly less than 120 or so days.
For actual forecasting, the end timestamp for the query is like a week in the future anyway, where no prices are available at all regardless of the exact end time.

It does, however, explain, why I used 20x4 instead of 24x4 to check for the last days data being 0 - since, as you recognized correctly, accidentally didn't have 24 hours and didn't know why. Not that it really matters.

So yes, I think my later patch did fix it. Happy to merge anyway since it makes intentions clearer I think? I assume this is tested properly?

@scruysberghs

Copy link
Copy Markdown
Contributor Author

Today I think I first ran into the issue where all actual next day prices where 0's, the bug that was fixed in db97100 . I then tried refetching the data and got this
afbeelding
I did test but to be sure we'll have to wait untill new prices are published tomorrow. Suggest you wait to merge at least untill tomorrow

@scruysberghs

Copy link
Copy Markdown
Contributor Author

The 24 -> 25 July day-ahead publication (the first day boundary after deploying) came through cleanly. The window that was broken before now holds real prices:

before (unpatched, UTC-midnight fetch):

2026-07-24T00:00:00+02:00   0.00
   ... eight quarter hours ...
2026-07-24T01:45:00+02:00   0.00

after (this change):

2026-07-25T00:00:00+02:00   17.592
2026-07-25T00:15:00+02:00   16.863
2026-07-25T00:30:00+02:00   16.343
2026-07-25T00:45:00+02:00   15.061

No zero blocks in the stored series, and the fetch URL now starts at 2026-07-24T22:00:00Z, i.e. local midnight for the BE zone.

@b3nn0

b3nn0 commented Jul 25, 2026

Copy link
Copy Markdown
Owner

cool, thanks for double checking

@b3nn0
b3nn0 merged commit 51d24ef into b3nn0:main Jul 25, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants