Skip to content

fix(weather): throttle failed forecast fetches, not only successes - #138

Merged
prorochestvo merged 1 commit into
alphafrom
fix/132-forecast-fetch-backoff
Aug 23, 2026
Merged

fix(weather): throttle failed forecast fetches, not only successes#138
prorochestvo merged 1 commit into
alphafrom
fix/132-forecast-fetch-backoff

Conversation

@prorochestvo

Copy link
Copy Markdown
Owner

WeatherForecastAgent's daily gate keys on the stored MAX(captured_at), which is written
only on success. A location that could not be fetched therefore stayed due for the rest of
the UTC day and was retried on every tick, each retry being another
openMeteoMaxAttempts requests.

Against the outage pattern already measured for this provider — 105 of 177 fetches meeting a
503 over five days, episodes running about three hours per location — the plan's "one
weighted call per location per day" became up to 24 ticks × 5 attempts = 120 calls per
location per day on the hourly cron. A persistent read fault on
ObtainLatestForecastCapture did the same thing from the other side, converting a storage
problem into upstream traffic.

What changed

A per-location marker under repository.ServiceMetaKeyForecastAttemptPrefix in
service_meta counts the day's failed attempts and stamps the last one. The wait doubles
after each failure, so tries land at roughly 0, 1, 3, 7 and 15 hours and then stop.

The spacing is the point, not just the cap: a flat budget burned in the first few ticks would
sit out the rest of the day and miss a three-hour outage recovering. weatherForecastRetryBase
moves the density, weatherForecastMaxDailyAttempts moves the count, and nothing else reads
either.

Every uncertainty resolves to "fetch." A marker that cannot be read, cannot be parsed, or
belongs to another day means a fresh budget — a wrong yes costs one request, a wrong no is a
location that silently stops updating. A failure to write the marker is logged and
swallowed for the same reason: the worst case is that this location keeps the old behaviour
until the write lands.

A deferred location is counted and logged as deferred=, never skipped=. "Already have
today's" and "failing and waiting out a backoff" are opposite states, and spelling both
skipped is what let the unthrottled version read as healthy.

Tests

Through Run: a failed fetch and a failed retain each leave a parseable marker; a spent
budget defers instead of refetching (verified to fail with the gate removed); a marker from
another day is a fresh budget; an unreadable or garbage marker never blocks a fetch; a
location with today's forecast stored writes no marker at all.

Directly on retryWindowOpen, which takes now explicitly and so needs no clock seam: seven
timing cases across the doubling schedule. Plus the marker round-trip, four malformed
encodings, and the wait schedule itself.

go vet, scripts/lint-checks.sh and golangci-lint clean; 41 packages green.

Refs #132

A capture is written only on success, so the calendar-day gate never
closed for a location that could not be fetched: it stayed due for the
rest of the UTC day and was retried on every tick, each retry being
another openMeteoMaxAttempts requests. Against the outage pattern already
measured for this provider — 105 of 177 fetches meeting a 503 over five
days, episodes about three hours per location — the planned one weighted
call per location per day became up to a hundred and twenty on an hourly
cron. A persistent read fault on ObtainLatestForecastCapture did the
same, converting a storage problem into upstream traffic.

Count the day's failed attempts per location in service_meta and double
the wait after each, so tries land at roughly 0, 1, 3, 7 and 15 hours and
then stop. The spacing is the point: a flat budget burned in the first
few ticks would miss a three-hour outage recovering. Move
weatherForecastRetryBase to change the density and
weatherForecastMaxDailyAttempts to change the count; nothing else reads
either.

Every uncertainty about the marker resolves to "fetch" — unreadable,
unparseable, or from another day all mean a fresh budget, since a wrong
yes costs one request while a wrong no is a location that silently stops
updating.

A deferred location is logged as deferred=, never skipped=. Conflating
"already have today's" with "failing and waiting" is what let the
unthrottled version read as healthy.

Refs: #132

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U6jNuzK5PZrhDWBxB3t2gY
@prorochestvo
prorochestvo merged commit aac05ab into alpha Aug 23, 2026
1 check passed
@prorochestvo
prorochestvo deleted the fix/132-forecast-fetch-backoff branch August 30, 2026 16:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant