fix(weather): throttle failed forecast fetches, not only successes - #138
Merged
Conversation
A capture is written only on success, so the calendar-day gate never closed for a location that could not be fetched: it stayed due for the rest of the UTC day and was retried on every tick, each retry being another openMeteoMaxAttempts requests. Against the outage pattern already measured for this provider — 105 of 177 fetches meeting a 503 over five days, episodes about three hours per location — the planned one weighted call per location per day became up to a hundred and twenty on an hourly cron. A persistent read fault on ObtainLatestForecastCapture did the same, converting a storage problem into upstream traffic. Count the day's failed attempts per location in service_meta and double the wait after each, so tries land at roughly 0, 1, 3, 7 and 15 hours and then stop. The spacing is the point: a flat budget burned in the first few ticks would miss a three-hour outage recovering. Move weatherForecastRetryBase to change the density and weatherForecastMaxDailyAttempts to change the count; nothing else reads either. Every uncertainty about the marker resolves to "fetch" — unreadable, unparseable, or from another day all mean a fresh budget, since a wrong yes costs one request while a wrong no is a location that silently stops updating. A deferred location is logged as deferred=, never skipped=. Conflating "already have today's" with "failing and waiting" is what let the unthrottled version read as healthy. Refs: #132 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U6jNuzK5PZrhDWBxB3t2gY
This was referenced Aug 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WeatherForecastAgent's daily gate keys on the storedMAX(captured_at), which is writtenonly on success. A location that could not be fetched therefore stayed due for the rest of
the UTC day and was retried on every tick, each retry being another
openMeteoMaxAttemptsrequests.Against the outage pattern already measured for this provider — 105 of 177 fetches meeting a
503 over five days, episodes running about three hours per location — the plan's "one
weighted call per location per day" became up to 24 ticks × 5 attempts = 120 calls per
location per day on the hourly cron. A persistent read fault on
ObtainLatestForecastCapturedid the same thing from the other side, converting a storageproblem into upstream traffic.
What changed
A per-location marker under
repository.ServiceMetaKeyForecastAttemptPrefixinservice_metacounts the day's failed attempts and stamps the last one. The wait doublesafter each failure, so tries land at roughly 0, 1, 3, 7 and 15 hours and then stop.
The spacing is the point, not just the cap: a flat budget burned in the first few ticks would
sit out the rest of the day and miss a three-hour outage recovering.
weatherForecastRetryBasemoves the density,
weatherForecastMaxDailyAttemptsmoves the count, and nothing else readseither.
Every uncertainty resolves to "fetch." A marker that cannot be read, cannot be parsed, or
belongs to another day means a fresh budget — a wrong yes costs one request, a wrong no is a
location that silently stops updating. A failure to write the marker is logged and
swallowed for the same reason: the worst case is that this location keeps the old behaviour
until the write lands.
A deferred location is counted and logged as
deferred=, neverskipped=. "Already havetoday's" and "failing and waiting out a backoff" are opposite states, and spelling both
skippedis what let the unthrottled version read as healthy.Tests
Through
Run: a failed fetch and a failed retain each leave a parseable marker; a spentbudget defers instead of refetching (verified to fail with the gate removed); a marker from
another day is a fresh budget; an unreadable or garbage marker never blocks a fetch; a
location with today's forecast stored writes no marker at all.
Directly on
retryWindowOpen, which takesnowexplicitly and so needs no clock seam: seventiming cases across the doubling schedule. Plus the marker round-trip, four malformed
encodings, and the wait schedule itself.
go vet,scripts/lint-checks.shandgolangci-lintclean; 41 packages green.Refs #132