ops(stage1): record the 2026-09-04 attempt, aborted at S1 on a slow upstream 503 - #63
Merged
Merged
Conversation
…pstream 503
Not Stage 1 evidence. The frozen corpus is thirteen cases with two arms each;
one case with one arm is not a smaller Stage 1 result. Stage 1 remains not
started and snapshot serving remains off.
What the attempt did produce is the first hard data on why the live arm keeps
failing -- the thing v1.18.2's diagnostics exist for, and the thing the
2026-09-02 report could not supply because it preserved only an error string.
The finding is that this was a SLOW 503, not a fast 429:
started_at 2026-09-04T09:45:37.817Z
finished_at 2026-09-04T09:50:09.599Z
elapsed_ms 271785.89
status 503 reason "Service Temporarily Unavailable" headers {}
Rate limiting is normally refused immediately, and 2026-09-02 was a 429. A 503
arriving after 4m32s of waiting is an upstream that could not complete the work
rather than one declining to accept it, so the "throttle" framing -- which came
from a single 429 and was never established -- is now in question.
headers {} is the allow-list RESULT, not an omission: v1.18.2 captures
Retry-After, RateLimit-* and X-RateLimit-*, and HMLR sent none. An honest
negative. There is still no observed quota to pace against, and no Retry-After
to say when a retry becomes reasonable.
Live latency across the three observations now on record: ~58s per call on
2026-09-02 with a 429 on the third; 200 in 172.95s at 09:12 today; 503 after
271.79s at 09:46 today. Three points is a trend line, not a diagnosis, and this
evidence does not separate upstream overload, a throttle expressed as latency,
scheduled load, or something keyed to us we cannot see.
The snapshot arm completed before the live arm failed: 50 rows in 314.61 ms,
containment holding within B5 across sectors B5 4-7. Both figures are single
observations and neither is a p95, so this is not the latency gate and must not
be quoted as one.
Also recorded, unexplained: PricePaidDataClient declares timeout 120
(ppd_client.py:155) yet this call ran 271.8s. Either the timeout is per socket
operation or something retried beneath it. It does not reopen the 2026-08-30
closure, which rests on the loop staying free during a 172.95s call, but it does
mean the blocking window can exceed the declared bound.
The stop condition set in advance -- abort at S1 means stop and do not retry --
was honoured. No second attempt was made.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Not Stage 1 evidence. One case, one arm. The frozen corpus is thirteen cases
with two arms each, and a report covering fewer is not a smaller Stage 1
result. Stage 1 remains not started; snapshot serving remains off.
What the attempt did produce is the first hard data on why the live arm keeps
failing — exactly what v1.18.2's diagnostics were built for, and what the
2026-09-02 report could not supply because it preserved only an error string.
The finding: a slow 503, not a fast 429
1. It took 4 minutes 32 seconds to fail. Rate limiting is normally refused
immediately, and 2026-09-02 was a 429. A 503 after 271.8s of waiting is an
upstream that could not complete the work, not one declining to accept it. The
"throttle" framing came from a single 429 and was never established — it is now
in question.
2.
headers: {}is the allow-list result, not an omission. v1.18.2 capturesRetry-After,RateLimit-*andX-RateLimit-*; HMLR sent none. An honestnegative: there is still no observed quota to pace against, and no
Retry-Afterto say when a retry becomes reasonable.
3. Live latency is degrading across observations.
/v1/ppd/compsB5 4BXThree points is a trend line, not a diagnosis. This evidence does not
separate upstream overload, a throttle expressed as latency, scheduled load on
HMLR's side, or something keyed to us we still cannot see.
The one comparison that did happen
The snapshot arm completed before the live arm failed: 50 rows in 314.61 ms,
containment holding within
B5across sectorsB5 4–B5 7, with the expectedcoverage_clampandfreshnesswarnings.Both figures are single observations and neither is a p95, so this is not the
latency gate and must not be quoted as one. It is the first side-by-side on
identical input, and the direction is not subtle.
Also recorded: a timeout discrepancy
PricePaidDataClientdeclarestimeout: float = 120(
property_core/ppd_client.py:155) yet this call ran 271.8s. Either thetimeout is per socket operation, or something retried beneath it. Unconfirmed
here, flagged for its own investigation.
It does not reopen the 2026-08-30 closure (#62), which rests on the loop
staying free during a 172.95s call — 345 health beats at a 2.3ms mean. But it
does mean the blocking window can exceed the declared bound, which matters to
anyone sizing timeouts or health-check budgets.
Discipline
--live-delay-secondsdeliberately left at the 2026-09-02 value of 2.0.Changing it without data would have confounded the one measurement worth
having — and on this run the gap between calls never came into play, since it
died on the first one.
installed_into_server_state: false,snapshot_routing_enabled: false,artifacts_downloaded: 0,snapshot_written_to: false.45-day bound.
record, do not retry. No second attempt was made.
addressisnull/<omitted>, noprices, no PAON/SAON, no postcodes beyond corpus geographies.
Next, in order
A 503 after 272s and a 429 after two calls are different failures.
future attempt should be bounded.
--live-delay-secondscannot be tuned againstevidence that does not exist.