Skip to content

ops(stage1): record the 2026-09-04 attempt, aborted at S1 on a slow upstream 503 - #63

Merged
paulieb89 merged 1 commit into
mainfrom
ops/stage1-attempt-2026-09-04
Sep 4, 2026
Merged

ops(stage1): record the 2026-09-04 attempt, aborted at S1 on a slow upstream 503#63
paulieb89 merged 1 commit into
mainfrom
ops/stage1-attempt-2026-09-04

Conversation

@paulieb89

Copy link
Copy Markdown
Owner

Not Stage 1 evidence. One case, one arm. The frozen corpus is thirteen cases
with two arms each, and a report covering fewer is not a smaller Stage 1
result
. Stage 1 remains not started; snapshot serving remains off.

What the attempt did produce is the first hard data on why the live arm keeps
failing — exactly what v1.18.2's diagnostics were built for, and what the
2026-09-02 report could not supply because it preserved only an error string.

The finding: a slow 503, not a fast 429

"live_timing": {
  "started_at":  "2026-09-04T09:45:37.817+00:00",
  "finished_at": "2026-09-04T09:50:09.599+00:00",
  "elapsed_ms":  271785.89,
  "outcome":     "error"
},
"status": 503, "reason": "Service Temporarily Unavailable", "headers": {}

1. It took 4 minutes 32 seconds to fail. Rate limiting is normally refused
immediately, and 2026-09-02 was a 429. A 503 after 271.8s of waiting is an
upstream that could not complete the work, not one declining to accept it. The
"throttle" framing came from a single 429 and was never established — it is now
in question.

2. headers: {} is the allow-list result, not an omission. v1.18.2 captures
Retry-After, RateLimit-* and X-RateLimit-*; HMLR sent none. An honest
negative: there is still no observed quota to pace against, and no Retry-After
to say when a retry becomes reasonable.

3. Live latency is degrading across observations.

When Request Result
2026-09-02 Stage 1 live arm ~58s per call; 429 on the third
2026-09-04 09:12 /v1/ppd/comps B5 4BX 200 in 172.95s
2026-09-04 09:46 Stage 1 live arm S1 503 after 271.79s

Three points is a trend line, not a diagnosis. This evidence does not
separate upstream overload, a throttle expressed as latency, scheduled load on
HMLR's side, or something keyed to us we still cannot see.

The one comparison that did happen

The snapshot arm completed before the live arm failed: 50 rows in 314.61 ms,
containment holding within B5 across sectors B5 4B5 7, with the expected
coverage_clamp and freshness warnings.

Both figures are single observations and neither is a p95, so this is not the
latency gate and must not be quoted as one
. It is the first side-by-side on
identical input, and the direction is not subtle.

Also recorded: a timeout discrepancy

PricePaidDataClient declares timeout: float = 120
(property_core/ppd_client.py:155) yet this call ran 271.8s. Either the
timeout is per socket operation, or something retried beneath it. Unconfirmed
here, flagged for its own investigation.

It does not reopen the 2026-08-30 closure (#62), which rests on the loop
staying free during a 172.95s call — 345 health beats at a 2.3ms mean. But it
does mean the blocking window can exceed the declared bound, which matters to
anyone sizing timeouts or health-check budgets.

Discipline

  • --live-delay-seconds deliberately left at the 2026-09-02 value of 2.0.
    Changing it without data would have confounded the one measurement worth
    having — and on this run the gap between calls never came into play, since it
    died on the first one.
  • Isolation confirmed by the run itself: installed_into_server_state: false,
    snapshot_routing_enabled: false, artifacts_downloaded: 0,
    snapshot_written_to: false.
  • Instance sha256 verified byte-identical after upload; 2 days old against a
    45-day bound.
  • Stop condition honoured. It was set in advance: abort at S1 means stop,
    record, do not retry. No second attempt was made.
  • Report hygiene-checked before commit — address is null/<omitted>, no
    prices, no PAON/SAON, no postcodes beyond corpus geographies.

Next, in order

  1. Establish what the upstream is actually doing before any further attempt.
    A 503 after 272s and a 429 after two calls are different failures.
  2. Settle the timeout discrepancy — cheap, local, and it changes how a
    future attempt should be bounded.
  3. Only then choose pacing. --live-delay-seconds cannot be tuned against
    evidence that does not exist.

…pstream 503

Not Stage 1 evidence. The frozen corpus is thirteen cases with two arms each;
one case with one arm is not a smaller Stage 1 result. Stage 1 remains not
started and snapshot serving remains off.

What the attempt did produce is the first hard data on why the live arm keeps
failing -- the thing v1.18.2's diagnostics exist for, and the thing the
2026-09-02 report could not supply because it preserved only an error string.

The finding is that this was a SLOW 503, not a fast 429:

    started_at  2026-09-04T09:45:37.817Z
    finished_at 2026-09-04T09:50:09.599Z
    elapsed_ms  271785.89
    status 503  reason "Service Temporarily Unavailable"  headers {}

Rate limiting is normally refused immediately, and 2026-09-02 was a 429. A 503
arriving after 4m32s of waiting is an upstream that could not complete the work
rather than one declining to accept it, so the "throttle" framing -- which came
from a single 429 and was never established -- is now in question.

headers {} is the allow-list RESULT, not an omission: v1.18.2 captures
Retry-After, RateLimit-* and X-RateLimit-*, and HMLR sent none. An honest
negative. There is still no observed quota to pace against, and no Retry-After
to say when a retry becomes reasonable.

Live latency across the three observations now on record: ~58s per call on
2026-09-02 with a 429 on the third; 200 in 172.95s at 09:12 today; 503 after
271.79s at 09:46 today. Three points is a trend line, not a diagnosis, and this
evidence does not separate upstream overload, a throttle expressed as latency,
scheduled load, or something keyed to us we cannot see.

The snapshot arm completed before the live arm failed: 50 rows in 314.61 ms,
containment holding within B5 across sectors B5 4-7. Both figures are single
observations and neither is a p95, so this is not the latency gate and must not
be quoted as one.

Also recorded, unexplained: PricePaidDataClient declares timeout 120
(ppd_client.py:155) yet this call ran 271.8s. Either the timeout is per socket
operation or something retried beneath it. It does not reopen the 2026-08-30
closure, which rests on the loop staying free during a 172.95s call, but it does
mean the blocking window can exceed the declared bound.

The stop condition set in advance -- abort at S1 means stop and do not retry --
was honoured. No second attempt was made.
@paulieb89
paulieb89 merged commit 91d2692 into main Sep 4, 2026
1 check passed
@paulieb89
paulieb89 deleted the ops/stage1-attempt-2026-09-04 branch September 4, 2026 15:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant