Skip to content

ops(stage1): capture live-arm status, rate-limit headers and request timing - #57

Merged
paulieb89 merged 1 commit into
mainfrom
ops/stage1-live-arm-diagnostics
Sep 2, 2026
Merged

ops(stage1): capture live-arm status, rate-limit headers and request timing#57
paulieb89 merged 1 commit into
mainfrom
ops/stage1-live-arm-diagnostics

Conversation

@paulieb89

Copy link
Copy Markdown
Owner

The first authorised Stage 1 comparison made three live HMLR requests: S1 and S2 completed, S3 returned HTTP 429. The comparator stopped correctly and did not retry — but its report preserved only "HTTPError: HTTP Error 429: Too Many Requests". The status, the reason and any Retry-After were on the exception at that moment and were discarded, so the question the failure raised — how long to wait before the next authorised run — cannot be answered from the evidence we have.

This adds the diagnostics, and nothing else.

What changed

Two files: tools/ppd_snapshot/stage1_shadow.py and its tests. No file under property_core/, app/, property_app/ or property_cli/ is touched, PPDCompsResponse is not widened, and there is no version, changelog or lockfile change.

1. The capture catches as well as returns. A success-only wrapper records nothing about the one case that needs explaining: a 429 raises out of the transport rather than returning, so the exception is the only place the status and headers exist. LiveEvidenceCapture now describes the failure at that seam and re-raises it untouched — observing a failure must not change what the caller sees, and must not retry it.

2. Failures record status, reason and an allow-listed header subset. Exactly Retry-After, RateLimit-* and X-RateLimit-*, matched case-insensitively, values truncated. An allow-list rather than a deny-list: a failed response's headers are upstream-controlled and can carry cookies and infrastructure detail. The response body is never readHTTPError is itself a readable file object, and pulling upstream content into an evidence file buys no diagnosis. status/reason are null for a non-HTTP failure (a timeout is a different fact from a status of zero); headers is {} when none qualify.

3. Every live observation is timed, on both the success and failure paths. The delay is applied after the call and was dwarfed by it — 2 s against 58 s — so realised spacing can only be read off timestamps, never inferred from live_delay_seconds.

Report schema added

cases[].live_timing        { started_at, finished_at, elapsed_ms, outcome }   // both paths
cases[].live_error_detail  { type, message, status, reason, headers }         // failure only
live_errors[]              { shape, error, type, message, status, reason, headers }

live_error and transport_evidence keep their existing shape.

Proof of failure-path capture (synthetic 429, no network)

"live_errors": [{ "shape": "S1", "status": 429, "reason": "Too Many Requests",
   "headers": { "retry-after": "120", "ratelimit-limit": "100",
                "ratelimit-remaining": "0", "x-ratelimit-reset": "1756820000" } }]
"live_timing": { "started_at": "...", "finished_at": "...", "elapsed_ms": 0.05, "outcome": "error" }
"live_calls_made": 1

Same run, with Set-Cookie, Server, Content-Type and a JSON body present on the response: none reached the report.

Semantics preserved, and asserted

One live observation per case; the existing budget guard unchanged; no retry on 429; no resume-from-case; no automatic backoff; no corpus change. A source-scan test fails if backoff, tenacity, max_retries or a retry loop ever appears on this path. Retry-After is captured so a human can pace the next authorised run — honouring it in-run by waiting and re-issuing would be a second observation for one case, which is the rule this gate exists to hold.

The report's key allow-list gained the new fields deliberately, as its own guard test demands. Allow-listed header names are exempted as a dynamic key space by asking the production allow-list rather than restating it, so a header the comparator would refuse to keep is still an un-reviewed key in the test.

Tests

17 new synthetic tests, no network (the module hard-fails sockets), covering all six required proofs. Focused suite tests/snapshot/test_stage1_shadow.py: 235 passed. ./scripts/validate.sh: exit 0 — 2011 passed, 27 skipped (was 1994).

Scope

No qualify, no compare, no HMLR or live call; no release, deploy or restart; no secret, flag or configuration change. Stage 1 has still not passed and snapshot serving remains off.

Held for review.

…timing

The first authorised Stage 1 run stopped on an HTTP 429 from HMLR and recorded
"HTTPError: HTTP Error 429: Too Many Requests" and nothing else. The status,
the reason and any Retry-After were all on the exception at that moment and
were discarded, so the one question the failure raised -- how long to wait
before the next authorised run -- could not be answered from the evidence.

Three additions, all process-local to the out-of-band comparator:

* LiveEvidenceCapture now catches as well as returns. A success-only wrapper
  records nothing about the case that needs explaining, because a 429 raises
  out of the transport instead of returning; the exception is the only place
  the status and headers exist. It is described at that seam and re-raised
  untouched.

* Failures record status, reason and an ALLOW-LISTED subset of response
  headers: Retry-After, RateLimit-* and X-RateLimit-*, matched case-
  insensitively and truncated. An allow-list rather than a deny-list, because a
  failed response's headers are upstream-controlled and can carry cookies and
  infrastructure detail. The response body is never read -- HTTPError is itself
  a readable file object, and pulling upstream content into an evidence file
  buys no diagnosis. status and reason are null for a non-HTTP failure, which
  is a different fact from a status of zero.

* Every live observation records started_at, finished_at and elapsed_ms, on
  both paths. The delay is applied after the call and was dwarfed by it -- 2s
  against 58s -- so realised spacing between live requests can only be read off
  timestamps, never inferred from live_delay_seconds.

Comparison semantics are unchanged and asserted so: one observation per case,
the same budget guard, no retry on 429, no resume-from-case, no backoff, no
corpus change. Nothing touches the serving application: PPDCompsResponse is not
widened and no file outside tools/ppd_snapshot/stage1_shadow.py changes.

The report's key allow-list gained the new fields deliberately, as its own test
requires. Allow-listed header names are exempted as a dynamic key space by
asking the production allow-list rather than restating it, so a header the
comparator would refuse to keep is still an un-reviewed key in the test.

Seventeen synthetic tests, no network: a 429 with allowed headers is recorded;
arbitrary headers and the body are excluded; the failure consumes exactly one
observation, is not retried, aborts, and still writes a partial report;
successful transport evidence is unchanged; timing permits spacing to be
computed; and the budget guard still refuses past one call per case.
@paulieb89
paulieb89 merged commit 123ba54 into main Sep 2, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant