ops(stage1): capture live-arm status, rate-limit headers and request timing - #57
Merged
Merged
Conversation
…timing The first authorised Stage 1 run stopped on an HTTP 429 from HMLR and recorded "HTTPError: HTTP Error 429: Too Many Requests" and nothing else. The status, the reason and any Retry-After were all on the exception at that moment and were discarded, so the one question the failure raised -- how long to wait before the next authorised run -- could not be answered from the evidence. Three additions, all process-local to the out-of-band comparator: * LiveEvidenceCapture now catches as well as returns. A success-only wrapper records nothing about the case that needs explaining, because a 429 raises out of the transport instead of returning; the exception is the only place the status and headers exist. It is described at that seam and re-raised untouched. * Failures record status, reason and an ALLOW-LISTED subset of response headers: Retry-After, RateLimit-* and X-RateLimit-*, matched case- insensitively and truncated. An allow-list rather than a deny-list, because a failed response's headers are upstream-controlled and can carry cookies and infrastructure detail. The response body is never read -- HTTPError is itself a readable file object, and pulling upstream content into an evidence file buys no diagnosis. status and reason are null for a non-HTTP failure, which is a different fact from a status of zero. * Every live observation records started_at, finished_at and elapsed_ms, on both paths. The delay is applied after the call and was dwarfed by it -- 2s against 58s -- so realised spacing between live requests can only be read off timestamps, never inferred from live_delay_seconds. Comparison semantics are unchanged and asserted so: one observation per case, the same budget guard, no retry on 429, no resume-from-case, no backoff, no corpus change. Nothing touches the serving application: PPDCompsResponse is not widened and no file outside tools/ppd_snapshot/stage1_shadow.py changes. The report's key allow-list gained the new fields deliberately, as its own test requires. Allow-listed header names are exempted as a dynamic key space by asking the production allow-list rather than restating it, so a header the comparator would refuse to keep is still an un-reviewed key in the test. Seventeen synthetic tests, no network: a 429 with allowed headers is recorded; arbitrary headers and the body are excluded; the failure consumes exactly one observation, is not retried, aborts, and still writes a partial report; successful transport evidence is unchanged; timing permits spacing to be computed; and the budget guard still refuses past one call per case.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The first authorised Stage 1 comparison made three live HMLR requests: S1 and S2 completed, S3 returned HTTP 429. The comparator stopped correctly and did not retry — but its report preserved only
"HTTPError: HTTP Error 429: Too Many Requests". The status, the reason and anyRetry-Afterwere on the exception at that moment and were discarded, so the question the failure raised — how long to wait before the next authorised run — cannot be answered from the evidence we have.This adds the diagnostics, and nothing else.
What changed
Two files:
tools/ppd_snapshot/stage1_shadow.pyand its tests. No file underproperty_core/,app/,property_app/orproperty_cli/is touched,PPDCompsResponseis not widened, and there is no version, changelog or lockfile change.1. The capture catches as well as returns. A success-only wrapper records nothing about the one case that needs explaining: a 429 raises out of the transport rather than returning, so the exception is the only place the status and headers exist.
LiveEvidenceCapturenow describes the failure at that seam and re-raises it untouched — observing a failure must not change what the caller sees, and must not retry it.2. Failures record status, reason and an allow-listed header subset. Exactly
Retry-After,RateLimit-*andX-RateLimit-*, matched case-insensitively, values truncated. An allow-list rather than a deny-list: a failed response's headers are upstream-controlled and can carry cookies and infrastructure detail. The response body is never read —HTTPErroris itself a readable file object, and pulling upstream content into an evidence file buys no diagnosis.status/reasonarenullfor a non-HTTP failure (a timeout is a different fact from a status of zero);headersis{}when none qualify.3. Every live observation is timed, on both the success and failure paths. The delay is applied after the call and was dwarfed by it — 2 s against 58 s — so realised spacing can only be read off timestamps, never inferred from
live_delay_seconds.Report schema added
cases[].live_timing { started_at, finished_at, elapsed_ms, outcome } // both paths cases[].live_error_detail { type, message, status, reason, headers } // failure only live_errors[] { shape, error, type, message, status, reason, headers }live_errorandtransport_evidencekeep their existing shape.Proof of failure-path capture (synthetic 429, no network)
Same run, with
Set-Cookie,Server,Content-Typeand a JSON body present on the response: none reached the report.Semantics preserved, and asserted
One live observation per case; the existing budget guard unchanged; no retry on 429; no resume-from-case; no automatic backoff; no corpus change. A source-scan test fails if
backoff,tenacity,max_retriesor a retry loop ever appears on this path.Retry-Afteris captured so a human can pace the next authorised run — honouring it in-run by waiting and re-issuing would be a second observation for one case, which is the rule this gate exists to hold.The report's key allow-list gained the new fields deliberately, as its own guard test demands. Allow-listed header names are exempted as a dynamic key space by asking the production allow-list rather than restating it, so a header the comparator would refuse to keep is still an un-reviewed key in the test.
Tests
17 new synthetic tests, no network (the module hard-fails sockets), covering all six required proofs. Focused suite
tests/snapshot/test_stage1_shadow.py: 235 passed../scripts/validate.sh: exit 0 — 2011 passed, 27 skipped (was 1994).Scope
No
qualify, nocompare, no HMLR or live call; no release, deploy or restart; no secret, flag or configuration change. Stage 1 has still not passed and snapshot serving remains off.Held for review.