ci(notice) + test(connscale): DAST nightly notified nobody, empty_claims SLO recorded only excursions (BACKLOG #318, #1211 limb one) - #594
Open
wshallwshall wants to merge 2 commits into
Conversation
…the excursions (BACKLOG #1211) The `empty_claims_monotonic` SLO writes a number into `observed` only once a reading has already left its band. A passing run records the literal string "monotonic" and discards every value, so the only samples that ever survived a run were the excursions -- and a sample selected on having excursioned cannot measure the distribution it excursioned from. #1211 requires that variance before anyone may touch the band, and the readings to establish it were being produced on every CI run and thrown away. This records them. The emission sits in the module fixture, which runs before any assertion, so a passing run and a failing one are recorded alike. WHY THE STEP SUMMARY AND NOT A PRINT. A bare `print` in the smoke test does not work: pytest captures a PASSING test's stdout and shows it only on failure, which leaves the tail-only sampling exactly as it was. Measured on a throwaway repo -- the sample line was absent from a green run's output. `$GITHUB_STEP_SUMMARY` renders on pass and fail alike, is already the idiom in 19 workflow files here, and needs no ci.yml change, because CI runs connscale through pytest rather than through a step this could hang an upload on. NOTHING HERE WIDENS THE BAND. That is limb two and it stays blocked until the samples exist. No constant in this change is a judgement about what the band should be. The pairing is EXTRACTED rather than copied. `monotonic_pairs` and `lane_label` are now the single definition of how readings are grouped into lanes and compared, read by both the SLO that fails on an excursion and the emitter that records every reading -- so the numbers a green run reports and the numbers a red run reports cannot drift apart. The group-by-(sweep_mode, claim_mode) rule that BACKLOG #1101 records as load-bearing moves with it. `_monotonic_slo` keeps its exact detail string: a differential over 4,648 cases (exhaustive over orderings of None / zero / ties / the exact band boundary, plus 4,000 randomized multi-lane sets) found zero divergence from the old inline implementation, and the differential was shown able to fail against a deliberately broken variant. Verified end to end: a real passing `tests/test_connscale_smoke.py` run wrote four readings to a step summary and left a prior step's content intact. Seven mutants, all killed, each by a distinct set of tests: emit only the excursions; hard-code the band instead of reading the tolerance; drop the claim-mode qualification from the lane label; render an undefined reading as zero; cap the table silently; truncate the summary instead of appending; swallow a failed write. Every mutant asserted a unique anchor and a changed file hash before scoring, and every restore was byte-identical. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…med the wrong workflow (BACKLOG #318) #318 carries this as a tracked follow-up rather than deferred scope: `dast.yml` has NO `pull_request` trigger, deliberately, and is not a required context, so every finding from the authenticated authorization sweep surfaced in the Actions tab and nowhere else. An authenticated security sweep reporting into the void is the exact shape `nightly-notice.yml` exists to end -- it just was not watching it. Widened `workflows:` to include DAST, which is the fix the item names. ALSO FIXED, because widening made it worse rather than introducing it: the issue BODY opened with a hardcoded "Nightly (scheduled) CI failed." whatever had actually run. The TITLE was already derived from $WF_NAME; the body was not, so a red Security run has been opening an issue whose first line names CI. With a third watched workflow that is a reader deciding what broke from a wrong sentence. It now reads from the same $WF_NAME the title does. NEW GUARD, one level up from the item. A watched name that no workflow answers to, or one whose workflow has no `schedule:` trigger, can never match: the notice job fires only when the completed run's event was `schedule`, so such an entry sits in the list looking like protection and matches nothing, forever, silently. That is the notice's own failure mode turned on itself. `test_every_watched_workflow_exists_and_can_actually_fire` asserts both arms for EVERY watched name, so a fourth workflow added later cannot be added wrongly. Its scan carries a positive control, because an empty name map would make every assertion in it vacuous. The module docstring claimed it "pins the three ways" the notice could stop working; there were already six tests. A count in prose has no checker and has to be maintained by whoever adds the next test, so it now states the kind rather than the number (SDS-3.6). Four mutants, all killed: drop DAST from the list; watch a name no workflow answers to; watch a real workflow that has no cron; restore the hardcoded CI in the body. Every mutant asserted a unique anchor and a changed file hash before scoring, and every restore was byte-identical. Two share a test by design -- they are its two arms -- and their failure messages were checked to differ, the typo case listing every name actually present. Scope: the nightly-notice widening only. Increment 2 of #318 (schema-driven breadth, the MLLP/TCP/X12 ingress fuzzing, the /ui plane, a TLS black-box target) is untouched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
wshallwshall
enabled auto-merge (squash)
August 25, 2026 20:12
wshallwshall
disabled auto-merge
August 25, 2026 20:15
wshallwshall
enabled auto-merge (squash)
August 25, 2026 20:16
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two commits from Builder 2, verified independently before pushing (tests run locally, ruff clean)
rather than taken on the handoff's word.
#318:
dast.ymlhas nopull_requesttrigger, deliberately, and isn't a required context, soevery finding from the authenticated authorization sweep surfaced in the Actions tab and nowhere
else.
nightly-notice.ymlnow watches it. Also fixed in the same pass: the issue body hardcoded"Nightly (scheduled) CI failed." regardless of which workflow actually ran -- it now reads
$WF_NAME, same as the title already did. New guard: a watched name that resolves to no workflow,or to one with no
schedule:trigger, can never match the notice job's schedule gate, so it wouldsit in the list looking like protection and matching nothing --
test_every_watched_workflow_exists_and_can_actually_firepins both arms for every watched name.
#1211 limb one: the
empty_claims_monotonicSLO recorded a reading only once it had alreadyleft its band, so a passing run kept nothing and the only samples that ever survived were the
excursions. Now records every reading on every run via
GITHUB_STEP_SUMMARY, from the modulefixture (runs before any assertion, so green and red are recorded alike). Does not widen the band
-- that's limb two and stays blocked; no constant here is a judgement about the band.
_monotonic_slowas refactored onto a sharedmonotonic_pairshelper as part of this; its detailstring is byte-identical, checked with a 4,648-case differential against the old inline version
(0 divergence).
Neither commit touches engine code. Verified locally: 30/30 in
test_connscale_empty_claims_per_msg.py+test_nightly_notice.py, ruff check clean on allchanged Python files.
Co-Authored-By: Claude Opus 5 noreply@anthropic.com