ci(notice) + test(connscale): DAST nightly notified nobody, empty_claims SLO recorded only excursions (BACKLOG #318, #1211 limb one)

.github/workflows/nightly-notice.yml

@@ -20,14 +20,23 @@ name: Nightly failure notice
 # SCOPE: schedule-only, deliberately. A push/PR failure is already visible on the PR itself; alerting
 # there would be pure noise, and `workflow_run` fires for every CI completion regardless of trigger.
 #
-# TWO WORKFLOWS, ONE ISSUE PER WORKFLOW. `Security` was added because its daily cron carries jobs that
-# do not run on a PR -- `released-line-audit` most of all, whose whole subject (the LATEST RELEASE) no
-# PR can change. The issue title is DERIVED from the completed workflow's name, so a green nightly CI
-# cannot close an issue opened by a red Security run: a single shared title would let one signal close
-# the other, which is the same silence this workflow exists to end.
+# THREE WORKFLOWS, ONE ISSUE PER WORKFLOW. `Security` was added because its daily cron carries jobs
+# that do not run on a PR -- `released-line-audit` most of all, whose whole subject (the LATEST
+# RELEASE) no PR can change. `DAST` was added for a stronger version of the same reason (BACKLOG
+# #318): it has NO `pull_request` trigger AT ALL, deliberately, so before this every DAST finding
+# surfaced in the Actions tab and nowhere else -- an authenticated authorization sweep reporting into
+# the void, which is precisely the shape this workflow exists to end.
+#
+# A WATCHED WORKFLOW MUST HAVE A `schedule:` TRIGGER. The job below fires only when the completed
+# run's event was `schedule`, so watching a workflow with no cron adds a name that can never match:
+# dead config that reads as coverage. `tests/test_nightly_notice.py` pins this for every watched name.
+#
+# The issue title is DERIVED from the completed workflow's name, so a green nightly CI cannot close an
+# issue opened by a red Security or DAST run: a single shared title would let one signal close
+# another, which is the same silence this workflow exists to end.
 on:
   workflow_run:
-    workflows: ["CI", "Security"]
+    workflows: ["CI", "Security", "DAST"]
     types: [completed]
 
 # Read-only by default; the one job that writes escalates to `issues: write` and nothing else.
@@ -93,7 +102,7 @@ jobs:
             exit 0
           fi
 
-          BODY="Nightly (scheduled) CI failed.
+          BODY="Nightly (scheduled) $WF_NAME failed.
 
           - run: $RUN_URL
           - commit: \`$HEAD_SHA\`
@@ -102,7 +111,8 @@ jobs:
           A scheduled run is not a PR context, so this failure appears nowhere else. For CI that is the
           server-DB store, load/throughput and service-smoke legs the three required \`test\` legs skip;
           for Security it is the daily dependency audits and the released-line audit, whose subject is a
-          published release that no pull request can change.
+          published release that no pull request can change; for DAST it is the entire workflow, which
+          carries no \`pull_request\` trigger by design.
 
           This issue is opened once and commented on each subsequent failure; it closes itself when a
           nightly goes green again."

harness/load/connscale/report.py

@@ -24,6 +24,8 @@
 from harness._spreadsheet import SPREADSHEET_FORMULA_TRIGGERS, spreadsheet_safe
 
 if TYPE_CHECKING:
+    from collections.abc import Callable
+
     from harness.load.connscale.compare import (
         ClaimModeComparison,
         FuseModeComparison,
@@ -244,6 +246,88 @@ def to_json_dict(self) -> dict[str, object]:
         }
 
 
+#: Hard cap on rows a CI step summary may carry. An oversized ``$GITHUB_STEP_SUMMARY`` write is dropped
+#: ENTIRELY rather than trimmed, so a large profile must lose rows instead of losing the whole surface.
+#: Truncation is always stated in the rendered text -- never a silent cap.
+_MAX_SUMMARY_ROWS = 200
+
+
+def lane_label(sweep_mode: str, claim_mode: str) -> str:
+    """The lane's name as every connscale reading spells it.
+
+    ``per_lane`` is the default claim mode and stays unqualified, so a pre-existing SLO detail string
+    is unchanged. Defined once because the emitter and the SLO must agree: a reading filed under
+    ``fixed_per_conn`` that a failure reports as ``fixed_per_conn/pooled`` is two distributions.
+    """
+    return sweep_mode if claim_mode == "per_lane" else f"{sweep_mode}/{claim_mode}"
+
+
+@dataclass(frozen=True)
+class MonotonicPair:
+    """One consecutive smaller-N-to-larger-N comparison inside a single lane.
+
+    The lane is ``(sweep_mode, claim_mode)`` — grouping by ``sweep_mode`` alone would chain a pooled
+    reading onto a per_lane one, which BACKLOG #1101 records as wrong independently of whether a
+    shipped profile currently triggers it.
+
+    This is the SINGLE definition of the pairing. :func:`monotonic_pairs` is read both by the SLO that
+    fails on an excursion and by the emitter that records every reading, so the numbers a passing run
+    reports and the numbers a failing run reports cannot drift apart.
+    """
+
+    label: str  # ``sweep_mode``, or ``sweep_mode/claim_mode`` when the claim mode is not per_lane
+    count: int  # the LARGER N — the reading being judged
+    value: float  # the metric at that N
+    prior: float  # the metric at the previous N in the same lane
+    tolerance_floor: float  # the FRACTION, e.g. 0.75 — what the detail string prints after "*"
+    threshold: float  # ``prior * tolerance_floor`` — the number ``value`` must actually beat
+    ok: bool
+
+
+def monotonic_pairs(
+    records: list[ConnScaleRecord],
+    key: Callable[[ConnScaleRecord], float | int | None],
+    *,
+    tolerance: float,
+) -> list[MonotonicPair]:
+    """Every consecutive comparison the loose monotonicity smoke makes, passing ones included.
+
+    The SLO only ever needed the violations. An excursion-only record cannot establish the metric's
+    variance, because the sample is selected on having already left the band (BACKLOG #1211) — so this
+    returns the whole sequence and lets the caller decide which half it wants.
+
+    Readings of ``None`` are skipped rather than failed, and a skipped reading does not become the
+    ``prior`` for the next N.
+    """
+    by_lane: dict[tuple[str, str], list[ConnScaleRecord]] = {}
+    for r in records:
+        by_lane.setdefault((r.sweep_mode, r.claim_mode), []).append(r)
+
+    floor = 1.0 - tolerance
+    pairs: list[MonotonicPair] = []
+    for (mode, claim_mode), rs in by_lane.items():
+        prev_val: float | None = None
+        for r in sorted(rs, key=lambda r: r.count):
+            val = key(r)
+            if val is None:
+                continue
+            v = float(val)
+            if prev_val is not None:
+                pairs.append(
+                    MonotonicPair(
+                        label=lane_label(mode, claim_mode),
+                        count=r.count,
+                        value=v,
+                        prior=prev_val,
+                        tolerance_floor=floor,
+                        threshold=prev_val * floor,
+                        ok=not (v < prev_val * floor),
+                    )
+                )
+            prev_val = v
+    return pairs
+
+
 @dataclass(frozen=True)
 class ConnScaleReport:
     profile: str
@@ -301,6 +385,74 @@ def to_json_dict(self) -> dict[str, object]:
     def to_json(self) -> str:
         return json.dumps(self.to_json_dict(), indent=2)
 
+    def render_readings_markdown(
+        self,
+        metric: str,
+        key: Callable[[ConnScaleRecord], float | int | None],
+        *,
+        tolerance: float,
+        context: dict[str, str] | None = None,
+        max_rows: int = _MAX_SUMMARY_ROWS,
+    ) -> str:
+        """Every reading of one monotonic metric as a markdown table — the passing ones included.
+
+        BACKLOG #1211 needs the metric's true variance, and the SLO records a number only when it has
+        already left the band. A sample selected on having excursioned cannot measure the distribution
+        it excursioned from, so this renders the whole sequence on every run.
+
+        Pure: returns the text and writes nothing. Row count is capped because an oversized
+        ``$GITHUB_STEP_SUMMARY`` write is dropped in full rather than trimmed; a truncation SAYS so.
+        """
+        pair_by_row = {
+            (p.label, p.count): p for p in monotonic_pairs(self.records, key, tolerance=tolerance)
+        }
+        rows: list[str] = []
+        dropped = 0
+        for r in sorted(self.records, key=lambda r: (r.sweep_mode, r.claim_mode, r.count)):
+            val = key(r)
+            if val is None:
+                continue
+            if len(rows) >= max_rows:
+                dropped += 1
+                continue
+            label = lane_label(r.sweep_mode, r.claim_mode)
+            pair = pair_by_row.get((label, r.count))
+            if pair is None:
+                # First reading in its lane: a real sample, but nothing to compare it against.
+                rows.append(f"| {label} | {r.count} | {float(val):.4g} | | | | first in lane |")
+            else:
+                margin = pair.value - pair.threshold
+                verdict = "within band" if pair.ok else "OUTSIDE BAND"
+                rows.append(
+                    f"| {label} | {r.count} | {pair.value:.4g} | {pair.prior:.4g} "
+                    f"| {pair.threshold:.4g} | {margin:+.4g} | {verdict} |"
+                )
+
+        head = [f"### connscale {metric} readings"]
+        ctx = {
+            "profile": self.profile,
+            "db_backend": self.db_backend or "sqlite",
+            **(context or {}),
+        }
+        head.append("")
+        head.append(" | ".join(f"{k}: {v}" for k, v in ctx.items()))
+        head.append("")
+        head.append(
+            f"Recorded on every run, pass or fail (BACKLOG #1211). Band is prior * "
+            f"{1.0 - tolerance:.2f}; a positive margin is inside it."
+        )
+        head.append("")
+        if not rows:
+            head.append(f"No {metric} reading was produced by this run.")
+            return "\n".join(head) + "\n"
+        head.append(f"| lane | N | {metric} | prior | band floor | margin | verdict |")
+        head.append("|---|---|---|---|---|---|---|")
+        out = head + rows
+        if dropped:
+            out.append("")
+            out.append(f"{dropped} further row(s) not shown: capped at {max_rows}.")
+        return "\n".join(out) + "\n"
+
     def to_csv(self) -> str:
         """One row per (sweep_mode, N) step — for spreadsheet curve plotting."""
         buf = io.StringIO()

harness/load/connscale/runner.py

@@ -51,6 +51,7 @@
     ConnScaleReport,
     NoLoss,
     SloCheck,
+    monotonic_pairs,
 )
 from harness.load.corpus import Corpus, build_corpus
 from harness.load.correlator import Correlator
@@ -1206,31 +1207,19 @@ def _monotonic_slo(  # type: ignore[no-untyped-def]
     timing-derived counters on noisy CI runners (mf-ci-test-flakes), so a small dip is jitter, not a
     regression. Fails only on a real drop (``v < prior * (1 - tolerance)``). Missing readings (None) are
     skipped, not failed."""
-    ok = True
-    detail_parts: list[str] = []
-    # Group by (sweep_mode, claim_mode), NOT sweep_mode alone (BACKLOG #1101). Chaining prev_val across
-    # claim modes compares per_lane against pooled, and compare.py:22-25 states pooled's empty-claim
-    # rate SHOULD be materially lower — so a correct engine would fail this the moment a profile set
-    # claim_modes = ["per_lane", "pooled"]. No shipped profile does, which is the only reason it has
-    # never fired; the grouping is wrong independently of that.
-    by_mode: dict[tuple[str, str], list[ConnScaleRecord]] = {}
-    for r in records:
-        by_mode.setdefault((r.sweep_mode, r.claim_mode), []).append(r)
-    floor = 1.0 - tolerance
-    for (mode, claim_mode), rs in by_mode.items():
-        ordered = sorted(rs, key=lambda r: r.count)
-        prev_val: float | None = None
-        for r in ordered:
-            val = key(r)
-            if val is None:
-                continue
-            v = float(val)
-            if prev_val is not None and v < prev_val * floor:
-                ok = False
-                label = mode if claim_mode == "per_lane" else f"{mode}/{claim_mode}"
-                detail_parts.append(
-                    f"{label}@N={r.count}: {v:.3g} < prior {prev_val:.3g} * {floor:.2f}"
-                )
-            prev_val = v
-    observed = "monotonic" if ok else "; ".join(detail_parts)
-    return SloCheck(name, f"non-decreasing vs N (±{int(tolerance * 100)}% jitter)", observed, ok)
+    # The pairing itself — including the group-by-(sweep_mode, claim_mode) rule BACKLOG #1101 records
+    # as load-bearing — lives in `monotonic_pairs`. The emitter that records EVERY reading reads the
+    # same function, so a passing run and a failing run cannot describe the sequence differently.
+    pairs = monotonic_pairs(records, key, tolerance=tolerance)
+    breaches = [p for p in pairs if not p.ok]
+    observed = (
+        "monotonic"
+        if not breaches
+        else "; ".join(
+            f"{p.label}@N={p.count}: {p.value:.3g} < prior {p.prior:.3g} * {p.tolerance_floor:.2f}"
+            for p in breaches
+        )
+    )
+    return SloCheck(
+        name, f"non-decreasing vs N (±{int(tolerance * 100)}% jitter)", observed, not breaches
+    )