fix(action): fail closed when the gate can't be evaluated - #61
Merged
Conversation
The Action failed OPEN — the most dangerous false-green of the audit set, since a green check can unblock a merge: - A crashed/absent `warden report aggregate` (subprocess throw, missing binary, unparseable stdout) defaulted the check to PASS "No aggregated results available." → now BLOCK "aggregate failed — gate not evaluated: <msg>". - `normalizeGate` coerced a missing or unrecognized gate decision to PASS (`String(g.decision ?? 'PASS')`) → now only an explicit, recognized PASS/WARN/BLOCK is trusted; anything else fails closed to BLOCK. A gate we could not evaluate must never read as a green merge signal. TDD: 3 red→green tests (unknown decision → BLOCK, missing decision → BLOCK, aggregate crash → BLOCK) plus a guard that an explicit PASS still passes. Bundle refreshed. Full barrier green: 1327 tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
QuintinBotes
added a commit
that referenced
this pull request
Jul 10, 2026
…e-green PASS (#62) Part 2 of the false-green audit follow-up (#60, #61). Each quality tier reported "0 failures → PASS" when it actually ran but measured/found nothing — "did not measure" must not read as "no problems". All WARN-direction: - perf/lighthouse: routes audited but every metric unreadable (runtime-error report → 0 CTRF tests) → WARN. evaluatePerfBudgetGate gains an auditedRouteCount arg (defaulted, so existing calls are unchanged); the caller passes results.length. - load/k6: the run issued zero requests (http_reqs === 0) → WARN. Missing latency/ error metrics normalize to 0 and spuriously pass every threshold, so the guard runs first. evaluateLoadGate now takes the normalized summary. - component: runner collected 0 tests (grep matched nothing / none discovered) → WARN. - api/pact: no interactions verified at all (no contracts found) → WARN. - i18n: gate gains an optional measurement arg — comparedLocaleCount === 0 (default locale missing / nothing to compare) → WARN. Backward-compatible. - cli/cuj-gate: gate enabled but 0 CUJ definitions loaded (bad cuj.dir) → WARN. TDD: 7 new/adjusted tests (component + i18n empty-report cases updated; new WARN tests for each tier) plus positive controls that real measurements still PASS. Full barrier green: 1334 tests, typecheck, build, lint. Action bundle unchanged. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
QuintinBotes
added a commit
that referenced
this pull request
Jul 10, 2026
Correctness patch bundling the false-green gate audit (#60, #61, #62): 16 empty/degenerate gate paths that returned a green PASS now signal honestly — WARN-direction throughout, except the GitHub Action which fails closed to BLOCK. Bumps all workspace package versions 0.4.0 → 0.4.1 and cuts the CHANGELOG section. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The most dangerous false-green of the audit set — the GitHub Action failed open, so a broken gate could post a green check and unblock a merge.
Fix (fail-closed)
warden report aggregate(subprocess throw, missing binary, unparseable stdout) defaulted the check toPASS "No aggregated results available."→ nowBLOCK"aggregate failed — gate not evaluated: <msg>".normalizeGatecoerced a missing/unrecognized gate decision to PASS (String(g.decision ?? 'PASS')) → now only an explicit, recognizedPASS/WARN/BLOCKis trusted; anything else →BLOCK.A gate we can't evaluate must never read as a green merge signal.
Tests (TDD)
3 red→green tests — unknown decision → BLOCK, missing decision → BLOCK, aggregate crash → BLOCK — plus a guard that an explicit PASS still passes. Bundle refreshed. Full barrier: 1327 tests, typecheck, build, lint.
Part 1 of the follow-up to #60 (the systemic false-green audit). Part 2 (quality tiers that ran but measured nothing → WARN, + CUJ-dir) is next.
🤖 Generated with Claude Code