fix(runner): quality tiers that measured nothing WARN instead of false-green PASS - #62
Merged
Merged
Conversation
…e-green PASS Part 2 of the false-green audit follow-up (#60, #61). Each quality tier reported "0 failures → PASS" when it actually ran but measured/found nothing — "did not measure" must not read as "no problems". All WARN-direction: - perf/lighthouse: routes audited but every metric unreadable (runtime-error report → 0 CTRF tests) → WARN. evaluatePerfBudgetGate gains an auditedRouteCount arg (defaulted, so existing calls are unchanged); the caller passes results.length. - load/k6: the run issued zero requests (http_reqs === 0) → WARN. Missing latency/ error metrics normalize to 0 and spuriously pass every threshold, so the guard runs first. evaluateLoadGate now takes the normalized summary. - component: runner collected 0 tests (grep matched nothing / none discovered) → WARN. - api/pact: no interactions verified at all (no contracts found) → WARN. - i18n: gate gains an optional measurement arg — comparedLocaleCount === 0 (default locale missing / nothing to compare) → WARN. Backward-compatible. - cli/cuj-gate: gate enabled but 0 CUJ definitions loaded (bad cuj.dir) → WARN. TDD: 7 new/adjusted tests (component + i18n empty-report cases updated; new WARN tests for each tier) plus positive controls that real measurements still PASS. Full barrier green: 1334 tests, typecheck, build, lint. Action bundle unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
QuintinBotes
added a commit
that referenced
this pull request
Jul 10, 2026
Correctness patch bundling the false-green gate audit (#60, #61, #62): 16 empty/degenerate gate paths that returned a green PASS now signal honestly — WARN-direction throughout, except the GitHub Action which fails closed to BLOCK. Bumps all workspace package versions 0.4.0 → 0.4.1 and cuts the CHANGELOG section. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 2 of the false-green audit follow-up (after #60 core gate + #61 Action fail-closed). Each quality tier reported
0 failures → PASSwhen it actually ran but measured/found nothing — "did not measure" must not read as "no problems". All WARN-direction (no currently-passing merge is newly blocked).PASSWARNhttp_reqs === 0)PASSWARNPASSWARNPASSWARNPASSWARNcuj.dir)PASSWARNNotable: k6 with zero requests spuriously passes every threshold (absent metrics normalize to 0), so its guard runs first. Signature tweaks are backward-compatible —
evaluatePerfBudgetGate'sauditedRouteCountis defaulted, and i18n's measurement arg is optional — so existing callers/tests are unchanged.Tests (TDD)
7 new/adjusted tests (component + i18n empty-report cases updated to WARN; new WARN test per tier) plus positive controls that real measurements still PASS. Full barrier: 1334 tests, typecheck, build, lint. Action bundle unchanged.
This closes the systemic false-green audit set from #60 (16 findings → all addressed across #60/#61/this PR).
🤖 Generated with Claude Code