Skip to content

fix(ci): endpoint-coverage gate is method-aware, wired into CI (F-18) - #276

Merged
anilguleroglu merged 1 commit into
mainfrom
fix/p1-test-quality-gates
Sep 6, 2026
Merged

fix(ci): endpoint-coverage gate is method-aware, wired into CI (F-18)#276
anilguleroglu merged 1 commit into
mainfrom
fix/p1-test-quality-gates

Conversation

@anilguleroglu

Copy link
Copy Markdown
Collaborator

Summary

P1 finding from the 2026-09-05 finance-institution assessment.

  • check-endpoint-coverage.ts couldn't tell coverage from coincidence. It concatenated every test file into one haystack and did fragment matching with no HTTP method check — a GET-only test anywhere in the suite could make a route "covered" for every other method on the same path, and an unrelated file sharing a path fragment could do the same. Rewrote isReferenced to require the method and every path fragment to appear in the same test file. Same stated design goal as before (no route-schema parsing, still cheap) — just method-aware and per-file instead of method-blind and suite-wide.

    This is a stricter, more honest check: total uncovered routes went from 119 (old matcher) to 380 (this one) on the same test suite — not a regression, the old number was simply wrong. Rebaselined accordingly (3 entries the new matcher does find real coverage for were removed). Migrating the ~84 test files that exercise the legacy routes/** tree instead of the live plugins/* Fastify API — a separate, much larger, pre-existing problem this assessment also names — is out of scope here; baselining today's true gap count honestly is the alternative to either quietly keeping the weaker check or taking on that migration inside this PR.

  • Wired npm run test:endpoints into CI as a blocking step. It existed but was never run automatically — a new endpoint could ship with zero same-file, method-matched test and nothing would catch it.

  • client-rag-memory.test.ts asserted a tenant-wide (projectId: undefined) queryRag call as correct. True for the legacy handler it actually calls — but that handler is dead code (routes/**, never registered in plugin.ts), and the equivalent behavior in the live plugin was a real cross-project data leak (F-02, fixed separately in a different PR). Added a file-header note and an inline comment on that specific assertion so a reader can't mistake a green run here for evidence about the live API's project scoping.

Explicitly out of scope for this PR

  • The fastify-api.ts test helper's auth fidelity — it's used by 31 test files; the actual failure mode needs more scoping before touching something this widely relied on.
  • npm audit's continue-on-error: true — needs a time-boxed exception mechanism (an allowlist with expiry) before it's safe to make blocking, not just removing the flag.
  • vitest coverage thresholds — needs an actual coverage run to pick a real, defensible number, not a guess.

Test plan

  • npx tsx scripts/check-endpoint-coverage.ts exits 0 against the rebaselined file
  • npx tsc --noEmit clean
  • npx eslint clean on changed files (one pre-existing unrelated warning left untouched)
  • Full vitest run: 5046 passed, 0 failed, 5 skipped

🤖 Generated with Claude Code

https://claude.ai/code/session_01KQq6TnVNHRNU6Wz1eQzPpD

Three concrete gaps from the assessment:

- check-endpoint-coverage.ts concatenated every test file into one
  haystack and did fragment matching with no HTTP method check -- a
  GET-only test anywhere in the suite could make a route "covered" for
  every other method, and an unrelated file sharing a path fragment
  could do the same. Rewrote isReferenced to require the method AND
  every path fragment to appear in the SAME test file. Still no route-
  schema parsing (same stated design goal as before), just method-
  aware and per-file instead of method-blind and suite-wide.

  This is a stricter, more honest check: total uncovered routes went
  from 119 (old matcher) to 380 (this one) on the same test suite --
  not a regression, the old number was wrong. Rebaselined accordingly;
  3 baseline entries the new matcher DOES find covered were removed.
  Migrating 84 test files that exercise the legacy `routes/**` tree
  instead of the live `plugins/*` Fastify API (a separate, much larger,
  pre-existing problem this assessment also names) is out of scope
  here -- baselining today's true gap count is the honest alternative
  to either silently keeping a weaker check or taking on that rewrite
  in this PR.

- Wired `npm run test:endpoints` into CI as a blocking step. It was
  only ever run manually before; a new endpoint could ship with no
  same-file, method-matched test and nothing would catch it.

- client-rag-memory.test.ts asserts a tenant-wide (`projectId:
  undefined`) queryRag call as CORRECT -- true for the legacy handler
  it actually calls, but that handler is dead code (routes/**, not
  registered in plugin.ts), and the equivalent behavior in the LIVE
  plugin was a real cross-project data leak (F-02, fixed separately).
  Added a file-header note and an inline comment on that specific
  assertion so a reader can't mistake a green run here for evidence
  about the live API's project scoping.

Not addressed here: the `fastify-api.ts` test helper's auth fidelity
(touches 31 test files; the actual failure mode needs more scoping
before changing something that widely used), `npm audit`'s
`continue-on-error: true` (needs a time-boxed exception mechanism
first, not just removing the flag), and vitest coverage thresholds
(needs an actual coverage run to pick a real number, not a guess).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KQq6TnVNHRNU6Wz1eQzPpD
@anilguleroglu
anilguleroglu requested a review from a team September 6, 2026 13:23
@anilguleroglu
anilguleroglu merged commit 41bb533 into main Sep 6, 2026
5 checks passed
@anilguleroglu
anilguleroglu deleted the fix/p1-test-quality-gates branch September 6, 2026 21:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant