[ci-gap][ci-health] Enforce test-discovery contract: every test-declaring file must be discovered by a runner

[claude]

Current state

Coverage 3/5, enforcement 4/5 → target 5/5.

Test discovery in this repo is suffix-exact and completely unguarded:

• jest.config.ts:14-18 — testMatchMap binds each TEST_ENV to a rigid glob: server → <rootDir>/tests/apollo-server/**/*.test.{ts,mts}, integration → <rootDir>/tests/integration/**/*.integration.test.{ts,tsx}, default (client) → <rootDir>/tests/unit/**/*.test.{ts,tsx,js,jsx}.
• playwright.config.ts — testMatch: ['**/*.spec.ts'] under the CLI-passed directories (tests/e2e, tests/visual).

Nothing anywhere verifies that a file which declares tests is actually matched by one of these globs. A spec named tests/e2e/modules/login-form.test.ts (the natural Jest-habit suffix) or tests/integration/auth-flow.test.tsx (missing the .integration infix) type-checks (tsconfig includes tests/**), lints clean, and never executes in any suite. CI goes green while certifying a test that has never run.

Partial indirect cover exists but is honest about its holes: the dual-100% coverage threshold in jest.config.ts:23-30 catches a dead Jest file only when it was the sole coverage source for some line. Assertion-depth tests over already-covered lines (edge cases, regression repros) die with zero signal, and Playwright has no coverage instrumentation at all. A future edit that accidentally narrows testMatch (or the rootsMap) and zeroes out a whole suite is equally invisible today.

The repo already has the right home for this invariant: the bats meta-test harness (tests/bats/*.bats, inventory contract in tests/bats/make-target-coverage.tsv) runs on every PR to main via .github/workflows/bats-testing.yml (make test-bats BATS_FORMATTER=tap, in the dev container). The discovery contract is missing from it.

Defect class prevented

Silently-unexecuted tests. A reviewer-approved PR adds tests/e2e/modules/password-reset.test.ts or tests/integration/token-refresh.test.tsx; every gate passes; the test has never run once. The green check is a false certificate — the regression the test was written to pin can ship at any time. The same class covers config drift: a testMatch/roots edit that quietly de-discovers an entire suite while its workflow exits green on "no tests found" semantics.

Evidence

No direct incident evidence yet — preventive, grounded in the coverage score.

Benchmark: Chromium presubmit verifies test files are referenced by build targets, and Bazel's test-discovery model makes an unreferenced test a build error. This gate retrofits the same "declared implies executed" invariant onto glob-based runners.

Proposed check

Tool: bats meta-test using jest --listTests (per TEST_ENV) and playwright test --list --reporter=json inside the dev container. Listing loads config only — no browsers needed.

Trigger: pull_request — rides the existing bats-testing.yml gate via make test-bats. No new workflow or job.

Mechanics:

1. Enumerate candidate files with grep -rlE '^\s*(test|it|describe)(\.\w+)?\(' over the five test roots (the (\.\w+)? group covers .each/.skip/.only/.describe modifiers, so no separate test\.describe alternative is needed). Helpers/builders/constants/utils under tests/e2e don't invoke these at top level — day-one false positives are zero.
2. Build the discovered set from three jest --listTests runs and one Playwright JSON listing. Normalize paths explicitly: jest prints absolute container paths (strip the repo root); Playwright JSON prints config-root-relative paths (config lives at repo root, so already repo-relative). Playwright's prod mode lists 3 projects (chromium, firefox, webkit), so the same file appears 3× — sort -u dedupes. Because every e2e/visual spec uses test.describe, specs nest inside child suites in the JSON — extract .file recursively, never with jq -r '.suites[].specs[].file'.
3. Fail printing orphan paths on any set difference; also assert each discovery command returns a non-zero count (guards against a config edit zeroing out a whole suite).
4. Register the new bats file in tests/bats/make-target-coverage.tsv per the existing harness-inventory contract.

#!/usr/bin/env bats
# tests/bats/test_discovery_contract.bats
#
# Contract: every file under the five test roots that declares tests
# (top-level `test(` / `it(` / `describe(`, incl. `.each`/`.skip`/`.only`
# modifiers via the `(\.\w+)?` group) must be discovered by a runner.
#
# Known future false-positive vector — do NOT widen the grep to fix it:
# a shared-suite helper (describe/it inside an exported function) placed
# under these roots would flag. Repo convention keeps such helpers in
# tests/utils/ and tests/builders/; the fix is moving the file, never
# editing this grep.

load test_helper

repo_root() { git rev-parse --show-toplevel; }

# jest --listTests prints absolute container paths; strip to repo-relative.
list_jest() {
  TEST_ENV="$1" bun x jest --listTests 2>/dev/null | sed "s|^$(repo_root)/||"
}

# Every e2e/visual spec uses test.describe, so the JSON reporter nests specs
# inside child suites — extract `.file` recursively (a flat
# `.suites[].specs[].file` would miss them and produce false orphans).
# `.file` is config-root-relative = repo-relative (config sits at repo root).
# Falls back to bun when jq is absent from the dev image.
list_playwright() {
  local json
  json=$(bun x playwright test --list --reporter=json tests/e2e tests/visual)
  if command -v jq >/dev/null 2>&1; then
    printf '%s' "$json" | jq -r '[.. | objects | .file? // empty] | .[]'
  else
    printf '%s' "$json" | bun -e '
      const seen = new Set();
      const walk = (n) => { if (n && typeof n === "object") {
        if (typeof n.file === "string") seen.add(n.file);
        Object.values(n).forEach(walk); } };
      walk(JSON.parse(await Bun.stdin.text()));
      console.log([...seen].join("\n"));'
  fi
}

@test "every test-declaring file is discovered by a runner" {
  declared=$(grep -rlE '^\s*(test|it|describe)(\.\w+)?\(' \
    tests/unit tests/integration tests/apollo-server tests/e2e tests/visual \
    | sort -u)
  # sort -u also dedupes the 3 prod Playwright projects listing each spec 3x.
  discovered=$( { list_jest client; list_jest integration; list_jest server; \
                  list_playwright; } | sed 's|^\./||' | sort -u)
  orphans=$(comm -23 <(printf '%s\n' "$declared") <(printf '%s\n' "$discovered"))
  [ -z "$orphans" ] || { echo "Undiscovered test files:"; echo "$orphans"; false; }
}

@test "no runner discovers zero tests (discovery not silently narrowed)" {
  for env in client integration server; do
    [ "$(list_jest "$env" | grep -c .)" -gt 0 ]
  done
  [ "$(list_playwright | grep -c .)" -gt 0 ]
}

Registration row appended to tests/bats/make-target-coverage.tsv (per the existing contract), and the file is picked up automatically by the recursive bun x bats -r tests/bats in the test-bats Make target — the existing bats-testing.yml required check enforces it with no workflow change.

Overlap notes: complements — does not duplicate — test-quality lint gates: eslint-plugin-jest/playwright lint only files their configs glob-match, and an undiscovered file is precisely the file no runner and no test-linter glob sees. Distinct from E2E route-coverage inventory (route surface vs file discovery).

Effort

M — the bats harness, dev-container plumbing, and PR workflow already exist; the work is the two list adapters (jest per-env, Playwright recursive JSON extraction with the bun fallback), explicit path normalization, the tsv registration, and a seeded-defect verification run.

Acceptance criteria

• tests/bats/test_discovery_contract.bats exists, is registered in tests/bats/make-target-coverage.tsv, and runs on every pull_request to main via the existing bats-testing.yml → make test-bats path.
• The bats testing check is required/blocking for merge (branch protection required status or merge queue includes it).
• The check has failed at least once on a deliberately seeded defect — e.g. a temporary tests/e2e/seeded-orphan.test.ts containing a top-level describe( (wrong suffix, discovered by no runner) — proving it fires, before the seed is removed.

---

Generated by Claude Code

[coderabbitai]

🔗 Related PRs

#15 - chore(#1): add mockoon along with dockerfile for dev [closed]
#30 - Tests memory leak for authentication page [closed]
#98 - feat(#91): enforce DRY with a jscpd duplication gate + dedupe notification styles [closed]
#127 - feat(#17): dev-container Playwright targets (test-e2e-dev, test-visual-dev, debug) [merged]
#134 - ci(#120): shard mutation testing and add concurrency to cut PR runtime [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!