62 Playwright tests in 14 files have never run in CI — playwright-test-path scopes the job to tests/e2e/spec-coverage

[rubenvdlinde]

What

.github/workflows/code-quality.yml:111 sets

playwright-test-path: "tests/e2e/spec-coverage"

so the E2E job only ever collects the 7 files under that directory. The job's own log states the scope it chose:

Found 7 Playwright test file(s) in tests/e2e/spec-coverage.

Everything at the tests/e2e/ root is outside it. That is 14 spec files containing 62 tests:

agent-detail-and-chrome  dashboard-and-agents  docs-screenshots  flow-builder-dialect
skill-bundle  skill-evals  skill-learnings  skill-maturity  skill-multifile-install
skill-provenance  skill-self-improvement  sync-conversion-probe  tutorial-captures
wave2-surfaces

Why this matters more than a missing-coverage ticket

They are not failing. They are unexecuted — and from outside the job, an unexecuted test is indistinguishable from a passing one. The E2E check reports green, the PR page shows a tick, and 62 assertions about the flow builder, the dashboard, the agent detail page and the whole skill surface contribute nothing to it.

This was found while verifying hermiq#175, which added two tests to flow-builder-dialect.spec.ts and got a green E2E job that never opened the file. The green was about to be read as evidence for a change it had not touched.

Not a one-line fix

Repointing the path at tests/e2e would collect all 62 immediately, and several would fail for reasons unrelated to the code under test:

• flow-builder-dialect.spec.ts addresses a flow by a UUID that exists on a local dev instance; CI seeds the Hydra Triage flow instead (spec-coverage/flow-seed.spec.ts)
• docs-screenshots.spec.ts and tutorial-captures.spec.ts are capture tools rather than assertions, and probably should not gate a PR at all

So the work is per-file triage into three buckets:

1. make it runnable in CI — replace instance-specific fixtures with what the seed provides, then move it under spec-coverage/
2. explicitly local-only — keep it at the root, and document at the top of the file that CI never runs it, so nobody reads a green E2E as covering it
3. not a test — move captures out of the *.spec.ts collection entirely

Suggested acceptance

• Every file under tests/e2e/ is in exactly one of the three buckets above, and each local-only file says so in its header.
• The E2E job's collected-file count is asserted, so a path that silently collects fewer files fails instead of passing quietly.

That last point is the part that keeps this from recurring: the current setup has no way to notice that it stopped looking.

[rubenvdlinde]

Measured in gate-19's own currency: this is worth 58 scenarios, and it is the only thing standing between hermiq and a non-zero e2e coverage figure

Two arms, one tree (5189fe3a), canonical .github @ 57bcb2b, with only the
resolved Playwright config changed:

arm	live @e2e refs	dead refs	covered	excluded	uncovered
tests/e2e/spec-coverage/playwright.config.ts — what CI runs	6	69	0	245	491
playwright.config.ts — root, testDir: './tests/e2e'	75	0	58	245	433

So hermiq's measured e2e coverage of its own spec tree is 0.0% — not because the tests
are missing, but because 69 refs sit in files the CI config never runs.

All 69 are dead for one mechanism, and the checker names it:

referenced only by a file no Playwright project runs — tests/e2e/<x>.spec.ts is outside
testDir or excluded by testIgnore/testMatch

Worst offenders: skill-self-improvement.spec.ts (35 refs), skill-evals.spec.ts (10),
skill-maturity.spec.ts (5), flow-builder-dialect.spec.ts (5).

This is not a gate defect — the gate is right

check_e2e_coverage.py::_resolve_config() deliberately prefers the config under
playwright-test-path over the root one, because the shared workflow does exactly that
(.github#331). It is reporting reality: those tests exist, carry correct anchors, and
never execute.

Caveats for whoever wires this

• 75 live refs but only 58 covered scenarios — 17 refs resolve to no scenario at all.
  Dangling anchors are still not reported by gate-19, so they will not announce themselves.
• Expect newly-reached failures. A test that has never run in CI is not a test known to
  pass; the "did not run" count dropping is normally followed by new reds, and those are
  not regressions.

Numbers produced while closing the E2E red in #189.

[rubenvdlinde]

Correction to my comment above: there are two locks on this door, not one

I wrote that wiring this issue is "the only thing standing between hermiq and a non-zero
e2e coverage figure." That was half right, and my own numbers contained the contradiction —
6 refs were live while covered was 0, which cannot both be true unless the live ones
credit nothing.

They credit nothing. All six refs CI actually runs are dangling:

DANGLING  agent-object-leaf::a-new-finding-triggers-the-seeded-triage-flow
DANGLING  agent-object-leaf::an-unresolvable-owner-blocks-dispatch
DANGLING  agent-object-leaf::seeding-twice-creates-one-agent
DANGLING  agent-object-leaf::the-flow-contains-no-hermiq-authored-http-step
DANGLING  agent-object-leaf::the-user-is-told-the-object-contributed-no-context
DANGLING  agent-tool-governance::the-agent-may-run-exactly-one-flow

openspec/specs/agent-object-leaf/ does not exist. That spec lives only in
openspec/changes/hermiq-agent-leaf/specs/ and
openspec/changes/hydra-console-agent-leaves/specs/ — it was never promoted to the
canonical tree. The sixth names a scenario slug agent-tool-governance does not contain.

Resolved against the 735 scenario refs actually present in openspec/specs/:

cause of coverage_pct: 0.0	refs
file no Playwright project runs (this issue)	69
anchor resolves to no scenario	6 of the 6 that run
root-config arm	75 live → 59 resolve, 16 dangling

gate-19 reports neither condition. A dangling @e2e anchor is silently not-coverage
(.github#354 / #357), so these six read as traceability to any human and count for
nothing.

The 58-scenario figure in my comment above still stands — it was computed from refs that
resolve. What was wrong is the framing: wiring this issue alone will not take those six
tests off zero, because 16 of the 75 anchors would still credit nothing and nothing would
say so.

Worth doing in the same pass: promote agent-object-leaf into openspec/specs/, and grep
every live anchor against the actual scenario set. @spec/@e2e targeting a change dir
dies silently when that change is archived.