360 scenarios blanket-excluded from gate-19 on reasons that are false — CSS variables and cascade order ARE browser-observable

[rubenvdlinde]

Summary

15 of nldesign's 42 specs carry a whole-spec @e2e exclude, suppressing 360 of 786
scenarios (46%). A further 225 are excluded per-scenario, so 585 of 786 (74%) are excluded and
only 59 were covered. Many of the stated reasons are false, and several are refuted by
nldesign's own test suite.

The reasons do not hold

An exclusion claims a browser cannot observe this. These claim otherwise:

spec	scenarios	stated reason	why it does not hold
vng-token-set	13	"CSS custom property values, colour palette, typography tokens … not testable"	getComputedStyle(el).getPropertyValue('--…') reads exactly these. The app is a theming app; CSS variable values are its entire observable output.
component-tokens	11	"CSS variable naming — not testable via DOM"	same
nextcloud-variable-mapping	11	"CSS variable mapping … file content"	same
css-architecture	55	"CSS cascade order — not DOM-testable"; "@font-face declarations — not DOM-testable"	Cascade order is observable from the ordered link[rel=stylesheet] list, and document.fonts enumerates loaded faces. dark-mode.spec.ts in this very repo already asserts stylesheet ORDER this way (expect(darkIndex).toBeGreaterThan(lightIndex)).
hide-slogan	31	"CSS selector behaviour on the login page … not DOM-testable"	The login page is a page; a browser can load it and assert the slogan is not rendered.
menu-labels	36	"CSS typography/layout rules"	observable via computed style
token-set-dropdown	6	"covered by admin-settings tests"	not an observability claim at all — and admin-settings itself had 14 uncovered scenarios

Several reasons are test-fixture objections wearing observability clothing — e.g.
vng-token-set: "requires selecting VNG token set — mutates IConfig". That is a setup cost, not
an impossibility; tests/e2e/workflows/*.workflow.spec.ts already mutate and restore config.

Per the fleet rule: reasons naming a test artifact hold; reasons naming a state of the
world rot. These name a state of the world, and it is not the case.

The excluding files are exclusion manifests

tests/e2e/spec-coverage/css-architecture.spec.ts contains 49 @e2e references and exactly one
test(). hide-slogan.spec.ts: 34 references, 1 test. component-tokens.spec.ts: 15, 1.
vng-token-set.spec.ts: 15, 1. They are documentation of what is not tested, shaped like a test
file.

To be explicit about what is not wrong: these excludes are not inflating the covered count.
A plain @e2e exclude <ref> credits nothing (verified against the checker's regexes), so
.github#345 does not fire here. The problem is only that the stated justifications are untrue.

Why this matters more for nldesign than elsewhere

nldesign is the government-theming app. Its contract is CSS variables carry the right values,
components use standard Nextcloud tokens, contrast meets WCAG AA. Those are precisely the claims
excluded as "not testable via DOM", and precisely the claims a browser is best placed to check. A
regression in vng-token-set's palette would currently be caught by nothing.

Proposed work

Reason-by-reason review of all 15 whole-spec excludes, replacing false ones with real tests:

1. Token-set value assertions (vng-token-set, component-tokens,
   nextcloud-variable-mapping) — apply the set, read the custom properties back off
   document.documentElement, assert the documented values. Also gives real WCAG AA contrast
   coverage.
2. Cascade order (css-architecture) — extend the existing dark-mode.spec.ts ordering
   pattern to the full documented layer sequence.
3. hide-slogan / menu-labels — toggle, load the login page / app menu, assert the rendered
   result; restore.

Genuinely non-observable scenarios remain excludable — token-sync-workflow (GitHub Actions),
prometheus-metrics (scrape format), the occ-only generator scenarios in dark-mode, and
no-enforcetheme-usage-in-the-codebase (a source-grep assertion) are all fair. The point is that
the reason must be true one scenario at a time.

Measured while taking gate-19 from 142 → 126 with 16 new browser tests
(admin-only enforcement across 6 specs; the theme-preview lifecycle). Those 16 deliberately
avoided the excluded specs, so this backlog is untouched by that work.

[rubenvdlinde]

The ceiling, measured — 126 is a floor, 509 is the honest number

Measured on 101869f (= origin/development) with the canonical 1735-line check_e2e_coverage.py from ConductionNL/.github@main, by disabling each inherited exclusion layer in the gate's own parser and re-running. The instrumented harness reproduces the shipped baseline exactly (786 / 75 / 585 / 126), so it is not perturbing the metric.

layers active	scenarios	covered	excluded	uncovered
as shipped (the reported number)	786	75	585	126 ← floor
drop requirement-level inheritance	786	77	545	164
drop whole-spec blanket markers	786	85	230	471
drop both (ceiling)	786	87	190	509

126 is a 4.0× understatement. The whole-spec layer alone hides 345 scenarios; requirement-level inheritance hides 38.

Layer split — report the split, never one percentage

Of the 585 exclusions:

360  61.5%  whole-spec blanket marker   (15 specs)
185  31.6%  scenario-level, own reason
 40   6.8%  requirement-level, inherited by every sibling (the .github#356 shape)

⚠️ This is not openbuild's profile (whole-spec 42.6%, requirement-level 46.8%). The dominant layer differs per app, so importing another app's diagnosis gives the wrong remediation order. Here the whole-spec markers are the entire problem.

Distinct reasons, with the #356 sentinel decoded

exclude_reason stores the literal string <inherited from requirement> for every requirement-level exclusion, so counting distinct values of that field collapses all 40 into one bucket. Recovering the real text — the requirement-level marker is picked up from any line in the requirement body, not just the ### heading — gives:

layer	scenarios	distinct reasons (key = (spec, reason))	share a reason
whole-spec	360	15	360 (100%)
requirement	40	15 (not 1)	37 (92.5%)
scenario-level	185	167	31 (16.8%)
total	585	197

The scenario-level layer is healthy — 167 distinct reasons for 185 scenarios. All of the blanket-ness is in the other two layers.

The sharpest evidence that the markers are false

13 scenarios are excluded as "not testable via browser UI" while a live, running @e2e test already names them. Computed as the intersection of the excluded set with collect_ref_status()'s covered refs:

whole-spec (10):
  component-tokens::bridge-maps-utrecht-tokens-to-nldesign
  css-architecture::standard-css-load-order-for-nldesign-design-system
  custom-css-overrides::file-is-written-by-the-save-endpoint
  extended-token-sets::settings-page-shows-all-token-sets
  hide-slogan::checkbox-reflects-current-state-on-load
  menu-labels::checkbox-reflects-current-state-on-load
  token-set-dropdown::dropdown-renders-with-all-token-sets
  token-set-dropdown::preview-reflects-new-selection
  token-sets::token-set-persisted-via-api
  vng-token-set::vng-appears-in-admin-dropdown
requirement (2):  token-set-apply-dialog::dialog-closed-with-cancel, ::admin-unchecks-a-row
scenario (1):     token-editor-ui::color-tokens-render-a-color-picker (+ ::admin-changes-a-color-token)

A marker asserting a scenario cannot be browser-tested, over a scenario that is browser-tested and passing, is refuted by the repository itself. css-architecture is the clearest case: its whole-spec marker covers 55 scenarios on the grounds that they "describe CSS cascade layers, file load order, and server-side PHP logic with no testable UI surface" — and one of those 55 has a green test.

(One ref appears twice in that list: token-set-apply-dialog::admin-unchecks-a-row is two different scenario titles collapsing to one slug — .github#354.)

The 15 whole-spec markers, by blast radius

 55  css-architecture         "CSS cascade layers, file load order ... no testable UI surface"
 53  token-sets               "Backend/filesystem/API spec"
 51  theming-sync             "Backend/API ... PHP service logic"
 43  prometheus-metrics       "API/backend metrics spec"
 36  menu-labels              "Backend/CSS spec ... PHP boot-time CSS injection"
 31  hide-slogan              "Backend/CSS/login-page spec"
 13  vng-token-set            "CSS custom property values, colour palette"
 12  docs-content             "Documentation/Docusaurus site spec"
 12  token-sync-workflow      "GitHub Actions / CI workflow spec"
 11  component-tokens         "pure CSS cascade / file-structure assertions"
 11  custom-css-overrides     "file write semantics, CSS cascade order"
 11  extended-token-sets      "auto-generation scripts, nightly sync"
 11  nextcloud-variable-mapping "CSS file content, variable mapping tables"
  6  token-set-dropdown       "covered by admin-settings tests"
  4  nl-design                "CSS variable usage rules"

Two of these reasons are not observability claims at all and should be re-examined first:

• token-set-dropdown — "covered by admin-settings tests." That is a delegation claim, not an impossibility, and it needs checking: admin-settings has uncovered scenarios of its own.
• token-sync-workflow / docs-content are genuinely out of browser reach (GitHub Actions scheduling, Docusaurus build output) and are the right shape — a reason naming a test artifact holds.

The CSS-family reasons are the false ones, and they are 61.5% of the pile: getComputedStyle().getPropertyValue() reads custom property values, the ordered link[rel=stylesheet] list is in the DOM, and document.fonts enumerates @font-face. Round 1 proved this constructively — 16 real tests, 142 → 126, zero exclusions added.

Method notes for anyone repeating this

• Use the canonical checker. A stale .github ships a 622-line version; missing @e2e in your log means you ran the stale one (canonical says without a running e2e test).
• Recover the requirement-level reason text before quoting a distinct-reason count, and say which key you used — reason vs exclude_reason vs (spec, reason) give three different answers, and an empty lookup reports a dramatic finding rather than reporting itself.
• Assert your instrumented harness reproduces the shipped number before trusting any variant of it.