E2E: 59 tests fail across 18 specs — the setup wizard opens over the app since the ADR-111 demo-data step

[rubenvdlinde]

59 E2E tests fail across 18 spec files: the setup wizard is open and swallowing every click

Bracketed on development — last green e8650455, first red 72546d4d, which is the merge of #1295 (feat/demo-data-adr-111).

59 failed
273 passed (29.2m)

Every failure is the same shape — locator.click: Test timeout of 60000ms exceeded — across 18 unrelated spec files (bank-statement-wizard, bill-import-modal, bookings-resource-calendar, AccountantPortalDashboard, …). Specs that share nothing except that they click on something.

Root cause, already diagnosed in buildiq

buildiq hit this today and fixed it in ConductionNL/buildiq#523. Its commit message is the explanation:

EVERY STEP, not just the completion key. Setting setup_completed_version alone does NOT close the wizard … SetupController::status() recomputes each step from its own evidence on every call and only then writes the completion key, so the key is an OUTPUT of that computation, never an input to it.

Since nextcloud-vue 2.21 an OUTSTANDING OPTIONAL step is enough to open the wizard (nextcloud-vue#806 fixed it short-circuiting on completed), so store and demo-data being undone is what puts the dialog up — even though completed is true because the required step is done.

So two things landed together and only bite in combination:

1. feat(demo-data): ship demo data for every schema shillinq supplies (ADR-111) #1295 added a demo-data setup step — a new optional step, initially undone.
2. nextcloud-vue 2.21 stopped short-circuiting on completed, so an undone optional step now opens the wizard on its own.

The wizard renders over the app, and 59 clicks land on the overlay instead of the control.

The fix buildiq used

In tests/e2e/ci-seed.sh, write every step's own evidence key, not just the completion key:

for kv in "registry_url=…" "demo_data_decided=skipped" "setup_completed_version=1"; do
    php occ config:app:set <app> "${kv%%=*}" --value="${kv#*=}"
done

demo_data_decided is the app's own "dealt with" flag rather than "objects exist" — writing it is exactly what an operator who declined the import would leave behind, which is the honest state for a CI instance.

shillinq's equivalent keys will differ; the principle is that each step is recomputed from its evidence, so the seeder has to satisfy the evidence.

Not caused by the hydra-gates or dependency work

The two shillinq PRs merged earlier today (#1285 dependencies, #1309 hydra-gates) are hours before the boundary commit and touch neither the manifest nor the seeder. The bracket puts it squarely on #1295.

Worth checking whether any other app that gained an ADR-111 demo-data step has the same latent break — the trigger is "a new optional setup step" plus "nextcloud-vue ≥ 2.21", not anything specific to shillinq.

[github-actions]

Thank you for opening this issue! It has been assigned to @rubenvdlinde for review.

What happens next:

1. A maintainer will review this issue and determine the appropriate action
2. If accepted, it may be converted into an OpenSpec change for structured development
3. You will be notified of any status changes

Triage labels:

• triage:accept-feature — accepted as a new feature
• triage:accept-bugfix — accepted as a bug fix
• triage:needs-info — more information needed
• triage:wontfix — will not be addressed
• triage:duplicate — duplicate of another issue

[rubenvdlinde]

Confirmed — and measured, not inferred

Run 33212749905 on c10cf6bd, the merge carrying both #1324 (the wizard fix) and #1328 (the SERVER_DIR correction):

	before	after
locator.click: Test timeout	59	0
specs executed	—	325
totals	59 failed / 273 passed	1 failed / 324 passed

The seeder line demo-data step marked decided (skipped) appears in the log, so the occ call fired and demo_data_decided closed the step.

So the diagnosis held: an undone optional setup step was opening the wizard over the SPA, and every click landed on the overlay. Satisfying that step's own evidence key closes it.

Why "specs executed: 325" is quoted here. An earlier run reported click-timeouts = 0 as well — because no spec ran at all, after my #1324 aborted the seeder on an unbound SERVER_DIR. A failure count that falls to zero because the work stopped looks identical to one that falls to zero because it was fixed. The positive count is what separates them, and it is the only reason this can be called confirmed.

One failure remains, and it is not this

✘ spec-coverage/demo-data-setup-step.spec.ts:95
  ADR-111 demo data › installing the demo data reports HOW MUCH landed, not just success
  Error: page.evaluate: Test timeout of 60000ms exceeded

A different shape entirely — a 60s timeout inside page.evaluate on a spec that installs the demo dataset, added in 38931fc a few hours ago. Seeding 499 schemas inside a 60s budget is a plausible cause, but that is a guess and it belongs to whoever wrote the spec.

Worth noting the seeder now marks the step skipped rather than installing, which is deliberate (it avoids seeding data other specs do not expect) — but it means this spec exercises the install path itself and is the one place that budget has to hold.

Also: shillinq's skip-discipline verdict is now V1 3 / V2 0 / V3 1 on 22/347 skipped. The single V3 is new, from the same recently-added spec file.