Skip to content

Parallelize E2E (slot-scope all daemon ports, 4 workers); stabilize visual smoke - #37

Merged
Aztec03hub merged 1 commit into
mainfrom
perf-e2e-parallel-v2
Jun 25, 2026
Merged

Parallelize E2E (slot-scope all daemon ports, 4 workers); stabilize visual smoke#37
Aztec03hub merged 1 commit into
mainfrom
perf-e2e-parallel-v2

Conversation

@Aztec03hub

Copy link
Copy Markdown
Owner

Summary

Makes the E2E scenarios suite run in parallel (was serial) and resolves the perpetually-red E2E visual smoke job.

Goal 1 — Parallelize E2E

Obsolete-pin removal

The daemon fixture used to PIN the broker MQTT ports to 1883/9001 because the web UI was believed to hardcode ws://<host>:9001/mqtt. The single-origin refactor (#23-26) made that pin obsolete: the browser now connects to a same-origin /mqtt WebSocket bridged onto the daemon's own web port (broker_ws_same_origin: true via /api/capabilities; resolved by mqtt-store.svelte.js resolveBrokerUrl/sameOriginBrokerUrl). The browser never touches the broker WS port.

What changed

  • e2e/fixtures/daemon.ts: ALL four daemon ports are now slot-scoped (portsForSlot): mcp=9930+N*10, web=9931+N*10, broker-tcp=9932+N*10, broker-ws=9933+N*10. The broker still binds its TCP/WS listeners (config broker.port/broker.ws_port, confirmed in config.py/broker.py) for the daemon's own + non-web clients, now on per-slot ports. Each daemon also gets its own $HOME (mkdtemp) + data dir, so isolation is total.
  • playwright.config.js: workers: CI ? 4, retries: CI ? 1 (the real flakes, LWT + rename races, were fixed in CI green: fix 3.10 version test, WS-bridge LWT, and settings rename regression #36, so 2 retries was overkill).
  • e2e/fixtures/browser.ts: fixed the benign-console-noise filter. The prior WIP regex (CLOSING|CLOSED) state did NOT match Chromium's actual verbatim message WebSocket is already in CLOSING or CLOSED state., so the teardown-race noise leaked and failed scenarios under worker contention. Corrected to match the whole phrase. All scenarios collect console errors through the single appPage fixture, so this one fix covers every spec.

Before / after (local, full suite, 246 tests)

Config Wall clock Result
Serial (--workers=1) 234s (~3.9 min) 245 passed
Parallel (--workers=4) ~115s (~1.9 min) 245 passed

~2x faster.

Isolation evidence (multiple runs)

  • With the broken regex (pre-fix), --workers=4 x3: 1, 2, 2 failures, ALL purely the benign WebSocket is already in CLOSING or CLOSED state. teardown message (243-244/246 passing).
  • With the corrected regex, --workers=4 x3: rc=0, 245 passed, 0 failed every run (117s / 115s / 115s).
  • Across every parallel run: zero port-bind errors, zero already in use, zero Daemon failed to start, zero state_unsafe_mutation, no state bleed. The only contention-surfaced issue was the cosmetic console message, now filtered. Per-worker isolation is proven.

Goal 2 — Visual smoke resolution

Decision: on-demand, with a CI-matching regeneration path (no docker available locally to render like the runner, which the preferred fix needs).

  • The test-e2e-visual job no longer runs on push/PR (that auto run was perpetually red, never blocking but red, because baselines were generated off-CI and flake ~1-4% vs GitHub's font render). It is now workflow_dispatch-gated.
  • Added a workflow_dispatch input update_baselines: when true, the job runs --update-snapshots ON the ubuntu-latest runner and commits the regenerated *-linux.png baselines back to the branch, so baselines can be made to match CI exactly (gh workflow run ci.yml -f update_baselines=true). When false, it compares + uploads diffs for review.
  • Bumped tolerance to a size-independent maxDiffPixelRatio: 0.02 (config + screenshot.ts). Note: Playwright takes Math.min of maxDiffPixels and maxDiffPixelRatio, so the old per-call maxDiffPixels: 500 would have neutered any ratio; the helper now defaults to the ratio and only forwards an absolute count when a caller explicitly tightens a snapshot.
  • Documented the regen-and-commit flow in web/e2e/README.md (also refreshed the stale pinned-port content).

Net: the pipeline shows no perpetual red for visuals, on-demand visual checks remain, and there's a one-click path to CI-matching baselines.

Gates

  • pytest -q: 1469 passed
  • pnpm test (vitest): 1195 passed (89 files)
  • pnpm build: compiles
  • E2E --workers=4: 3/3 runs green, 245 passed each
  • ruff / basedpyright: N/A (no Python touched)

🤖 Generated with Claude Code

@Aztec03hub
Aztec03hub merged commit c6a7439 into main Jun 25, 2026
8 checks passed
@Aztec03hub
Aztec03hub deleted the perf-e2e-parallel-v2 branch June 25, 2026 20:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant