Skip to content

Frontend E2E: give each Playwright run its own dev-server port - #580

Merged
WilfordGrimley merged 1 commit into
masterfrom
fix/playwright-ephemeral-port
Jul 29, 2026
Merged

Frontend E2E: give each Playwright run its own dev-server port#580
WilfordGrimley merged 1 commit into
masterfrom
fix/playwright-ephemeral-port

Conversation

@WilfordGrimley

@WilfordGrimley WilfordGrimley commented Jul 29, 2026

Copy link
Copy Markdown

frontend/playwright.config.ts hardcoded http://localhost:3000 in both
use.baseURL and webServer.url. Two concurrent frontend E2E runs on one
box therefore collided — the same bug class #571 just fixed for the Python
test harness's Docker containers, with a nastier failure mode.

Off CI (reuseExistingServer: !process.env.CI) Playwright does not
start a second dev server when something already answers on that URL — it
reuses it. So the second run silently pointed at the first run's
next dev, i.e. tested the first worktree's checkout, and then lost the
server outright the moment the first run finished and tore it down.

Reproduced on unmodified master first

Two worktrees at origin/master, same four spec files, second started 3s
after the first:

run specs result
C (first, 1 spec) New 1 passed
D (second, 4 specs) New HomepagePanel Stats WhatsThatPWA 8 of 10 failed

Every one of D's failures was a page.goto error out of
tests/test-utils.ts:34 or an assertion about UI D's branch never
touched. Nothing in the output mentions a port. This is precisely the
"looks like a broken change, costs an afternoon" shape.

The fix

resolvePort() asks the kernel for a free port (bind port 0, read the
assignment back, release) unless PLAYWRIGHT_PORT is set, then exports
the result into the environment. That export matters: Playwright's worker
processes each re-load the config in their own process, so without it
every worker would draw a different port. baseURL, webServer.url and
the webServer command all derive from that single value, and
playwright.perf.config.ts inherits it for free by spreading the base
config.

PLAYWRIGHT_PORT=3000 restores the previous behaviour exactly, including
reusing a npm run dev you already have running — worth keeping for a
fast edit-run loop, since it skips the dev-server boot every invocation.

next dev is now given an explicit --port, which is load-bearing
rather than cosmetic: Next only walks to the next free port when it chose
the port itself. Given one explicitly it exits with EADDRINUSE (verified
directly, not assumed).

The favicon MSW handler had to move with it. It is in defaultHandlers,
so every E2E test loads it, and @msw/playwright runs it in the
Playwright Node process rather than in the page — it was fetching
http://localhost:3000/favicon.ico out-of-band, which a per-run port
breaks. It now reads the port back out of the environment, keeping the
literal 3000 only as the fallback for a caller that never went through the
Playwright config (jest), which is exactly the previous behaviour.

What this does NOT close, stated plainly

The kernel's assignment is released before next dev binds it, and that
gap spans npm run dev plus Next's boot — seconds, not microseconds.
Playwright's webServer.url has no port-0 read-back equivalent, so unlike
#571 there is nothing holding the binding across the gap. A foreign
process taking the port inside that window remains possible.

What that window can no longer produce is a wrong-but-passing run.
Because of the explicit --port, losing the race aborts at webServer
startup with Error: Process from config.webServer exited early — loud,
attributable, and fixed by re-running. The silent cross-talk mode, which
was the expensive one, is gone.

Two runs in the same directory still collide regardless of port (both
next dev servers write frontend/.next; Playwright writes
playwright/.auth/, test-results/ and the report there too).
Concurrency is safe across worktrees, which is how this box actually runs.

Both residuals are written up in docs/troubleshooting.md next to #571's
entry, so the next person loses minutes rather than an afternoon.

Verification

Two concurrent runs, the standard #571 met — a single passing run proves
nothing about a concurrency bug. Against this branch, both runs given the
same six spec files (including the favicon-dependent
DynamicLogo.visual.spec.ts) and started simultaneously:

  • 20/20 and 20/20 passed, four workers each. The 4-workers-each result
    is also what proves the env export reaches worker processes.
  • ss -ltnp sampled every 8s for the whole overlap showed two
    next-server listeners on distinct ports (*:42133, *:45759)
    throughout — never one, never the same.
  • Re-running the master repro scenario against this branch: the
    D-equivalent run passed 10/10 (was 8-of-10 failing).

Plus:

  • Full local E2E suite: 335 passed, 6 skipped (npx playwright test).
  • PLAYWRIGHT_PORT=39123 pinned run: 1 passed.
  • python3 .github/scripts/docs_lint.py → clean.
  • npx jest src/features/stats/StatsPage.test.tsx (a mocks/handlers
    consumer) → 3 passed.
  • pre-commit: prettier, eslint, EOF/whitespace, readme parity all pass.

CI is unaffected: each test-frontend.yml shard is its own container, so
shards never shared a port anyway, and nothing in the workflow or the
.github/actions/test-frontend composite assumes 3000.

🤖 Generated with Claude Code

https://claude.ai/code/session_013NhYmT1PxCcyemA16dFDxN

`frontend/playwright.config.ts` hardcoded `http://localhost:3000` in both
`use.baseURL` and `webServer.url`, so two concurrent frontend E2E runs on
one box collided - the same bug class #571 just fixed for the Python test
harness's Docker containers, with a worse failure mode.

Off CI (`reuseExistingServer: !process.env.CI`) Playwright does not start
a second dev server when something already answers on that URL, it
REUSES it. So the second run silently tested the FIRST worktree's
checkout, then lost the server outright when that run tore it down.
Reproduced on unmodified master: two overlapping runs of the same four
spec files, the second started 3s after the first - first passed 1/1,
second failed 8 of 10, every failure pointing at application code rather
than at the port.

`resolvePort()` now asks the kernel for a free port (bind port 0, read
the assignment back, release) unless `PLAYWRIGHT_PORT` is set, and
exports it into the environment so Playwright's worker processes - which
each re-load the config in their own process - inherit the same port
rather than drawing their own. `baseURL`, `webServer.url` and the
`webServer` command all derive from that one value;
`playwright.perf.config.ts` inherits it for free by spreading the base
config. `PLAYWRIGHT_PORT=3000` restores the old behaviour, including
reuse of an already-running `npm run dev`.

`next dev` is now given an explicit `--port`. That matters: Next only
walks to the next free port when it chose the port itself; given one
explicitly it exits with EADDRINUSE. So the one thing this fix does NOT
close - the window between the kernel releasing the probe and Next
binding, which spans `npm run dev` plus Next's boot - can only produce a
loud abort at webServer startup, never a wrong-but-passing run. Unlike
#571 there is no port-0 read-back on `webServer.url` to hold the binding
across that gap; that residual, and the fact that two runs in the SAME
directory still collide over `frontend/.next`, are written up in
docs/troubleshooting.md rather than left to be rediscovered.

The `favicon` MSW handler had to move with it. It is in
`defaultHandlers`, so every E2E test loads it, and `@msw/playwright` runs
it in the Playwright NODE process rather than in the page - it was
fetching `http://localhost:3000/favicon.ico` out-of-band, which a
per-run port breaks. It now reads the port back out of the environment,
keeping the literal 3000 only as the fallback for a caller that never
went through the Playwright config (jest), which is exactly the previous
behaviour.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013NhYmT1PxCcyemA16dFDxN
@WilfordGrimley
WilfordGrimley force-pushed the fix/playwright-ephemeral-port branch from f757a24 to cc12cc0 Compare July 29, 2026 16:06
@WilfordGrimley
WilfordGrimley merged commit 27d6cca into master Jul 29, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant