Frontend E2E: give each Playwright run its own dev-server port - #580
Merged
Conversation
`frontend/playwright.config.ts` hardcoded `http://localhost:3000` in both `use.baseURL` and `webServer.url`, so two concurrent frontend E2E runs on one box collided - the same bug class #571 just fixed for the Python test harness's Docker containers, with a worse failure mode. Off CI (`reuseExistingServer: !process.env.CI`) Playwright does not start a second dev server when something already answers on that URL, it REUSES it. So the second run silently tested the FIRST worktree's checkout, then lost the server outright when that run tore it down. Reproduced on unmodified master: two overlapping runs of the same four spec files, the second started 3s after the first - first passed 1/1, second failed 8 of 10, every failure pointing at application code rather than at the port. `resolvePort()` now asks the kernel for a free port (bind port 0, read the assignment back, release) unless `PLAYWRIGHT_PORT` is set, and exports it into the environment so Playwright's worker processes - which each re-load the config in their own process - inherit the same port rather than drawing their own. `baseURL`, `webServer.url` and the `webServer` command all derive from that one value; `playwright.perf.config.ts` inherits it for free by spreading the base config. `PLAYWRIGHT_PORT=3000` restores the old behaviour, including reuse of an already-running `npm run dev`. `next dev` is now given an explicit `--port`. That matters: Next only walks to the next free port when it chose the port itself; given one explicitly it exits with EADDRINUSE. So the one thing this fix does NOT close - the window between the kernel releasing the probe and Next binding, which spans `npm run dev` plus Next's boot - can only produce a loud abort at webServer startup, never a wrong-but-passing run. Unlike #571 there is no port-0 read-back on `webServer.url` to hold the binding across that gap; that residual, and the fact that two runs in the SAME directory still collide over `frontend/.next`, are written up in docs/troubleshooting.md rather than left to be rediscovered. The `favicon` MSW handler had to move with it. It is in `defaultHandlers`, so every E2E test loads it, and `@msw/playwright` runs it in the Playwright NODE process rather than in the page - it was fetching `http://localhost:3000/favicon.ico` out-of-band, which a per-run port breaks. It now reads the port back out of the environment, keeping the literal 3000 only as the fallback for a caller that never went through the Playwright config (jest), which is exactly the previous behaviour. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013NhYmT1PxCcyemA16dFDxN
WilfordGrimley
force-pushed
the
fix/playwright-ephemeral-port
branch
from
July 29, 2026 16:06
f757a24 to
cc12cc0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
frontend/playwright.config.tshardcodedhttp://localhost:3000in bothuse.baseURLandwebServer.url. Two concurrent frontend E2E runs on onebox therefore collided — the same bug class #571 just fixed for the Python
test harness's Docker containers, with a nastier failure mode.
Off CI (
reuseExistingServer: !process.env.CI) Playwright does notstart a second dev server when something already answers on that URL — it
reuses it. So the second run silently pointed at the first run's
next dev, i.e. tested the first worktree's checkout, and then lost theserver outright the moment the first run finished and tore it down.
Reproduced on unmodified master first
Two worktrees at
origin/master, same four spec files, second started 3safter the first:
NewNewHomepagePanelStatsWhatsThatPWAEvery one of D's failures was a
page.gotoerror out oftests/test-utils.ts:34or an assertion about UI D's branch nevertouched. Nothing in the output mentions a port. This is precisely the
"looks like a broken change, costs an afternoon" shape.
The fix
resolvePort()asks the kernel for a free port (bind port 0, read theassignment back, release) unless
PLAYWRIGHT_PORTis set, then exportsthe result into the environment. That export matters: Playwright's worker
processes each re-load the config in their own process, so without it
every worker would draw a different port.
baseURL,webServer.urlandthe
webServercommand all derive from that single value, andplaywright.perf.config.tsinherits it for free by spreading the baseconfig.
PLAYWRIGHT_PORT=3000restores the previous behaviour exactly, includingreusing a
npm run devyou already have running — worth keeping for afast edit-run loop, since it skips the dev-server boot every invocation.
next devis now given an explicit--port, which is load-bearingrather than cosmetic: Next only walks to the next free port when it chose
the port itself. Given one explicitly it exits with
EADDRINUSE(verifieddirectly, not assumed).
The
faviconMSW handler had to move with it. It is indefaultHandlers,so every E2E test loads it, and
@msw/playwrightruns it in thePlaywright Node process rather than in the page — it was fetching
http://localhost:3000/favicon.icoout-of-band, which a per-run portbreaks. It now reads the port back out of the environment, keeping the
literal 3000 only as the fallback for a caller that never went through the
Playwright config (jest), which is exactly the previous behaviour.
What this does NOT close, stated plainly
The kernel's assignment is released before
next devbinds it, and thatgap spans
npm run devplus Next's boot — seconds, not microseconds.Playwright's
webServer.urlhas no port-0 read-back equivalent, so unlike#571 there is nothing holding the binding across the gap. A foreign
process taking the port inside that window remains possible.
What that window can no longer produce is a wrong-but-passing run.
Because of the explicit
--port, losing the race aborts at webServerstartup with
Error: Process from config.webServer exited early— loud,attributable, and fixed by re-running. The silent cross-talk mode, which
was the expensive one, is gone.
Two runs in the same directory still collide regardless of port (both
next devservers writefrontend/.next; Playwright writesplaywright/.auth/,test-results/and the report there too).Concurrency is safe across worktrees, which is how this box actually runs.
Both residuals are written up in
docs/troubleshooting.mdnext to #571'sentry, so the next person loses minutes rather than an afternoon.
Verification
Two concurrent runs, the standard #571 met — a single passing run proves
nothing about a concurrency bug. Against this branch, both runs given the
same six spec files (including the favicon-dependent
DynamicLogo.visual.spec.ts) and started simultaneously:is also what proves the env export reaches worker processes.
ss -ltnpsampled every 8s for the whole overlap showed twonext-serverlisteners on distinct ports (*:42133,*:45759)throughout — never one, never the same.
D-equivalent run passed 10/10 (was 8-of-10 failing).
Plus:
npx playwright test).PLAYWRIGHT_PORT=39123pinned run: 1 passed.python3 .github/scripts/docs_lint.py→ clean.npx jest src/features/stats/StatsPage.test.tsx(amocks/handlersconsumer) → 3 passed.
CI is unaffected: each
test-frontend.ymlshard is its own container, soshards never shared a port anyway, and nothing in the workflow or the
.github/actions/test-frontendcomposite assumes 3000.🤖 Generated with Claude Code
https://claude.ai/code/session_013NhYmT1PxCcyemA16dFDxN