Repository navigation
Test harness: browser resolution in TestMain, WAND_BROWSER_* selection, no flaky skips #53
Description
Parent
Spec: #33 (Baseline release). Section numbers refer to its Implementation Decisions.
What to build
The suite launches browsers exactly the way users do and CI chooses the browser with environment variables; a red run always means a real bug because no flaky skip, fixed port or public-network dependency remains. Spec section 12; ticket #15, #20.
Acceptance criteria
- The root suite's
init()preload is gone;TestMainruns Browser resolution once and fails with the resolved path in the message; every run prints the browser path and version actually used. - Tests launch only through
launcher.New()(plus the User mode test); no test-only browser knob exists; CI selects withWAND_BROWSER_*. - The pooled tester count follows
GOMAXPROCS; the ci-test wrapper remains the entry point settingGODEBUG=tracebackancestors=100forleakcheck. - The two flaky-skipped tests (page navigate error on Windows, hijack mock whole response without body) are root-caused and un-skipped on every Tier 1 OS, or deleted with the covering test named; the dead tests (native drag, old browser, lint ignore, CDP mass browser close) are deleted; the environment guards stay (fonts container-only; binary size non-Windows non-container with the threshold lowered to lock in the leakless removal; profile dir flag-gated).
- No test uses a fixed port; no test reaches the public internet except the managed-browser download (asserted from the CDP log or a network-off run).
- A deliberately failing local run leaves no browser or temporary user-data directory behind; the Gate is green.
Blocked by
- Suite green on Chrome 152, Windows and Go 1.23 #43 (Suite green on Chrome 152, Windows and Go 1.23)
- Browser resolution: system browser first, WAND_BROWSER_* and platform errors #45 (Browser resolution: system browser first, WAND_BROWSER_* and platform errors)
- Orphan guard replaces leakless #46 (Orphan guard replaces leakless)
Activity
Pre-fetch the browser, keep ExecLine race-free, allow the Go 1.27 bin…
Fly-Playgroud commentedon Sep 6, 2026
Two flakes observed in the Gate on 2026-09-06, both under
-raceonubuntu-latest, both green on a rerun of the same SHA, neither named in the acceptance criteria yet. Recording them here so the harness rewrite root-causes them rather than leaving reruns as the fix:lib/cdpTestCrash:panic: Fail in goroutine after TestCrash has completed(pull_request run of 8ae22a6, https://github.com/headlesslab/wand/actions/runs/34011758779, attempt 1; the push run of the same SHA passed). A goroutine of the test still calls the testing API after the test returned, so the same test that Suite green on Chrome 152, Windows and Go 1.23 #43 reworks for Windows also needs its goroutine joined before it ends.TestScrollScreenshotErrors:setup_test.go:268: leaking MockClient.stub(push run of 4232d3c onmain, https://github.com/headlesslab/wand/actions/runs/34012804428, attempt 1; the PR branch runs of the same tree passed). One of theg.Panicblocks panicked before itsMockClientstub was consumed, so the harness's leak check tripped.Both runs carried only leaf-package and README changes (#74, #75), so neither is a regression signal.
Fly-Playgroud commentedon Sep 6, 2026
Input from #43 (PR #76): the root flake
TestScrollScreenshotErrors: leaking MockClient.stub.Three Gate hits so far, every one on a push run and green on
gh run rerun --failed: main after PR #75 (4232d3c), then both commits of PR #76 (2ddd70a, 6c6c7e7). The pull_request run of the same SHA passed each time. Nothing in those diffs touches the test orScrollScreenshot.Mechanism, read from the code rather than seen in a log:
TestScrollScreenshotErrors(page_test.go) wraps sixMustScrollScreenshotcalls ing.Panic, each after stubbing the first call of one CDP method (stubErr/stub,setup_test.go). A stub is consumed only when the stubbed method is reached. IfMustScrollScreenshotpanics for another reason first,g.Panicis satisfied all the same (it accepts any panic and prints no value), the stub stays armed, and the nextstubor the tester cleanup'ssetCall(nil)reportsleaking MockClient.stub. That is why the job log carries this one line and nothing about the real failure.Two candidates for the real failure, neither confirmed: the unstubbed
Page.captureScreenshot(FromSurface: falseinpage.go) failing under-racewith four pooled browsers on the 4-vCPU runner, orPage.getLayoutMetricsreading the 4500 pxscroll-y.htmlfixture before layout, sincePage.Navigatereturns right afterPage.navigateresponds and does not wait for load.Suggested shape of the fix, for this ticket's "a red run always means a real bug": make each
g.Panicblock assert that the recovered value is the stubbed error, so an unrelated panic fails with its own message; and wait for load before the first stub.TestPageWaitIdlehanging to theTimeoutEachpanic under local load (28 pooled browsers on Windows, passes alone) is the other pre-existing hang seen while verifying #43.Fly-Playgroud commentedon Sep 6, 2026
Observed while implementing #45 (PR #79), for the system-browser suites this ticket and #61 will run: on Windows 11 with the branded Google Chrome 152 as the System browser,
lib/launcher'sTestManagedfailed two runs out of three at the wait for the crashed browser's user-data directory to be removed (15 s budget; the branded Chrome keeps the directory busy afterBrowser.crash), while the same test passed three out of three on Chrome for Testing 152.0.7977.82. The Gate now pinsWAND_BROWSER_BINto the Managed browser, so it does not see this; a system-browser run on Windows will.Fly-Playgroud commentedon Sep 6, 2026
The Gate's load flakes, collected from the pull requests that hit them, so that this ticket's "a red run always means a real bug" covers them. Each has failed on one sibling run of a SHA while the other passed, and each passes alone; the rerun of the failed job was green every time.
TestPageWaitIdle(root):[wand_test.TimeoutEach] TestPageWaitIdle timeout after 1m0s, the whole binary panics and the coverprofile is lostMustWaitIdleisWaitIdle(time.Minute), whose JS helper isrequestIdleCallback(resolve, {timeout})with that minute as the fallback; the harnessTimeoutEachis also one minute, so when a loaded renderer never goes idle the JS fallback and the test timeout race at 60 s and the test panics first. A shorterWaitIdletimeout thanTimeoutEach, or waiting for what the click actually produces, would fix it.TestPageErr(root): the harness cleanup atsetup_test.go:199getsNo target with given id foundfromTargetCloseTarget, followed by thesync: negative WaitGroup counterpanics ofTestSetUserAgent's server that follow any failureTargetGetTargetsand the close; the cleanup should tolerate a target that closed itself.TestScrollScreenshotErrors(root):setup_test.go:268: leaking MockClient.stubMustScrollScreenshotcalls ing.Panic, each after stubbing one CDP method; a stub leaks when the call panics for another reason before the stubbed method is reached, andg.Panicaccepts any panic, so the real cause never prints. Candidates: the unstubbedPage.captureScreenshot(FromSurface: false) failing under-raceload, orPage.getLayoutMetricsseeing the 4500 px fixture before layout, sinceNavigatereturns right afterPage.navigate. A fix asserts the panic value against the stubbed error and waits for load before stubbing.TestCrash(lib/cdp):panic: Fail in goroutine after TestCrash has completed, under-raceTestManaged(lib/launcher): the wait for the crashed browser's user-data directory to be removed exceeds its 15 s budget, then the second handshake gets anet.OpErrorinstead of the expectedBadHandshakeErrorBrowser.crash. The Gate pinsWAND_BROWSER_BINto the Managed browser since #79, so only a system-browser run sees it.None of these has an issue of its own; they are listed here because this ticket owns the harness and the no-flake rule (spec #33, section 12). If the harness work here does not reach them, they need a ticket of their own before #64 (three consecutive green Nightlies).
Harness cleanup asserts that no target is left; TestSetUserAgent on i…