Skip to content

ci: dispatchable Windows race soak to hunt the one-off runtime crash - #54

Closed
deadpoets wants to merge 1 commit into
mainfrom
diag/windows-race-soak
Closed

ci: dispatchable Windows race soak to hunt the one-off runtime crash#54
deadpoets wants to merge 1 commit into
mainfrom
diag/windows-race-soak

Conversation

@deadpoets

Copy link
Copy Markdown
Owner

Diagnostic tooling for the open item from #53 — not a fix, and not a gate.

What it hunts

CI hit this once, on test (windows-latest) under -race (run 31982627382):

fatal error: stack not a power of 2
runtime.stackfree(...)  stack.go:468
runtime.gfget(...)      proc.go:5534
runtime.newproc1 / newproc  <- testing.(*T).Run

The runtime pulled a cached goroutine off its free list whose stack size is not
a power of two — runtime state that something corrupted. The secondary panic
showed a write access violation into read-only image memory, with g/m fields
holding image (0x140…) and Windows (0x7FF…) addresses.

Why tooling rather than a fix

Because I could not find the cause, and I would rather say so than ship a guess.

Read-cleared by mechanism, not by glance:

Suspect Why it's cleared
scrubframe_amd64.s Writes exactly [SP, SP+32768) — its own frame, guaranteed present by the prologue's stack check. Never touches R14 (g) or X15.
wipe_amd64.s secureWipe Flush bounds computed before REP STOSB consumes DI/CX; aligned-down start stays in-page.
scrub_frame_test.go Reads dead stack, never writes it.
terminationwipe.go Pure Go, no unsafe.
Windows mapping path Reserve rounded + 2*pageSize PAGE_NOACCESS, commit only the middle, VirtualFree on the reservation base. Guards are reserved-uncommitted, so a touch faults.
securearena.go AddCleanup Closure captures only slabBytes and a uintptr key from regionKey; the janitor map holds the region, so no uintptr-as-live-pointer lifetime bug.
janitor.release Both paths (take, then takeWiped) remove the region before freeing, so no dangling entry survives to be written through.

Not reproduced: ~500 local processes across Go 1.26.5 and 1.26.6, at
GOMAXPROCS=4 to match the runners, plus a GODEBUG=clobberfree=1 pass. No
upstream Go issue matches (golang/go#12192 is 1.4.2-era; #76786 is an unrelated
proposal).

So the remaining lever is sampling, on the image where it actually appeared.

What it does

Builds the race test binary once, then runs it N times on windows-latest with
GOTRACEBACK=system — the default traceback elides runtime frames, and the
runtime frames are the evidence. Failing output goes to the job log in a
::group:: and the first 40 lines to the step summary.

Dispatch inputs: iterations (200), gomaxprocs (4), godebug, package. So
a hypothesis can be tested without editing the file. Daily at 09:00 UTC, clear
of the 07:00 fuzz run. timeout-minutes: 90 so a hang cannot burn six hours.

Two things worth knowing

A green soak is not a clean bill of health, and the workflow says so in its
own summary output rather than leaving a green tick to be misread later. It
bounds the rate; it does not clear the defect.

workflow_dispatch only works once the workflow is on the default branch.
This has to merge before it can hunt anything — it cannot be exercised from this
PR, which is an argument for merging it early rather than late.

The header comment says to delete it once the crash is understood. A soak nobody
reads is worse than no soak, because it looks like coverage.

…crash

CI saw `fatal error: stack not a power of 2` once on windows-latest under
-race: the runtime found a cached goroutine on its free list whose stack size
is not a power of two, reached through gfget from testing.(*T).Run. A re-run of
the same job passed, roughly 500 local processes across Go 1.26.5 and 1.26.6
never reproduced it, and no upstream Go issue matches.

For a library that writes to memory the runtime does not manage, "seen once,
went away" is not a conclusion. One CI run per PR samples this once; this
workflow samples it a few hundred times on the runner image where it actually
appeared, with GOTRACEBACK=system so the runtime frames survive into the log —
the default traceback elides exactly the frames that are the evidence.

Diagnostic tooling, not a gate: nothing depends on it and no branch protection
references it, so a red run is a finding to look at rather than a broken build.
Inputs cover iteration count, GOMAXPROCS, GODEBUG (clobberfree=1 and friends)
and the module directory, so a hypothesis can be tested without editing the
file. Daily at 09:00 UTC, clear of the 07:00 fuzz run.

A green soak is reported honestly as bounding the rate rather than clearing the
defect, because a green tick on a workflow named "soak" is exactly the kind of
thing that later gets cited as proof it was fixed.

The header says to delete this once the crash is understood. A soak nobody
reads is worse than none, because it looks like coverage.
@deadpoets

Copy link
Copy Markdown
Owner Author

Landed on main as 3b6474c via the integration train in #55.

Now that it is on the default branch, workflow_dispatch will register and the daily 09:00 UTC schedule becomes active — it could not be exercised from its own PR.

Rebased locally so the commit carries the maintainer's SSH signature; different SHA, hence the manual close.

@deadpoets deadpoets closed this Aug 18, 2026
@deadpoets
deadpoets deleted the diag/windows-race-soak branch August 18, 2026 23:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant