ci: dispatchable Windows race soak to hunt the one-off runtime crash - #54
Closed
deadpoets wants to merge 1 commit into
Closed
ci: dispatchable Windows race soak to hunt the one-off runtime crash#54deadpoets wants to merge 1 commit into
deadpoets wants to merge 1 commit into
Conversation
…crash CI saw `fatal error: stack not a power of 2` once on windows-latest under -race: the runtime found a cached goroutine on its free list whose stack size is not a power of two, reached through gfget from testing.(*T).Run. A re-run of the same job passed, roughly 500 local processes across Go 1.26.5 and 1.26.6 never reproduced it, and no upstream Go issue matches. For a library that writes to memory the runtime does not manage, "seen once, went away" is not a conclusion. One CI run per PR samples this once; this workflow samples it a few hundred times on the runner image where it actually appeared, with GOTRACEBACK=system so the runtime frames survive into the log — the default traceback elides exactly the frames that are the evidence. Diagnostic tooling, not a gate: nothing depends on it and no branch protection references it, so a red run is a finding to look at rather than a broken build. Inputs cover iteration count, GOMAXPROCS, GODEBUG (clobberfree=1 and friends) and the module directory, so a hypothesis can be tested without editing the file. Daily at 09:00 UTC, clear of the 07:00 fuzz run. A green soak is reported honestly as bounding the rate rather than clearing the defect, because a green tick on a workflow named "soak" is exactly the kind of thing that later gets cited as proof it was fixed. The header says to delete this once the crash is understood. A soak nobody reads is worse than none, because it looks like coverage.
Owner
Author
|
Landed on Now that it is on the default branch, Rebased locally so the commit carries the maintainer's SSH signature; different SHA, hence the manual close. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Diagnostic tooling for the open item from #53 — not a fix, and not a gate.
What it hunts
CI hit this once, on
test (windows-latest)under-race(run 31982627382):The runtime pulled a cached goroutine off its free list whose stack size is not
a power of two — runtime state that something corrupted. The secondary panic
showed a write access violation into read-only image memory, with
g/mfieldsholding image (
0x140…) and Windows (0x7FF…) addresses.Why tooling rather than a fix
Because I could not find the cause, and I would rather say so than ship a guess.
Read-cleared by mechanism, not by glance:
scrubframe_amd64.s[SP, SP+32768)— its own frame, guaranteed present by the prologue's stack check. Never touches R14 (g) or X15.wipe_amd64.s secureWipeREP STOSBconsumes DI/CX; aligned-down start stays in-page.scrub_frame_test.goterminationwipe.gorounded + 2*pageSizePAGE_NOACCESS, commit only the middle,VirtualFreeon the reservation base. Guards are reserved-uncommitted, so a touch faults.securearena.goAddCleanupslabBytesand auintptrkey fromregionKey; the janitor map holds the region, so no uintptr-as-live-pointer lifetime bug.janitor.releasetake, thentakeWiped) remove the region before freeing, so no dangling entry survives to be written through.Not reproduced: ~500 local processes across Go 1.26.5 and 1.26.6, at
GOMAXPROCS=4to match the runners, plus aGODEBUG=clobberfree=1pass. Noupstream Go issue matches (golang/go#12192 is 1.4.2-era; #76786 is an unrelated
proposal).
So the remaining lever is sampling, on the image where it actually appeared.
What it does
Builds the race test binary once, then runs it N times on
windows-latestwithGOTRACEBACK=system— the default traceback elides runtime frames, and theruntime frames are the evidence. Failing output goes to the job log in a
::group::and the first 40 lines to the step summary.Dispatch inputs:
iterations(200),gomaxprocs(4),godebug,package. Soa hypothesis can be tested without editing the file. Daily at 09:00 UTC, clear
of the 07:00 fuzz run.
timeout-minutes: 90so a hang cannot burn six hours.Two things worth knowing
A green soak is not a clean bill of health, and the workflow says so in its
own summary output rather than leaving a green tick to be misread later. It
bounds the rate; it does not clear the defect.
workflow_dispatchonly works once the workflow is on the default branch.This has to merge before it can hunt anything — it cannot be exercised from this
PR, which is an argument for merging it early rather than late.
The header comment says to delete it once the crash is understood. A soak nobody
reads is worse than no soak, because it looks like coverage.