Skip to content

ci(soak): run go test -race ./... by default — the only shape that has reproduced the crash - #60

Closed
deadpoets wants to merge 1 commit into
mainfrom
ci/soak-matches-ci-invocation
Closed

ci(soak): run go test -race ./... by default — the only shape that has reproduced the crash#60
deadpoets wants to merge 1 commit into
mainfrom
ci/soak-matches-ci-invocation

Conversation

@deadpoets

Copy link
Copy Markdown
Owner

The soak was measuring the wrong thing, and its clean results were not the
evidence they looked like.

What happened

The workflow looped a prebuilt single-package test binary. Across 1250
iterations it found nothing — while two ordinary CI runs reproduced the
crash
:

# Run Error
1 31982627382 fatal error: stack not a power of 2 (goroutine free list)
2 32201035150 fatal error: acquireSudog: found s.elem != nil in cache (sudog cache)

ci.yml runs go test -race ./..., which compiles and runs the core and
redact test binaries concurrently. For a bug whose signature is a stray
write landing in runtime free-list caches, what the allocator hands back next
is the entire question — and concurrent processes change that answer. The
prebuilt-binary loop is several times faster per iteration, but it is not the
same experiment.

mode now selects between them and defaults to gotest, because a default
that is fast and has never reproduced is worse than a slow one that has.

The signature is now specific

Both sightings are per-P runtime free-list caches found in an impossible
state:

Sighting Cache Impossible state
1 gfree stack size not a power of two
2 sudog released sudog with elem != nil

In both, the crashing frame is innocuous — TestCapabilities_String, then
t.Parallel(). The goroutine merely trips over damage done earlier. That is a
stray write into memory the Go runtime owns.

One useful discriminator fell out of sighting 2. Making a field non-nil
cannot be done by a zero-wipe — that would set it to nil. So secureWipe is a
weaker suspect than the canary fill (which writes a random pattern) and ordinary
buffer writes. Recorded in the workflow header.

What this does not claim

It does not claim #56 fixes this. That PR's tree is not what crashed, and its
defect wipes another live, mapped secmem buffer — destroying that buffer's
secret without obviously reaching Go-owned memory. A soak against the #56 branch
is running to see whether the rate moves.

Validated: bash syntax checked over the extracted 54-line script, YAML parsed.

… reproduced

The soak looped a prebuilt single-package binary. That ran 1250 iterations and
found nothing, while two ordinary CI runs reproduced the crash — so the soak was
measuring the wrong thing and its clean results were not the evidence they
looked like.

ci.yml runs `go test -race ./...`, which compiles and runs the core and redact
test binaries CONCURRENTLY. For a bug whose signature is a stray write landing
in runtime free-list caches, "what the allocator hands back next" is the entire
question, and concurrent processes change that answer. The prebuilt-binary loop
is faster per iteration but is not the same experiment.

`mode` selects between them and defaults to gotest, because a default that is
fast and has never reproduced is worse than a slow one that has.

Header records the second sighting: acquireSudog found a released sudog with
elem != nil, where the first was the goroutine free list with a non-power-of-two
stack size. Both are per-P free-list caches in an impossible state, and in both
the crashing frame is innocuous — the goroutine trips over earlier damage.

Also notes the discriminator that fell out of sighting 2: making a field NON-nil
cannot be done by a zero-wipe, so the canary fill (random pattern) and ordinary
buffer writes are better suspects than secureWipe.
@deadpoets

Copy link
Copy Markdown
Owner Author

Landed on main as eb1bbfd via the integration train in #61.

The soak now defaults to go test -race ./..., matching what ci.yml actually runs — the only shape that has ever reproduced the corruption. The prebuilt-binary loop remains available via mode=binary.

Rebased locally so the commit carries your SSH signature; different SHA, hence the manual close.

@deadpoets deadpoets closed this Aug 19, 2026
@deadpoets
deadpoets deleted the ci/soak-matches-ci-invocation branch August 19, 2026 00:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant