Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
112 changes: 112 additions & 0 deletions .github/workflows/soak-windows.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
name: Windows Race Soak

# Hunts a rare runtime-state corruption that CI has seen exactly once and this
# project cannot reproduce locally.
#
# On 2026-08-17, `test (windows-latest)` died with:
#
# fatal error: stack not a power of 2
# runtime.stackfree(...) stack.go:468
# runtime.gfget(...) proc.go:5534
# runtime.newproc1 / newproc <- testing.(*T).Run
#
# The runtime found a cached goroutine on its free list whose stack size is not
# a power of two — i.e. runtime state that something corrupted. A re-run of the
# same job passed, ~500 local processes never reproduced it, and no upstream Go
# issue matches. For a library whose entire purpose is writing to memory the
# runtime does not manage, "we saw it once and it went away" is not a conclusion
# anyone should accept.
#
# One CI run per PR samples this once. This workflow samples it a few hundred
# times on the same runner image, which is the only place it has ever appeared.
# It is diagnostic tooling, not a gate: nothing depends on it, and a red run here
# is a finding to investigate rather than a broken build.
#
# GOTRACEBACK=system is the point of the exercise — the default traceback elides
# runtime frames, and the runtime frames are the evidence.
#
# Delete this workflow once the crash is understood. A soak that nobody reads is
# worse than no soak, because it looks like coverage.

on:
schedule:
- cron: "0 9 * * *" # 09:00 UTC daily, clear of the 07:00 fuzz run
workflow_dispatch:
inputs:
iterations:
description: "Test processes to run (each is one full package run)"
default: "200"
gomaxprocs:
description: "GOMAXPROCS per process (runners have 4 CPUs; the crash appeared at the default)"
default: "4"
godebug:
description: "GODEBUG for each process, e.g. clobberfree=1 (empty for none)"
default: ""
package:
description: "Module directory to soak"
default: "."

permissions:
contents: read

env:
# Exact patch, kept in step with ci.yml. The one observation was on 1.26.6,
# and a soak against a different toolchain would not be evidence about it.
GO_VERSION: "1.26.6"

jobs:
soak:
runs-on: windows-latest
# Generous but bounded: 200 race-built processes at GOMAXPROCS=4 runs well
# under an hour, and a hang should not burn a 6-hour default.
timeout-minutes: 90
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
with:
go-version: ${{ env.GO_VERSION }}
- uses: ./.github/actions/setup-workspace

- name: soak
shell: bash
working-directory: ${{ github.event.inputs.package || '.' }}
env:
ITERATIONS: ${{ github.event.inputs.iterations || '200' }}
GOMAXPROCS: ${{ github.event.inputs.gomaxprocs || '4' }}
GODEBUG: ${{ github.event.inputs.godebug || '' }}
GOTRACEBACK: system
run: |
set -uo pipefail
go test -race -c -o soak.exe .
echo "soaking $ITERATIONS processes at GOMAXPROCS=$GOMAXPROCS, GODEBUG='${GODEBUG:-}'"

failures=0
# Pure-bash counter: seq is coreutils and not guaranteed on the
# runner's Git bash.
for ((i = 1; i <= ITERATIONS; i++)); do
if ! out=$(./soak.exe -test.count=1 2>&1); then
failures=$((failures + 1))
echo "::group::FAILURE on iteration $i"
echo "$out"
echo "::endgroup::"
# The first ~40 lines carry the fatal error and the faulting
# stack; the full traceback is in the group above.
{
echo "### iteration $i"
echo '```'
echo "$out" | head -40
echo '```'
} >> "$GITHUB_STEP_SUMMARY"
fi
done

echo "$failures failures in $ITERATIONS processes" | tee -a "$GITHUB_STEP_SUMMARY"
# A soak that goes green tells us the rate is below what this sample
# can see — NOT that the bug is gone. Say so, so a green tick here is
# never mistaken for a clean bill of health.
if [ "$failures" -eq 0 ]; then
echo "No failures observed. This bounds the rate; it does not clear the defect." \
>> "$GITHUB_STEP_SUMMARY"
exit 0
fi
exit 1
30 changes: 30 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,36 @@ mark the stability commitment.
> This repo holds three independently versioned Go modules; entries are tagged
> by module. Untagged entries belong to the core `secmem` module.

### Changed

- **`secmem-crypto`, `examples`: `golang.org/x/crypto` 0.54.0 → 0.55.0.**
Maintenance, not a fix: the `vuln` job was green against 0.54.0, so nothing
outstanding was reachable. Recorded because a `require` change in
`secmem-crypto` raises the floor for everyone importing it, which is the same
reason `secmem-crypto/v0.3.1` was a dependency-only release with an entry of
its own.

The two modules have to move together. `examples` pins `secmem-crypto` with a
`replace`, but a replace does not exempt the `require` line from minimum
version selection: bumping only `secmem-crypto` makes MVS select 0.55.0 for
`examples` too, while `examples/go.mod` still asks for 0.54.0 — and CI runs
readonly, so that is a hard error before any package loads.

### Fixed

- `WipeAllSecrets` no longer lets one borrowed buffer strand another's secret.
The emergency wipe's second pass blocked on the deferred regions sequentially,
in map-iteration order. Because `tryWipeInPlace` defers a region whose lock is
held at the instant it looks — including momentarily — a buffer that was
merely mid-`WithBytes` during the first pass could be serialized behind a
genuinely stuck borrow and keep its plaintext for as long as that borrow ran.
That is precisely the hostage situation the two-pass split exists to prevent,
reintroduced by the second pass itself. Each deferred region is now waited on
independently, so ordering cannot matter. Present since the two-pass wipe
landed, and reachable on any `WipeAllSecrets` call — including from
`InstallTerminationWipe` — whenever a second buffer was in use at the moment
the wipe began.

## [secmem-crypto/v0.3.2] - 2026-08-16

Retracts `secmem-crypto/v0.3.0`, and documents the module.
Expand Down
2 changes: 1 addition & 1 deletion examples/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ go 1.26
require (
github.com/deadpoets/secmem v0.3.0
github.com/deadpoets/secmem/secmem-crypto v0.1.0
golang.org/x/crypto v0.54.0
golang.org/x/crypto v0.55.0
)

require (
Expand Down
4 changes: 2 additions & 2 deletions examples/go.sum
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
filippo.io/edwards25519 v1.2.0 h1:crnVqOiS4jqYleHd9vaKZ+HKtHfllngJIiOpNpoJsjo=
filippo.io/edwards25519 v1.2.0/go.mod h1:xzAOLCNug/yB62zG1bQ8uziwrIqIuxhctzJT18Q77mc=
golang.org/x/crypto v0.54.0 h1:YLIA59K4fiNzHzjnZt2tUJQjQtUWfWbeHBqKtk3eScw=
golang.org/x/crypto v0.54.0/go.mod h1:KWL8ny2AZdGR2cWmzeHrp2azQPGogOv+HeQaVEXC2dk=
golang.org/x/crypto v0.55.0 h1:+KWHjbgOaAQ66dh/YlkZKHlz9ZUlq61AFirAR9ntP8M=
golang.org/x/crypto v0.55.0/go.mod h1:uq0V9dE/fzQuJtbnL+2EhWOE63vo164FY8xqEnV9xis=
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/term v0.45.0 h1:NwWyBmoJCbfTHpxrWoZ9C6/VxOf7ic219I8xZZFdrf0=
Expand Down
46 changes: 38 additions & 8 deletions registry.go
Original file line number Diff line number Diff line change
Expand Up @@ -350,11 +350,26 @@ func (j *janitor) tryWipeInPlace(key uintptr) (done bool, err error) {
// each, and returns any canary/wipe errors joined.
//
// Two passes, deliberately. The first wipes every region whose lock is free at
// that instant; the second blocks on whatever is left. A single goroutine parked
// inside a long WithBytes callback therefore delays only ITS OWN buffer — every
// other secret in the process is already zeroed by the time the first pass
// returns. A one-pass loop would let that one borrow hold the whole emergency
// wipe hostage in registry-map order.
// that instant; the second blocks on whatever is left, one goroutine per region.
// A single goroutine parked inside a long WithBytes callback therefore delays
// only ITS OWN buffer — every other secret in the process is either already
// zeroed by the first pass or wiped the moment its own lock frees. A one-pass
// loop would let that one borrow hold the whole emergency wipe hostage in
// registry-map order.
//
// The second pass must not be a sequential loop, which is the bug this shape
// fixes. tryWipeInPlace reports "not done" for a lock held at the instant it
// looks, including a momentary one, so a buffer that is merely mid-WithBytes
// during the first pass lands in deferred alongside the genuinely stuck one.
// Blocking on those in slice order — which is map-iteration order, i.e. random —
// then serializes an innocent buffer behind the long borrow, recreating exactly
// the hostage situation the two-pass split exists to prevent. Waiting on each
// region independently makes the ordering irrelevant.
//
// The cost is one goroutine per region still locked at the end of the first
// pass. That set is normally empty and is bounded by the number of live
// registrations; bounding it with a worker pool would reintroduce the bug as
// soon as the stuck borrows outnumbered the workers.
//
// The already-wiped set is swept too. Those regions are still mapped and their
// owners are still usable, so anything written to one since the last wipe is a
Expand Down Expand Up @@ -384,11 +399,26 @@ func (j *janitor) wipeAllInPlace() error {
deferred = append(deferred, key)
}
}
if len(deferred) == 0 {
return errs
}

var (
mu sync.Mutex
wg sync.WaitGroup
)
wg.Add(len(deferred))
for _, key := range deferred {
if err := j.wipeInPlace(key); err != nil {
errs = errors.Join(errs, err)
}
go func() {
defer wg.Done()
if err := j.wipeInPlace(key); err != nil {
mu.Lock()
errs = errors.Join(errs, err)
mu.Unlock()
}
}()
}
wg.Wait()
return errs
}

Expand Down
118 changes: 118 additions & 0 deletions registry_emergency_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ package secmem
import (
"bytes"
"errors"
"sync"
"testing"
"time"
)
Expand Down Expand Up @@ -89,6 +90,123 @@ func TestWipeAllSecrets_UnborrowedBuffersWipeWhileOneIsBorrowed(t *testing.T) {
}
}

// TestWipeAllSecrets_TransientBorrowDoesNotStrandOtherSecrets pins the second
// pass of the emergency wipe.
//
// The sibling test above covers a buffer that is free during the first pass.
// This one covers the case that actually regressed: a buffer that is borrowed
// only MOMENTARILY. tryWipeInPlace reports "not done" for a lock held at the
// instant it looks, so such a buffer is demoted into the deferred set next to a
// genuinely stuck one. While the second pass blocked on those in slice order —
// map-iteration order, i.e. random — an innocent buffer landing behind the long
// borrow was held hostage, which is the very thing the two-pass split exists to
// prevent.
//
// The interleaving is forced rather than raced, and the detector is
// deterministic rather than probabilistic:
//
// - Both buffers are borrowed BEFORE WipeAllSecrets is called, so the first
// pass cannot take either and both are guaranteed to land in the deferred
// set. No sleep and no timing assumption.
// - The fixed wipe waits on every deferred region independently, so each has a
// writer queued on its OWN lock. A sequential second pass can only ever have
// one queued at a time. Requiring a queued writer on BOTH locks therefore
// fails against the unfixed code regardless of which key map iteration put
// first — the ordering that made the original flake intermittent cannot
// rescue it here.
// - The stuck borrow is never released while the assertion runs, so the idle
// buffer can only reach zero if the wipe waited on it independently.
func TestWipeAllSecrets_TransientBorrowDoesNotStrandOtherSecrets(t *testing.T) {
if !platformHasSecureMemory {
t.Skip("no secure memory on this platform")
}
secret := bytes.Repeat([]byte{0x5A}, 40)

newBuf := func(what string) *SecureBuffer {
b, err := NewBuffer(append([]byte(nil), secret...))
if err != nil {
// Allocation failure is an environment condition, not a defect.
t.Skipf("NewBuffer(%s): %v", what, err)
}
return b
}
stuck := newBuf("stuck")
defer func() { _ = stuck.Destroy() }()
idle := newBuf("idle")
defer func() { _ = idle.Destroy() }()

var borrows sync.WaitGroup
var stuckOnce, idleOnce sync.Once
releaseStuck := make(chan struct{})
releaseIdle := make(chan struct{})
freeStuck := func() { stuckOnce.Do(func() { close(releaseStuck) }) }
freeIdle := func() { idleOnce.Do(func() { close(releaseIdle) }) }

// borrow parks a goroutine inside WithBytes and returns once the callback is
// confirmed to be running, so the region's lock is definitely held on return.
borrow := func(b *SecureBuffer, until <-chan struct{}) {
entered := make(chan struct{})
borrows.Add(1)
go func() {
defer borrows.Done()
_ = b.WithBytes(func([]byte) {
close(entered)
<-until
})
}()
<-entered
}

wipeDone := make(chan error, 1)

// Teardown runs LIFO, so this is registered in reverse of the order it must
// happen: release both borrows, let the wipe finish, and only then let the
// deferred Destroy calls above run. A Destroy racing a live borrow would
// block forever and hang the test instead of failing it, and every assertion
// below can abort, so both releases have to be idempotent and unconditional.
defer func() { <-wipeDone }()
defer borrows.Wait()
defer freeStuck()
defer freeIdle()

borrow(stuck, releaseStuck)
borrow(idle, releaseIdle)

go func() { wipeDone <- WipeAllSecrets() }()

// The deterministic detector: both deferred regions must be waited on at the
// same time. A sequential second pass can only ever queue one writer, so it
// fails here whichever key map iteration happened to put first.
waitForWritersWaiting(t, stuck.mu, 1)
waitForWritersWaiting(t, idle.mu, 1)

// End to end: releasing ONLY the idle borrow must zero that buffer while the
// unrelated borrow is still held.
freeIdle()
deadline := time.Now().Add(10 * time.Second)
for {
zeroed := true
if err := idle.WithBytes(func(p []byte) {
for _, x := range p {
if x != 0 {
zeroed = false
}
}
}); err != nil {
t.Fatalf("idle.WithBytes: %v", err)
}
if zeroed {
break
}
if time.Now().After(deadline) {
t.Error("idle buffer still holds its secret while an unrelated buffer is borrowed — " +
"the emergency wipe's blocking pass serialized it behind another region's borrow")
return
}
time.Sleep(time.Millisecond)
}
}

// TestWipeAllSecrets_DestroyReclaimsMapping covers the deferred unmap. The
// emergency wipe deliberately leaves regions mapped so a late access reads
// zeros instead of faulting — but once the owner calls Destroy it has stated
Expand Down
8 changes: 6 additions & 2 deletions sealcipher_windows_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -117,9 +117,13 @@ func TestSealCipher_RealOverflowStillDetected(t *testing.T) {
t.Fatalf("NewEmptyBuffer: %v", err)
}

// Overflow into the canary slack (unsealed, plaintext state).
// Overflow into the canary slack (unsealed, plaintext state). corruptCanary,
// not a fixed byte: the pattern is random and process-global, so writing a
// literal 0x00 here plants no overflow at all on the ~1/256 of runs where
// that pattern byte is already 0x00, and this test then fails claiming the
// cipher masked a violation that was never inflicted.
base := uintptr(unsafe.Pointer(&buf.region.inner[0]))
probeWrite(base+uintptr(cap(buf.data)), 0x00)
corruptCanary(base + uintptr(cap(buf.data)))

// Cycle the cipher: encrypts the corrupted slack, decrypts it back —
// the corruption must survive the round trip and still be detected.
Expand Down
2 changes: 1 addition & 1 deletion secmem-crypto/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ go 1.26
require (
filippo.io/edwards25519 v1.2.0
github.com/deadpoets/secmem v0.3.0
golang.org/x/crypto v0.54.0
golang.org/x/crypto v0.55.0
)

require golang.org/x/sys v0.47.0 // indirect
Expand Down
4 changes: 2 additions & 2 deletions secmem-crypto/go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@ filippo.io/edwards25519 v1.2.0 h1:crnVqOiS4jqYleHd9vaKZ+HKtHfllngJIiOpNpoJsjo=
filippo.io/edwards25519 v1.2.0/go.mod h1:xzAOLCNug/yB62zG1bQ8uziwrIqIuxhctzJT18Q77mc=
github.com/deadpoets/secmem v0.3.0 h1:CKSnqP/yvmQuAyipKzknUPEwsYc6leQ3AnxjIFazkJQ=
github.com/deadpoets/secmem v0.3.0/go.mod h1:E3WiQ0gBANyIV5SiXQ4oRrasbv5aZ5zC3gYwbyf+DcQ=
golang.org/x/crypto v0.54.0 h1:YLIA59K4fiNzHzjnZt2tUJQjQtUWfWbeHBqKtk3eScw=
golang.org/x/crypto v0.54.0/go.mod h1:KWL8ny2AZdGR2cWmzeHrp2azQPGogOv+HeQaVEXC2dk=
golang.org/x/crypto v0.55.0 h1:+KWHjbgOaAQ66dh/YlkZKHlz9ZUlq61AFirAR9ntP8M=
golang.org/x/crypto v0.55.0/go.mod h1:uq0V9dE/fzQuJtbnL+2EhWOE63vo164FY8xqEnV9xis=
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/term v0.45.0 h1:NwWyBmoJCbfTHpxrWoZ9C6/VxOf7ic219I8xZZFdrf0=
Expand Down
Loading