Race-Flake Hardening v2
PR #59 closed the first round of race-flake hardening, but go test regularly hangs 5-15 minutes in CI (PRs #144, #146, #156 all hit this). New round of diagnosis + fix.
Why
CI-Reliability is a prerequisite for SOTA-Velocity. A PR that waits 15 minutes on go test is a PR nobody reviews promptly. Throughput drops. Bugs slip through because reviewers are tired.
This is not a "rewrite go test" problem — it's a "find the specific test or pattern that hangs" problem. The fix is targeted, not global.
What ships
Diagnosis pass
- Run
go test -race -count=1 -v ./... on the current main, capture full output
- Identify every test that takes >30 seconds on the first run, >5s on subsequent runs
- Categorize each by root cause: shared SQLite DB, missing httptest cleanup, leaked goroutines, blocking I/O without timeout
internal/testutil/ package
Helpers extracted from the most common patterns:
// IsolatedSQLite returns a *bbolt.DB or *sql.DB opened in t.TempDir(),
// automatically closed at test end. No shared state between tests.
func IsolatedSQLite(t *testing.T) *sql.DB
// CleanEnv sets env vars for the test and restores them at cleanup.
// Replaces the manual os.Setenv + t.Cleanup pattern.
func CleanEnv(t *testing.T, kv map[string]string)
// WithTimeout fails the test if it doesn't return within d.
// Replaces ad-hoc context.WithTimeout scattered across test files.
func WithTimeout(t *testing.T, d time.Duration, fn func())
// GoroutineLeakCheck fails the test if any goroutine started by fn
// is still running when fn returns. Catches the "leaked goroutine
// waits on a channel that never closes" pattern.
func GoroutineLeakCheck(t *testing.T, fn func())
Per-test fixups
For each of the top 5 hanging tests, apply the matching fix. The fixes are mechanical, not architectural.
Acceptance criteria
go test ./cmd/sin-code/ ./cmd/sin-code/internal/ -count=1 completes in under 5 minutes on the current runner pool- The same command with
-race completes in under 10 minutes
- No test relies on a shared global state (SQLite DB, env var, port) that isn't t.TempDir'd or t.Cleanup'd
- New
internal/testutil/ is documented and has ≥ 90% test coverage
- A
docs/CI-RUNBOOK.md section "go test is slow" is added with the top 5 hang reasons
Mandates
- M2 (single binary):
internal/testutil lives in the same module, no new dep
- M7 (race-free): every new test helper is itself race-tested
Related
Estimated scope
~1-2 weeks, 1-2 PRs. Diagnose first, then fix. The diagnostic pass is the high-value part — knowing which test hangs and why.
Race-Flake Hardening v2
PR #59 closed the first round of race-flake hardening, but
go testregularly hangs 5-15 minutes in CI (PRs #144, #146, #156 all hit this). New round of diagnosis + fix.Why
CI-Reliability is a prerequisite for SOTA-Velocity. A PR that waits 15 minutes on
go testis a PR nobody reviews promptly. Throughput drops. Bugs slip through because reviewers are tired.This is not a "rewrite go test" problem — it's a "find the specific test or pattern that hangs" problem. The fix is targeted, not global.
What ships
Diagnosis pass
go test -race -count=1 -v ./...on the current main, capture full outputinternal/testutil/packageHelpers extracted from the most common patterns:
Per-test fixups
For each of the top 5 hanging tests, apply the matching fix. The fixes are mechanical, not architectural.
Acceptance criteria
go test ./cmd/sin-code/ ./cmd/sin-code/internal/ -count=1completes in under 5 minutes on the current runner pool-racecompletes in under 10 minutesinternal/testutil/is documented and has ≥ 90% test coveragedocs/CI-RUNBOOK.mdsection "go test is slow" is added with the top 5 hang reasonsMandates
internal/testutillives in the same module, no new depRelated
docs/CI-RUNBOOK.md§3.4 — thego test: timeoutfailure patternscripts/ci-precheck.sh— the executable that mirrors §13.2; will be updated to use the new helpersEstimated scope
~1-2 weeks, 1-2 PRs. Diagnose first, then fix. The diagnostic pass is the high-value part — knowing which test hangs and why.