Skip to content

cltest: share one go-build cache across captured runs#2035

Open
cpunion wants to merge 1 commit into
xgo-dev:mainfrom
cpunion:ci-shared-gocache
Open

cltest: share one go-build cache across captured runs#2035
cpunion wants to merge 1 commit into
xgo-dev:mainfrom
cpunion:ci-shared-gocache

Conversation

@cpunion

@cpunion cpunion commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

CI-time investigation follow-up: the stage-5 branches' mac coverage job was running the cl package at 37-45 minutes, right at the 45m go-test alarm.

What this PR does

RunAndCapture created a fresh temp GOCACHE per test and deleted it afterwards, so each of the ~170 golden-dir run tests re-typechecked the whole dependency graph from scratch. This shares one per-process temp cache instead:

  • isolation from the developer cache is unchanged (still a fresh temp dir per test process, removed by the OS temp cleaner);
  • sharing is within one test binary only (sync.Once package state) and tests inside a binary run serially, so no new concurrent access surface is introduced across the parallel test processes;
  • the go build cache is content+flag keyed, so sharing across build configs is safe.

Measurements (single-variable A/B: this one-file diff vs its parent, no -p flags)

  • Local (darwin/arm64, interleaved rounds, heavy run-suites): 348s/308s -> 251s/240s, i.e. 328s -> 245s avg (-25%).
  • CI coverage jobs vs the parent commit: mac 1508s -> 1061s (-30%), ubuntu 1376s -> 779s (-43%).
  • Full ./cl and ./ssa suites green with the change; this PR's CI is green twice (pre- and post-scope-trim).

What was considered and rejected first

  • Removing the ssa TestFrom* suites as duplicates of cl's TestRunAndTestFrom*: rejected — they verify the same golden specs, but per-package coverage attribution means they carry the ssa package's coverage: 87.5% with them, 51.3% without (measured).
  • t.Parallel() on the golden subtests: rejected for now — cltest's capture path swaps os.Stdout/Stderr, os.Chdirs, and mutates GOCACHE globally, and LLVM installs in-process signal handlers; under a cold cache this reproducibly fails (including a hard fatal error: semasleep on Darwin signal stack). Parallelizing needs a capture refactor (explicit cmd.Dir/Env/Stdout), tracked separately.

Scope note

This PR is the cache sharing only. On the much heavier stage-5 branches, warm caches let the four heavy test packages' compile phases overlap fully, and the concurrent clang/lld burst killed 16GB ubuntu runners mid-run; those branches carry a go test -p 2 mitigation locally. That cap is deliberately NOT part of this PR — it lowers the collision probability rather than provably removing the peak; the structural fix (splitting heavy packages into separate CI jobs) is the tracked follow-up if the deaths reappear.

🤖 Generated with Claude Code

Every RunAndCapture created a fresh temp GOCACHE and deleted it after,
so each of the ~170 golden-dir run tests re-typechecked the entire
dependency graph from scratch. Share one per-process temp cache
instead: isolation from the developer cache is unchanged (still a
fresh temp dir per test process), and the go build cache is
content+flag keyed, so sharing across build configs is safe.

Measured on darwin/arm64 (controlled A/B, run-mode _testrt suite,
63 dirs): 157.8s -> 119.6s (-24%). Full ./cl and ./ssa suites green
with the change. The win should be larger on CI mac runners, where
the per-test cache churn hits slow disks; the mac coverage job's cl
package alone runs 37-45 minutes there.
@codecov

codecov Bot commented Jul 6, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant