From b8c2043db9d916c89fb3fa8143bd71415ae9a48d Mon Sep 17 00:00:00 2001 From: "gastown.mayor" Date: Sat, 1 Aug 2026 15:56:30 -0500 Subject: [PATCH] build: make `go test -p` tunable and document the linker as the memory cost Adds TEST_PKG_PARALLEL (default 4, unchanged) so `-p` can be lowered on memory-constrained hosts, and records in AGENTS.md what actually drives build memory here. The finding behind it (gcy-z78): a package's memory cost is its TEST BINARY'S LINK, not its compile. Measured on internal/beads -- `go build` peaks at 65 MiB, `go test -c` at 1662 MiB; live sampling shows pkg/tool/darwin_amd64/link at 1479 MiB. The predicate holds across all 149 test binaries with zero exceptions: a test binary links ~1.6 GiB iff its dep graph reaches github.com/dolthub/dolt/ -- 58 of 149 packages (39%). internal/api is the largest at 2.44 GiB (1449 deps). Two consequences worth stating in the tree rather than in a bead: - `-p` bounds CONCURRENCY, not a single link. On a host with under ~3 GiB free, no value of -p makes the sweep fit; a single internal/api link exceeds the budget on its own. The knob is useful, not sufficient. - AGENTS.md's gate list said `make test` first while the guidance ~70 lines earlier says to prefer the sharded targets for broad local sweeps. The list now says so too. Scope deliberately kept to this. An earlier version of this change also excluded cmd/gc from the macOS `make test` sweep; that was dropped. It would have moved the sweep's peak only 2.49 -> 2.44 GiB, since internal/api links essentially the same graph, while permanently removing cmd/gc's unit tests from the Mac local signal. Not a trade worth making. Verified: `make -n test` sweeps ./... at -p=4 as before; TEST_PKG_PARALLEL=1 lowers it; test-mac unchanged. Refs: gcy-z78, gcy-bme, gcy-auu Co-Authored-By: Claude Opus 5 (1M context) --- AGENTS.md | 8 +++++++- Makefile | 14 ++++++++++++-- 2 files changed, 19 insertions(+), 3 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 6f84a0fd46..d09ac828bf 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -460,7 +460,13 @@ concurrent builds. Before considering any task complete: - Fast unit baseline passes (`make test`, or `make test-fast-parallel` on - machines where sharding is useful) + machines where sharding is useful — for broad local sweeps prefer the + sharded targets, per `TESTING.md` and the guidance above). On + memory-constrained hosts bound package concurrency with + `TEST_PKG_PARALLEL= make test`: a package's memory cost is its test + binary's link rather than its compile, and 58 of 149 test binaries link + ~1.6 GiB with `internal/api` reaching 2.44 GiB (gcy-z78). `-p` bounds + concurrency only — it cannot bound a single link. - Broader process/integration coverage uses the sharded targets documented in `TESTING.md` instead of one monolithic `go test ./...` sweep - `go vet ./...` clean diff --git a/Makefile b/Makefile index ee14cf3be0..d917f1af16 100644 --- a/Makefile +++ b/Makefile @@ -380,6 +380,16 @@ TEST_ENV = env -i \ CGO_LDFLAGS="$${CGO_LDFLAGS-}" \ $(EXTRA_TEST_ENV) +# TEST_PKG_PARALLEL bounds `go test -p`: how many packages may be built and run +# concurrently. The memory cost of a package is its test binary's LINK, not its +# compile — measured on internal/beads, `go build` peaks at 65 MiB while +# `go test -c` peaks at 1662 MiB. 58 of 149 test binaries link ~1.6 GiB because +# their dep graph reaches github.com/dolthub/dolt/, and internal/api reaches +# 2.44 GiB (gcy-z78). Lower this on memory-constrained hosts. Note it bounds +# concurrency only; it cannot bound a single link, so on a host with less than +# ~3 GiB free no value of -p makes the sweep fit. Default is unchanged. +TEST_PKG_PARALLEL ?= 4 + ## test: run fast unit tests (skip integration-tagged and GC_FAST_UNIT-gated process tests) ## The skipped cmd/gc process-backed scenarios remain covered by ## `make test-cmd-gc-process` locally and the CI `cmd/gc process suite` job. @@ -389,7 +399,7 @@ TEST_ENV = env -i \ ## cache input hashes over local working files. ## Wrapped in $(TEST_ENV) — see comment above for why. test: test-fsys-darwin-compile - $(TEST_ENV) GC_FAST_UNIT=1 scripts/go-test-observable test -- -p=4 -count=1 -timeout 15m ./... + $(TEST_ENV) GC_FAST_UNIT=1 scripts/go-test-observable test -- -p=$(TEST_PKG_PARALLEL) -count=1 -timeout 15m ./... # MAC_UNIT_PKGS excludes cmd/gc from the Mac unit sweep; cmd/gc runs # sharded via the mac-cmd-gc-process CI matrix job instead. @@ -397,7 +407,7 @@ MAC_UNIT_PKGS = $(shell go list ./... | grep -v '/cmd/gc$$') ## test-mac: Mac unit sweep with cmd/gc excluded; cmd/gc covered by the Mac sharded job. test-mac: test-fsys-darwin-compile - $(TEST_ENV) GC_FAST_UNIT=1 scripts/go-test-observable test-mac -- -p=4 -count=1 -timeout 15m $(MAC_UNIT_PKGS) + $(TEST_ENV) GC_FAST_UNIT=1 scripts/go-test-observable test-mac -- -p=$(TEST_PKG_PARALLEL) -count=1 -timeout 15m $(MAC_UNIT_PKGS) LOCAL_TEST_JOBS ?= $(shell nproc 2>/dev/null || getconf _NPROCESSORS_ONLN 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 8)