Skip to content

build: make go test -p tunable and document the linker as the memory cost #5

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
atbrace wants to merge 1 commit into
base: main
Choose a base branch
from
Open

build: make go test -p tunable and document the linker as the memory cost #5

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -460,7 +460,13 @@ concurrent builds.
Before considering any task complete:

- Fast unit baseline passes (`make test`, or `make test-fast-parallel` on
machines where sharding is useful)
machines where sharding is useful — for broad local sweeps prefer the
sharded targets, per `TESTING.md` and the guidance above). On
memory-constrained hosts bound package concurrency with
`TEST_PKG_PARALLEL=<n> make test`: a package's memory cost is its test
binary's link rather than its compile, and 58 of 149 test binaries link
~1.6 GiB with `internal/api` reaching 2.44 GiB (gcy-z78). `-p` bounds
concurrency only — it cannot bound a single link.
- Broader process/integration coverage uses the sharded targets documented in
`TESTING.md` instead of one monolithic `go test ./...` sweep
- `go vet ./...` clean
Expand Down
14 changes: 12 additions & 2 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -380,6 +380,16 @@ TEST_ENV = env -i \
CGO_LDFLAGS="$${CGO_LDFLAGS-}" \
$(EXTRA_TEST_ENV)

# TEST_PKG_PARALLEL bounds `go test -p`: how many packages may be built and run
# concurrently. The memory cost of a package is its test binary's LINK, not its
# compile — measured on internal/beads, `go build` peaks at 65 MiB while
# `go test -c` peaks at 1662 MiB. 58 of 149 test binaries link ~1.6 GiB because
# their dep graph reaches github.com/dolthub/dolt/, and internal/api reaches
# 2.44 GiB (gcy-z78). Lower this on memory-constrained hosts. Note it bounds
# concurrency only; it cannot bound a single link, so on a host with less than
# ~3 GiB free no value of -p makes the sweep fit. Default is unchanged.
TEST_PKG_PARALLEL ?= 4

## test: run fast unit tests (skip integration-tagged and GC_FAST_UNIT-gated process tests)
## The skipped cmd/gc process-backed scenarios remain covered by
## `make test-cmd-gc-process` locally and the CI `cmd/gc process suite` job.
Expand All @@ -389,15 +399,15 @@ TEST_ENV = env -i \
## cache input hashes over local working files.
## Wrapped in $(TEST_ENV) — see comment above for why.
test: test-fsys-darwin-compile
$(TEST_ENV) GC_FAST_UNIT=1 scripts/go-test-observable test -- -p=4 -count=1 -timeout 15m ./...
$(TEST_ENV) GC_FAST_UNIT=1 scripts/go-test-observable test -- -p=$(TEST_PKG_PARALLEL) -count=1 -timeout 15m ./...

# MAC_UNIT_PKGS excludes cmd/gc from the Mac unit sweep; cmd/gc runs
# sharded via the mac-cmd-gc-process CI matrix job instead.
MAC_UNIT_PKGS = $(shell go list ./... | grep -v '/cmd/gc$$')

## test-mac: Mac unit sweep with cmd/gc excluded; cmd/gc covered by the Mac sharded job.
test-mac: test-fsys-darwin-compile
$(TEST_ENV) GC_FAST_UNIT=1 scripts/go-test-observable test-mac -- -p=4 -count=1 -timeout 15m $(MAC_UNIT_PKGS)
$(TEST_ENV) GC_FAST_UNIT=1 scripts/go-test-observable test-mac -- -p=$(TEST_PKG_PARALLEL) -count=1 -timeout 15m $(MAC_UNIT_PKGS)

LOCAL_TEST_JOBS ?= $(shell nproc 2>/dev/null || getconf _NPROCESSORS_ONLN 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 8)

Expand Down
Loading