test: warn when the host is oversubscribed during the test suite, move JuliaLowering_stdlibs to node1#62392
Open
IanButterworth wants to merge 6 commits into
Open
Conversation
IanButterworth
changed the title
IanButterworth
added
test
IanButterworth
force-pushed
the
ib/test-oversubscription-warning
branch
from
a245f96 to
c0aa2f8
Compare
Add a periodic monitor to Base.runtests that prints a warning, alongside the currently-running tests, when the machine's load average exceeds its CPU count. Load average is a whole-machine metric, so this surfaces both tests/subprocesses spawning more threads than cores and contention from other jobs sharing the host. The interval and the load-to-CPU factor are configurable via JULIA_TEST_OVERSUBSCRIPTION_INTERVAL and JULIA_TEST_OVERSUBSCRIPTION_FACTOR. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
IanButterworth
force-pushed
the
ib/test-oversubscription-warning
branch
from
c0aa2f8 to
3fcafff
Compare
Extend the oversubscription warning to annotate each running test with its current CPU usage. Usage is derived by differencing two `ps` snapshots of cumulative CPU time and summing over each test worker's whole process subtree, so CPU spent in subprocesses a test spawns counts toward it, which is where most over-100% usage comes from. CPU time is used rather than the `ps` %CPU column because the latter is unreliable on some platforms (FreeBSD reports ~0). The figure is omitted on Windows. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
IanButterworth
force-pushed
the
ib/test-oversubscription-warning
branch
from
3fcafff to
8cecdc6
Compare
The per-test CPU figure in the oversubscription warning was read from the `ps` `%cpu` column, which is a platform-dependent decaying average: on FreeBSD it reports ~0, so every test showed 0%, and on macOS the column produced no usable figure at all. Sample cumulative CPU time from `ps` instead and difference two snapshots over the reporting interval, summed across each worker's process subtree. This is portable across macOS, Linux and the BSDs. CI showing 0% on FreeBSD: https://buildkite.com/julialang/julia-pr/builds/494 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
FreeBSD's `ps` still showed 0% for every test because `ps -o pid=,ppid=,time=` is parsed differently there: the text after the first `=` is taken as a header, collapsing the output to a single column, so no CPU-time rows were parsed and every delta was zero. macOS and Linux split on the commas first, which is why only FreeBSD was affected. Use separate `-o pid= -o ppid= -o time=` flags, which suppress each header unambiguously on all `ps` variants. CI showing 0% on FreeBSD: https://buildkite.com/julialang/julia-pr/builds/496 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Express the load average and CPU capacity as percentages (100% == one core), the same unit as the per-test figures, and add the summed usage the test workers own, so it can be compared directly against total system load to see how much of the overload is the test suite's versus other processes on the host. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
JuliaLowering_stdlibsto node1 because it was building a sysimage and precompiling stdlibs in parallel to a full breadth of test workers. Massively oversubscribing at times.Before
JuliaLowering_stdlibswas moved to node1: https://buildkite.com/julialang/julia-pr/builds/457#019f666e-515e-4c46-9370-307e25f5627a