stella-tools: custom:: tests flake on a 5s spawn budget under parallel load, producing false red gates

[macanderson]

Problem

Several crates/stella-tools/src/custom.rs tests hard-code a 5000 ms tool timeout (script_tool(..., 5000), and one inline timeout_ms: 5000). That budget covers spawning a real subprocess, and it is not enough on a loaded machine — the failure is a timeout, not an assertion, so it reads as a genuine break:

thread 'custom::tests::oversized_output_is_elided_middle_out' panicked at
crates/stella-tools/src/custom.rs:1348:46:
expected ok: custom tool `t` timed out after 5000ms

Observed 2026-08-07 running cargo test --workspace on main (06da846c) while other cargo jobs were running on the same box:

• custom::tests::oversized_output_is_elided_middle_out
• custom::tests::scalar_inputs_are_exported_as_env_vars
• custom::tests::set_routes_custom_names_and_falls_through_for_others

Confirmed flaky, not broken: re-running the same suite unloaded passes every test — cargo test -p stella-tools --lib custom:: → 36 passed; 0 failed in 4.23 s. The whole workspace run was otherwise clean (82 suites ok, this the only failing one).

Why it is worth fixing rather than tolerating

The failure mode is indistinguishable from a real regression at a glance, and it lands in the required fmt + clippy + test check. Someone diagnosing a red gate spends the expensive minutes before discovering it is load, and — worse — a genuinely red gate can get waved through as "probably that flake again". A timeout in a test whose subject is not timeout behaviour is a measurement artifact, not a property worth asserting.

Build

Raise the budget for every test whose subject is not the timeout itself. DEFAULT_TIMEOUT_MS is already 30 s (custom.rs:106) and MAX_TIMEOUT_MS is 10 min, so these tests are running an order of magnitude tighter than production default for no stated reason. Options, cheapest first:

1. Replace the literal 5000s with a single named test constant (e.g. const TEST_TIMEOUT_MS: u64 = 30_000;) so the budget is stated once and is obviously not load-bearing. Tests that do assert timeout behaviour keep their own short, deliberate value with a comment saying why it is short.
2. If any of these tests is slow enough that 30 s is genuinely tight, that is a separate finding — say so rather than raising the number again.

Call sites to update: custom.rs around lines 1060, 1077, 1098, 1182, 1210, 1312, 1396 (rg -n "5000" crates/stella-tools/src/custom.rs).

Verify

The flake is load-dependent, so "it passes now" proves nothing. Reproduce it first, then show the fix holds under the same load:

# repro: run the suite while the box is busy
cargo build --workspace &  cargo test -p stella-tools --lib custom::

Definition of done: no test in custom:: fails with a timed out after message under that load, and any test that still carries a short timeout has a comment stating that the timeout is the thing under test.

Constraints

• Do not simply #[ignore] them — these cover real behaviour (output elision, env export, name routing).
• Do not raise DEFAULT_TIMEOUT_MS or MAX_TIMEOUT_MS; this is a test-fixture problem, not a production-default problem.

Found while verifying main was green after the #1965/#1970/#1971 unbreaks.