Skip to content

test: run the suite in parallel (bats --jobs), 48 min -> ~4.5 min - #296

Merged
AltDoug merged 2 commits into
mainfrom
perf/parallel-test-suite
Oct 30, 2026
Merged

AltDoug merged 2 commits into
mainfrom
perf/parallel-test-suite

Conversation

@AltDoug

@AltDoug AltDoug commented Oct 30, 2026

Copy link
Copy Markdown
Owner

What

  • CI installs GNU parallel (brew / apt) and runs bats --jobs <CPUs> tests/; serial bats tests/ when parallel is missing.
  • Three tests made safe to run side by side:
    • tests/helpers.bash: one FOUND_ISSUES_CACHE_DIR per test (BATS_TEST_TMPDIR). Auto-fix run logs (FI_AF_RUNS) live there; the classify tests read each other's.
    • tests/autofix-cancel.bats: sleep lengths unique to the test process plus an exact pkill, instead of pkill -f 'sleep 471[34]', which killed other tests' processes. tests/fix-batch8-long-root.bats had the same pkill without ever starting those sleeps; removed.
    • tests/cli-sync-default-branch.bats: its own XDG_CACHE_HOME, instead of the real ~/.cache plus a teardown that emptied it for every test.
  • CONTRIBUTING.md: how to run the suite in parallel.

Why

Ledger tests/autofix-sweep.bats (operator decision 2026-10-29): the serial suite reached 48 min on the Mac against the 60-min fix ship gate.

Evidence

  • Serial, 2026-10-29: 1..2055, 2055 ok, about 48 min.
  • bats --jobs 14 tests/ before the fixes: 258 s, 4 failures (cancel ×2, classify ×2). After them: 254 s, 261 s, 264 s, each 1..2055 ok=2055 notok=0, no leftover sleeps.
  • The CI step's logic was driven locally: with parallel on PATH it printed bats jobs: 14; without it, bats jobs: 1 and the serial path.

Also logs tests/helpers.bash:25: tests that keep the real HOME write ~/.cache/found-issues (measured: mode_foo_bar rewritten during a suite run).

🤖 Generated with Claude Code

https://claude.ai/code/session_01MVx3HWniZnrXYfPiCo5Lrs

AltDoug and others added 2 commits October 29, 2026 23:32
CI installs GNU parallel and runs one bats job per CPU (serial when
parallel is absent). Measured on the Mac: 48 min serial for 2055 tests,
254-264 s with --jobs 14 (3 clean runs in a row).

Three tests were not safe side by side:
- tests/helpers.bash: one cache dir per test (BATS_TEST_TMPDIR); auto-fix
  run logs (FI_AF_RUNS) live there and the classify tests mixed them.
- tests/autofix-cancel.bats: sleep lengths unique to the test process and
  an exact pkill, instead of pkill 'sleep 471[34]' killing other tests'.
  tests/fix-batch8-long-root.bats carried the same pkill without ever
  starting those sleeps; removed.
- tests/cli-sync-default-branch.bats: its own XDG_CACHE_HOME instead of
  the real ~/.cache plus a teardown that emptied it for everyone.

Logs tests/helpers.bash:25 (tests keeping the real HOME write ~/.cache).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVx3HWniZnrXYfPiCo5Lrs
@AltDoug
AltDoug enabled auto-merge (squash) October 30, 2026 03:33
@AltDoug
AltDoug merged commit 9192fd8 into main Oct 30, 2026
7 checks passed
@AltDoug
AltDoug deleted the perf/parallel-test-suite branch October 30, 2026 03:50
AltDoug added a commit that referenced this pull request Oct 30, 2026
…0, tests/autofix-sweep.bats) (#297)

Claude-Session: https://claude.ai/code/session_01MVx3HWniZnrXYfPiCo5Lrs

Co-authored-by: Diogo Silva Sena <227921880+DougBTW@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant