persistent-tasks: race-tolerant retry in run_qa (fix flaky Aqua failure fleet-wide)

[ChrisRackauckas-Claude]

Please ignore until reviewed by @ChrisRackauckas. Draft, opened by an agent.

The durable, fleet-wide fix for the flaky Aqua persistent-tasks QA failure (SciML/SciMLSensitivity#1554), so repos stop needing per-repo workarounds.

Root cause

run_qa's persistent-tasks check (via Aqua.test_all → Aqua.test_persistent_tasks) cold-precompiles a throwaway wrapper of the package in a subprocess. On a large top-of-stack dependency graph (e.g. SciMLSensitivity), that parallel precompile intermittently trips Julia's <module> is missing from the cache … may be precompilable after restarting julia race; the child exits before loading the wrapper, so its done.log sentinel is never written and Aqua misreports it as a persistent task. The known [sources]-stripping workaround (with_clean_persistent_tasks_sources) does not cover this — it's a precompile race, not a [sources] error.

Fix

run_qa now disables Aqua's single-shot persistent_tasks sub-check and runs a race-tolerant _has_persistent_tasks_with_retry instead: it re-runs the wrapper precompile only when the child exits without writing done.log. Precompilation stays parallel (fast); the retry recompiles just the few modules the race left uncached.

Serial precompile was tried and rejected — it removes the race but is ~6.6× slower, writing done.log ~837s before a cold serial tree finishes and tripping Aqua's tmax=10s (verified 3/3 cold failures). Retry is the only approach that passes cold without slowing the probe.

The genuine check is unchanged: a real persistent task loads fine and then holds the process open past tmax, still detected — the retry only fires on the missing-done.log race, unreachable for a real task. Runs inside with_clean_persistent_tasks_sources (keeps broken-[sources] stripping) and is skipped when persistent-tasks is opted out (:persistent_tasks in aqua_broken, or persistent_tasks = false in aqua_kwargs).

Verification (Julia 1.12.6)

• SciMLTesting's own suite: 345/345 pass with the change (the stand-in run_qa tests exercise the retry on the warm/happy path).
• Cold-cache, real path (with_clean_persistent_tasks_sources(() -> @test !_has_persistent_tasks_with_retry(SciMLSensitivity)), JULIA_NUM_PRECOMPILE_TASKS=4, wiped cache): 3/3 pass, the race fired each time (Accessors … missing from the cache), the retry logged + recovered, done.log written on the warm retry, JULIA_NUM_PRECOMPILE_TASKS unchanged (no leak), active project restored. The unpatched single-shot fails cold; serial trips tmax.

Runic-clean.

Follow-up

Once released: SciML/SciMLSensitivity's test/QA/aqua.jl reverts to a plain run_qa(SciMLSensitivity; …) (dropping its local retry) — see the companion SciMLSensitivity PR. Other repos with local persistent-tasks workarounds can do the same. The truly-upstream fix belongs in Aqua itself (race-tolerant persistent-tasks precompile, JuliaTesting/Aqua.jl#315); this is the SciML-fleet fix until that lands.

🤖 Generated with Claude Code