Skip to content

Add opt-in test group environment isolation#23

Draft
ChrisRackauckas-Claude wants to merge 4 commits into
SciML:mainfrom
ChrisRackauckas-Claude:agent/isolate-test-group-environments
Draft

Add opt-in test group environment isolation#23
ChrisRackauckas-Claude wants to merge 4 commits into
SciML:mainfrom
ChrisRackauckas-Claude:agent/isolate-test-group-environments

Conversation

@ChrisRackauckas-Claude

Copy link
Copy Markdown
Member

Summary

  • add the public isolate_group_environments keyword to run_tests and run_everything
  • run environment-backed All, Everything, and umbrella members in fresh Julia processes when opted in
  • preserve direct named-group and default in-process behavior
  • preserve the active project and Julia runtime flags in child processes
  • bump SciMLTesting to 2.3.0 and document the public keyword

Motivation

Julia cannot unload a package after a test group activates a different environment. Aggregate suites could therefore combine a package module loaded from one environment with source resolved from another. This is reproducible when downstream groups require incompatible LinearSolve versions.

Implementation notes

The child re-enters the outer test entrypoint with the member group selected. Entry-point discovery prefers Base.PROGRAM_FILE and falls back to the external run_tests call site for eval-driven tests. Isolation is lazy and opt-in, so existing callers and directly selected groups retain their current behavior.

Tests performed locally

  • Julia 1.10.11 full test suite: 299 passed, 0 failed
  • Julia 1.12.6 full test suite: 314 passed, 0 failed
  • synthetic same-UUID package fixture verifies default same-process behavior and opt-in fresh-process package selection
  • wrapper re-entry, runtime flag propagation, and explicit test_dir from julia -e are covered
  • whole-repository Runic inplace and check/diff: exit 0
  • git diff --check: exit 0

Process

I reproduced the cross-environment package-version leak, reduced it to a same-process activation mismatch, implemented the opt-in dispatcher path, added regression fixtures, ran both supported local Julia test suites, and ran final repository-wide formatting and hygiene checks.

Review

Ignore this PR until reviewed by @ChrisRackauckas.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
@ChrisRackauckas-Claude

Copy link
Copy Markdown
Member Author

Scratchpad update for commit 48aa317:

  1. Reproduced the failure as a loaded-module mismatch across per-group environments.
  2. Chose an opt-in public keyword with a false default to preserve existing behavior.
  3. Isolated only environment-backed aggregate members; direct named groups stay in process.
  4. Added tests for same-UUID package replacement, default behavior, wrapper entrypoints, child Julia flags, and eval-driven direct groups.
  5. Ran the full suite on Julia 1.10.11 (299 passed) and 1.12.6 (314 passed), followed by whole-tree Runic and diff hygiene checks.

The stacked ModelingToolkit consumer will remain separate and will depend on SciMLTesting 2.3.

@ChrisRackauckas-Claude

Copy link
Copy Markdown
Member Author

Terminal CI follow-up: all 12 checks are green, including Julia LTS/current/prerelease on Linux, macOS, and Windows, plus downgrade, Runic, and spelling. This matches the local full-suite results of 299/299 on Julia 1.10 and 314/314 on Julia 1.12. No CI-specific failure appeared.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Resolve the combined additive public API version as 2.5.0.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Keep version 2.5.0 for the additive isolation API.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
@ChrisRackauckas-Claude

Copy link
Copy Markdown
Member Author

Follow-up audit for 13471de / head 830e813:

While reproducing the MTSL #495 IntegrationTest failure, I found that a fresh isolated group process fixed the loaded-module collision but could silently stop testing a path-developed upstream package. A standalone fresh GROUP=Downstream Pkg.test() was green (31/31), but its nested resolver selected registered ModelingToolkitStandardLibrary v2.29.4 instead of the clean-main checkout under test.

The follow-up preserves the active test sandbox's path-tracked packages in isolated children:

  • the child records Pkg.dependencies() path sources before activating its group environment;
  • those paths join the existing deduplicated single Pkg.develop call;
  • a group's own [sources] entry remains authoritative;
  • the behavior is gated to isolated-child reentry, so directly selected named groups are unchanged.

The regression develops an unregistered UpstreamProbe in the outer project and verifies that the child receives it, while a same-name root/group VersionProbe verifies group [sources] precedence.

Real-graph validation used the stacked MTK isolation change from SciML/ModelingToolkit.jl#4749, with clean-main MTSL and this SciMLTesting checkout developed by path. A narrowed GROUP=All aggregate spawned the actual isolated Downstream child. Its resolver explicitly retained path MTK, MTKBase, MTSL, SciMLTesting, and SciCompDSL; resolved LinearSolve 4.3.0 + ArnoldiMethod 0.4.0; successfully precompiled LinearSolveArnoldiMethodExt; and finished:

Downstream | 31 pass / 31 total | 6m51.5s
Testing ModelingToolkit tests passed

Final validation after merging current main normally (no force push):

  • Julia 1.10.11: 330/330 passed
  • Julia 1.12.6: 345/345 passed
  • repository-wide Runic check passed
  • all tracked TOML parsed; git diff --check passed

The merged additive API version remains 2.5.0. Every new commit includes the required Chris co-author trailer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants