Skip to content

Preserve old-style test dependency floors in downgrade tests#119

Draft
ChrisRackauckas-Claude wants to merge 2 commits into
SciML:masterfrom
ChrisRackauckas-Claude:agent/preserve-downgraded-test-extras
Draft

Preserve old-style test dependency floors in downgrade tests#119
ChrisRackauckas-Claude wants to merge 2 commits into
SciML:masterfrom
ChrisRackauckas-Claude:agent/preserve-downgraded-test-extras

Conversation

@ChrisRackauckas-Claude

Copy link
Copy Markdown
Member

Ignore this PR until it has been reviewed by @ChrisRackauckas.

Problem

While auditing released SciML packages for minimum-compatible downgrade correctness, I found that julia-actions/julia-downgrade-compat@v2 resolves old-style [extras] / [targets].test dependencies into the root Manifest, but ordinary Pkg.test can replace those orphaned floor entries inside its sandbox.

The deterministic control reproduced on Julia 1.10.11 and 1.12.6:

  • root Manifest: DataStructures 0.17.0
  • unmodified Pkg.test(allow_reresolve=false) sandbox: DataStructures 0.18.22
  • promoted-project sandbox: DataStructures 0.17.0

This matters broadly: the current released-package inventory contains 1,351 registry-backed test extras across 232 projects / 117 repositories after the deliberate source and Mooncake exclusions. It also exposed a real masked ADTypes floor failure.

Changes

  • After downgrade resolution, temporarily promote only [targets].test ∩ [extras] into the active project's [deps].
  • Preserve no_promote and [sources] exclusions.
  • Remove only newly promoted duplicates from [weakdeps].
  • Keep the resolved Manifest untouched through the existing build and test actions.
  • Restore the original Project file byte-for-byte in an always() cleanup step.
  • Apply the same behavior to root and sublibrary downgrade workflows.
  • Leave new-style test/Project.toml handling unchanged.
  • Document the behavior and the previously undocumented mode / no_promote inputs.

Tests

The workflow-embedded Julia source is exercised directly by the repository self-tests and asserted identical in both reusable workflows.

  • Julia 1.10.11: full test/runtests.jl passed.
  • Julia 1.12.6: full test/runtests.jl passed.
  • New coverage: 39 assertions for workflow synchronization/order, target selection, weakdep promotion, [sources] and no_promote, @. and nested projects, new-style no-op behavior, exact restoration on success/failure, and a real Pkg.buildPkg.test locked-floor sandbox.
  • The sandbox test observes FloorFixtureDep 0.1.0 and verifies the Manifest bytes are unchanged.
  • Runic 1.7.0: pass.
  • actionlint + shellcheck: pass.
  • typos: pass.
  • git diff --check: pass.

Local evidence hashes:

  • Julia 1.10.11 log: 38ddbb3d1c07e9fd06a99ae53f3b47fac87deb7804852a670bd10e733a6a94e2
  • Julia 1.12.6 log: ece87b4eff0ba2c4446db191b70c6baced6ac3583c0ca5da5e8bb985746a7a4e
  • Empty successful Runic/actionlint/typos logs: SHA-256 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Process notes

I read the exact action implementation at fab1defb76df9fd672f63c94df73ce131d32e134, traced Julia Pkg's sandbox preservation behavior, reproduced the false green before changing the workflow, validated the production ordering on both supported Julia lines, and then had the final patch reviewed independently. That review caught and removed one non-public test helper and prompted explicit restoration/new-style coverage before this commit was published.

Keep floor-resolved test extras reachable through build and Pkg.test, while preserving source/no-promote exclusions and restoring each Project.toml byte-for-byte. Add cross-version sandbox, cleanup, and workflow synchronization coverage.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>

Copy link
Copy Markdown
Member Author

Validation for head d56b637c373957c15a55b6d0e292bbd04ad9c124 (base f8bbdfe74f4c95803c7b678812afdbca8a066f5e):

  • PATH=... TMPDIR=<workspace>/audit/tmp/... JULIA_DEPOT_PATH=<workspace>/audit/depots/... julia +1.10 --startup-file=no --color=yes test/runtests.jl — exit 0. New sets: synchronization/order 12/12; selection/exclusions 9/9; no-op 2/2; new-style nested environment 3/3; exact restoration 9/9; locked sandbox/Manifest 4/4. Log SHA-256 38ddbb3d1c07e9fd06a99ae53f3b47fac87deb7804852a670bd10e733a6a94e2.
  • Same full suite on Julia 1.12.6 — exit 0 with the same counts. Log SHA-256 ece87b4eff0ba2c4446db191b70c6baced6ac3583c0ca5da5e8bb985746a7a4e.
  • Runic.main(["--check", "--diff", "."]) with Runic 1.7.0 — exit 0.
  • actionlint -color (including shellcheck) — exit 0.
  • typos . — exit 0.
  • git diff --check — exit 0.
  • Independent read-only review found no remaining production, public-API, YAML, shell, restoration, or coverage issue after its two findings were corrected.

The external action itself was not modified. The pre-fix controls and fleet exposure are archived in the broader SciML downgrade audit.

Copy link
Copy Markdown
Member Author

Checked after the one-hour no-poll boundary at 2026-07-21 21:44:43 EDT.

PR #119 remains a draft at exact head
d56b637c373957c15a55b6d0e292bbd04ad9c124, based on current master
f8bbdfe74f4c95803c7b678812afdbca8a066f5e. Git and connector readback agree;
the local and remote feature branches are identical, HEAD..origin/master is
empty, and GitHub reports the PR mergeable with a clean merge state.

Exact-head run
29881032400
completed successfully. All three jobs passed:

  • compute_affected_sublibraries.jl tests (job 88801742762);
  • actionlint + shellcheck (job 88801742783);
  • monorepo TagBot resolver tests (job 88801742794).

The first job's name is narrower than its command: it runs the complete
julia --color=yes test/runtests.jl, including the new old-style-extra
promotion and byte-restoration coverage. Its hosted log includes the 9/9
selection test, 9/9 success/failure restoration test, the locked-floor sandbox,
and all other workflow synchronization/order assertions, with no failure or
error. Thus the production helper is covered both by the previously reported
Julia 1.10/1.12 local runs and by exact-head hosted CI.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
@ChrisRackauckas-Claude

Copy link
Copy Markdown
Member Author

Addendum for commit 4d16630:

The reusable downgrade workflow previously exported an empty GROUP whenever the caller omitted the optional group input. Packages that use a raw environment default such as get(ENV, "GROUP", "All") therefore selected no test group. The workflow now exports GROUP only when the caller supplied a nonempty value, preserving each package default otherwise.

Local validation on the exact committed tree:

  • Julia 1.10.11: full test/runtests.jl passed, including the new 6-assertion workflow regression test.
  • Julia 1.12.6: full test/runtests.jl passed, including the same regression test.
  • Runic 1.7.0: pass.
  • actionlint with shellcheck: pass.
  • typos: pass.
  • git diff --check: pass.

Hosted checks for this new head are still pending.

@ChrisRackauckas-Claude

Copy link
Copy Markdown
Member Author

Hosted CI is terminal for head 4d16630 after the conservative no-poll window. All required checks passed:

  • actionlint + shellcheck: success (job 88814511303)
  • compute_affected_sublibraries.jl tests: success (job 88814511310)
  • monorepo TagBot resolver tests: success (job 88814511323)

Run: https://github.com/SciML/.github/actions/runs/29885291362

The PR remains draft/open, mergeable, and CLEAN.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants