Repeated self-hosted runner disconnects in SciMLOperators downstream LinearSolve cell

[ChrisRackauckas-Claude]

Summary

The LinearSolve.jl/InterfaceII cell in SciMLOperators' downstream workflow has repeatedly ended because its self-hosted runner lost communication with GitHub Actions. This is an infrastructure issue: the affected job has no Julia test failure to debug, and the same workflow cell later passed on unchanged workflow configuration.

Affected cell:

Downstream LinearSolve.jl/InterfaceII / Downstream Tests - InterfaceII

The workflow invokes SciML/.github/.github/workflows/downstream.yml@v1.

Affected runs

GitHub attached the same runner-level annotation to the LinearSolve cell in each run:

The self-hosted runner lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.

• https://github.com/SciML/SciMLOperators.jl/actions/runs/29223216729 (job 86732259844, commit bd79a08a8f25c06cf5df8c5c8251141c84618665)
• https://github.com/SciML/SciMLOperators.jl/actions/runs/29221485099 (job 86727335539)
• https://github.com/SciML/SciMLOperators.jl/actions/runs/29208191777 (job 86691212542)
• https://github.com/SciML/SciMLOperators.jl/actions/runs/29014025369 (job 86104381113)
• https://github.com/SciML/SciMLOperators.jl/actions/runs/29013087868 (job 86101254571)
• https://github.com/SciML/SciMLOperators.jl/actions/runs/29012530820 (job 86099385760)

In run 29223216729, the downstream test step was still reported as in_progress when the job closed. The job log blob now returns 404, so there is no package-level exception or failed assertion available to investigate. That run started at 2026-07-13 04:11:26 UTC and ended at 04:39:28 UTC.

Control run and bisectability

The same cell passed in https://github.com/SciML/SciMLOperators.jl/actions/runs/29235992715, which started at 2026-07-13 08:35:48 UTC and completed at 08:59:24 UTC. Its log contains substantive Test Summary output. The SciMLOperators commits between the failed run and this green run did not change the downstream workflow or its LinearSolve dependency configuration.

There is therefore no source commit to bisect for these disconnects: GitHub reports termination outside the Julia test process, and the same reusable workflow/cell subsequently completed successfully without a relevant source or workflow fix.

Requested investigation

Please inspect the self-hosted runner service, lifecycle, network connectivity, and resource/OOM logs around the affected timestamps. In particular, it would be useful to determine why this LinearSolve downstream cell repeatedly loses its runner after roughly 20–30 minutes while sibling downstream cells complete.

This issue intentionally tracks only the runner disconnects. Any downstream group-selection cleanup should be handled separately.

[ChrisRackauckas-Claude]

Additional self-hosted runner disconnect observed in SciMLSensitivity PR #1552 (unrelated YAML-only actionlint configuration change): https://github.com/SciML/SciMLSensitivity.jl/actions/runs/29400248355/job/87304080767

The job completed as failure while its test step remained marked in_progress; its check annotation states: The self-hosted runner lost communication with the server. Runner: self-hosted-4vcpu-8gb-vv9dn-runner-fs629. Job: Callbacks1, Julia 1.11. Started 2026-07-15T08:47:05Z; failed 2026-07-15T09:26:08Z.

[ChrisRackauckas-Claude]

Another self-hosted runner disconnect: ModelingToolkit PR #4755, ModelingToolkitBase downgrade job https://github.com/SciML/ModelingToolkit.jl/actions/runs/29396991783/job/87293166606

Runner self-hosted-4vcpu-8gb-vv9dn-runner-mnsfc. The test step started 2026-07-15T07:26:27Z and remained marked in_progress when the job completed as failure at 2026-07-15T09:28:46Z. The check annotation is the standard self-hosted runner lost communication message.

[ChrisRackauckas-Claude]

Additional self-hosted runner disconnect on ModelingToolkit master head 97aec8f91ac95f60739560c4536d3fd1ed53a93a: QA job https://github.com/SciML/ModelingToolkit.jl/actions/runs/29398258063/job/87297059043

Runner self-hosted-4vcpu-8gb-vv9dn-runner-rj6f6. The test step started 2026-07-15T07:51:08Z and remained marked in_progress when the job completed failure at 2026-07-15T08:50:50Z. The annotation is the standard self-hosted runner lost communication message.

[ChrisRackauckas-Claude]

Three additional self-hosted runner disconnects occurred in the current SciMLSensitivity master CI run https://github.com/SciML/SciMLSensitivity.jl/actions/runs/29395878784 at master ad5f2ffc:

• SDE2 / Julia 1.11: https://github.com/SciML/SciMLSensitivity.jl/actions/runs/29395878784/job/87291188158 (runner self-hosted-4vcpu-8gb-vv9dn-runner-65q8m)
• Core7 / Julia 1.11: https://github.com/SciML/SciMLSensitivity.jl/actions/runs/29395878784/job/87291188285 (runner self-hosted-4vcpu-8gb-vv9dn-runner-wjq5m)
• Core1 / Julia 1.11: https://github.com/SciML/SciMLSensitivity.jl/actions/runs/29395878784/job/87291188397 (runner self-hosted-4vcpu-8gb-vv9dn-runner-qsvws)

All three completed as failure while their test steps remained marked in_progress, with the standard self-hosted runner lost communication annotation.

[ChrisRackauckas-Claude]

Additional runner-loss examples and controls from the SciMLBase #1445/#1446 investigation:

Every failing job below has the same distinguishing shape: GitHub's The self-hosted runner ... lost communication with the server annotation, the main test/docs step left in_progress, and later steps still pending. These are runner terminations, not completed Julia test failures.

SciMLBase

• #1445 Downstream / Julia 1
• #1446 LinearSolve.jl/All
• clean master LinearSolve.jl/All
• #1446 Downstream / Julia 1
• #1446 Downstream / Julia pre
• #1446 DownstreamAD / Julia 1

Controls point away from a SciMLBase source regression:

• The exact #1445 base SHA (db9ea39e) passed the same Downstream lane on a GitHub-hosted runner: job 87288090171.
• I also ran that exact SHA locally from a clean depot with Julia +1, GROUP=Downstream, and JULIA_NUM_THREADS=11; it exited 0 with Testing SciMLBase tests passed.
• For recent comparable LinearSolve/All runs, 7/7 routed to self-hosted-4vcpu-8gb-* ended in runner loss, while 3/3 routed to GitHub-hosted runners passed. One direct control is job 87305228663, which checked out the same LinearSolve SHA (6487e3d1) and passed.
• A fresh local four-precompile-task reproduction on Julia 1.10.11, current clean SciMLBase a96e480e, and LinearSolve 6487e3d1 also exited 0 with Testing LinearSolve tests passed.
• That local run's summed descendant-RSS sampler peaked at 9.03 GiB. This is not a physical-memory/OOM measurement because summed RSS can double-count shared pages, but it does indicate the 8 GiB workers may have little headroom during this suite.

SciMLBase #1447's SciMLTesting compat update does not explain the split: the exact pre-#1447 Downstream control already resolved SciMLTesting 2.2.0 and passed, while LinearSolve's own test environment resolves SciMLTesting 1.8.0 and passes on GitHub-hosted/local controls. There is therefore no meaningful source-bisect predicate for these runner losses.

Other repositories

• DiffEqProblemLibrary, OrdinaryDiffEq Regression_I / Julia 1.10
• MethodOfLines #600, docs
• MethodOfLines #600, QA
• FastPower ci(downgrade): fix empty-mode regression + add project input #67 Core, LTS
• FastPower ci(downgrade): fix empty-mode regression + add project input #67 Core, current
• FastPower ci(downgrade): fix empty-mode regression + add project input #67 Core, pre

The three small FastPower Core lanes each lost a separate 4-vCPU/8-GiB runner after only about 10–12 minutes in the test step. That makes this broader than one heavy downstream workload and keeps runner lifecycle/network failure in play alongside possible memory pressure.

Could the ARC/scale-set host lifecycle, kernel OOM, pod eviction, and runner-network logs for these runner names/timestamps be checked? The package logs disappear with the workers, so the infrastructure-side termination reason is the missing evidence.

[ChrisRackauckas-Claude]

One additional SciMLBase #1446 runner loss landed after the rollup above: SymbolicIndexingInterface.jl/Core / Julia 1, job 87324223726, runner self-hosted-4vcpu-8gb-vv9dn-runner-xq7hr. Its main test step (#12) was still in_progress at termination, matching the same runner-lost pattern.

[ChrisRackauckas-Claude]

Additional exact-main downgrade runner loss: LinearSolve run 29869440725, Core job 88765706119 at ebb23fe0beb78b58070fbe44ef8ecd8fec5eef1e.

• Runner: self-hosted-4vcpu-8gb-dqgwf-runner-hxvw6 (ubuntu-latest label).
• Job: 2026-07-21 21:18:54Z–21:43:56Z; the Julia test step started at 21:26:41Z and remained in_progress when the job closed.
• Check annotation: The self-hosted runner lost communication with the server...
• The job-log endpoint returns 404, so there is no Julia exception, failed assertion, or resolver error to bisect.

I replayed the exact checkout locally from a fresh workspace-local depot with Julia 1.10.11 and two threads. julia-actions/julia-downgrade-compat@v2 at exact action commit fab1defb76df9fd672f63c94df73ce131d32e134 successfully produced the strict deps minimum manifest, using the workflow's stdlib skip list and Mooncake no-promote setting. I then ran the repository's official GROUP=Core harness through Pkg.test(; allow_reresolve = false).

Local result: exit 0, all 24 Core groups completed, 1,727 pass / 3 pre-existing broken / 0 fail / 0 error, ending with Testing LinearSolve tests passed. Manifest SHA-256: 5b901c2b445014acdc374f3826fb9d8a44a0c00013ffd27c1f56169d84d0c44e.

This exact-SHA control plus the unfinished hosted test step makes this another instance of the infrastructure signature tracked here; there is no source-level first-bad commit for this occurrence.

[ChrisRackauckas-Claude]

FiniteVolumeMethod.jl PR #127 adds another exact-tree reproduction of this runner-loss pattern.

Failed self-hosted jobs (run 29872534841):

• LTS job 88777224570, runner self-hosted-4vcpu-8gb-dqgwf-runner-f67j9: test step remained in progress from 22:15:03 UTC until the job ended at 22:55:17 UTC. The only annotation is “The self-hosted runner lost communication with the server”; later steps stayed pending.
• prerelease job 88777224625, runner self-hosted-4vcpu-8gb-dqgwf-runner-2gjm6: test step remained in progress from 22:15:23 UTC until the job ended at 23:34:44 UTC, with the same sole annotation and pending later steps.
• Both downloadable-log endpoints now return BlobNotFound/404, so there is no Julia exception, kernel OOM, pod-eviction, or runner-service log available in Actions.

Source control:

• Current main is e9fdb36edcdc7a8a528d47b19aaece5f5da2427e, tree 28a4ec396541dc0f3fd1359bb73b9b6360b21aca.
• PR #126 head 660ae97783fb208168a82442916042e3b23bbf38 has that exact same tree.
• PR #127 changes only test/qa/qa.jl, not Core.

Strong hosted control: PR #126 run 29861895811, on the byte-identical tree and the same reusable workflow/action revisions, routed to GitHub-hosted runners:

• LTS job 88740531862: Core | 1288226 / 1288226 | 37m43.6s, pass.
• prerelease job 88740531876: Core | 1288226 / 1288226 | 78m27.4s, pass.

Exact current-main local reproductions used GROUP=Core, one Julia thread, workspace-local temp/depot paths, and the workflow’s exact invocation:
Pkg.test(; coverage=true, julia_args=["--check-bounds=yes","--compiled-modules=yes","--depwarn=yes"], force_latest_compatible_version=false, allow_reresolve=true).

• Julia 1.10.11: wrapper exit 0; 1288226 / 1288226 passed in 71m07.2s.
• Julia 1.13.0-rc1, fresh solo run: wrapper exit 0; 1288226 / 1288226 passed in 132m39.0s.
• The first prerelease attempt shared the machine with the LTS run and hit its deliberate 120-minute outer timeout while still compute-active; it is excluded from the pass/fail conclusion.

A descendant-tree RSS sampler reached maximum summed RSS values of 13,787,580 KiB (LTS) and 13,842,268 KiB (prerelease solo). This is only resource-pressure evidence: summing per-process RSS double-counts shared pages and does not prove an OOM or eviction. An orphaned compiler from an earlier aborted setup also added contention before it was found and terminated, so the contended prerelease attempt is not quantitative evidence.

There is no source-level bisect predicate here: the same byte-identical tree and command pass on hosted runners and locally, while the failures are runner disconnects with no Julia failure. The useful next evidence is ARC/pod/node/runner-service data around the two job windows (kernel OOM, cgroup limit, eviction, runner restart, network loss). Until that is available, routing these heavy Core variants to hosted or larger-memory runners looks safer than treating the disconnect as a package regression.

[ChrisRackauckas-Claude]

Timing clarification for the FiniteVolumeMethod result above: e9fdb36 was exact upstream/main when both clean-base reproductions were launched. A final fetch after the long prerelease run found PR #127 had merged as current main 0ac4d923ac1b0bc89fafdaefdd61d65a56c2947e. The complete e9fdb36..0ac4d92 delta is only the three-line test/qa/qa.jl change; src, test/core_tests.jl, test/core, docs, and Project.toml are byte-identical. Therefore the Core inputs exercised by both passing runs remain identical to current main.