Repository navigation
Conversation
bhimrazy
requested review from
ethanwharris,
justusschock and
tchaton
as code owners
October 7, 2026 18:57
Contributor
🌩️ Required checks status: Pending 🟡Groups summary🟡 pytorch_lightning: Tests workflow
These checks are required after the changes to 🟢 fabric: Docs
These checks are required after the changes to 🟢 pytorch_lightning: Docs
These checks are required after the changes to 🟢 lightning_fabric: CPU workflow
These checks are required after the changes to 🟢 mypy
These checks are required after the changes to 🟢 install
These checks are required after the changes to Thank you for your contribution! 💜
|
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #22010 +/- ##
=======================================
Coverage 87% 87%
=======================================
Files 270 270
Lines 24015 24015
=======================================
Hits 20855 20855
Misses 3160 3160 |
macos-14 is retired on 2026-11-02, but on macos-26 gloo/spawned-process startup takes ~35s per test with torch>=2.7, pushing pl-cpu/fabric-cpu past their timeouts. Use macos-15 for the test matrices for now and leave a TODO to move to macos-26 in a follow-up. Also stop asserting the rich progress-bar width in hooks.rst; it depends on the it/s text and so on runner speed.
bhimrazy
changed the title
…; move to macos-26 On the macos-15 and macos-26 runner images, macOS Local Network Privacy blocks mDNS queries from non-root processes, so any hostname lookup or reverse-DNS lookup (even of 127.0.0.1 / ::ffff:127.0.0.1) blocks for ~35s before falling back. torch's TCPStore formats the peer address with getnameinfo() on every client connect (c10d socket.cpp, SocketImpl ctor), so each ddp_spawn/gloo test paid ~35s in rendezvous (TCPStore 35.0s vs 0.05s), regardless of torch version, USE_LIBUV, MASTER_ADDR or /etc/hosts entries. Root processes are exempt: run the test command via sudo on macOS (TCPStore 0.05s, fabric spawn tests 2-5s again). See actions/runner-images#14568 and #10924.
bhimrazy
changed the title
lkubb
added a commit
to salt-extensions/salt-extension-copier
that referenced
this pull request
Oct 22, 2026
macOS 15+ runner images block mDNS queries from non-root processes (Local Network Privacy), stalling each unresolvable hostname lookup ~35s. Salt's grains collection performs several unbounded serial lookups per process, exceeding the salt-factories start timeout, so the rendered projects' integration suites failed with FactoryNotStarted on macos-15/26. Root processes are exempt, hence run the test suite via sudo on macOS, both in this repo's CI and in the rendered test workflow. Refs: * actions/runner-images#14568 * actions/setup-python#1223 * saltstack/salt#69840 * Lightning-AI/pytorch-lightning#22010
lkubb
added a commit
to salt-extensions/salt-extension-copier
that referenced
this pull request
Oct 22, 2026
macOS 15+ runner images block mDNS queries from non-root processes (Local Network Privacy), stalling each unresolvable hostname lookup ~35s. Salt's grains collection performs several unbounded serial lookups per process, exceeding the salt-factories start timeout, so the rendered projects' integration suites failed with FactoryNotStarted on macos-15/26. Root processes are exempt, hence run the test suite via sudo on macOS, both in this repo's CI and in the rendered test workflow. Refs: * actions/runner-images#14568 * actions/setup-python#1223 * saltstack/salt#69840 * Lightning-AI/pytorch-lightning#22010
lkubb
added a commit
to salt-extensions/salt-extension-copier
that referenced
this pull request
Oct 23, 2026
macOS 15+ runner images block mDNS queries from non-root processes (Local Network Privacy), stalling each unresolvable hostname lookup ~35s. Salt's grains collection performs several unbounded serial lookups per process, exceeding the salt-factories start timeout, so the rendered projects' integration suites failed with FactoryNotStarted on macos-15/26. Root processes are exempt, hence run the test suite via sudo on macOS, both in this repo's CI and in the rendered test workflow. Refs: * actions/runner-images#14568 * actions/setup-python#1223 * saltstack/salt#69840 * Lightning-AI/pytorch-lightning#22010
lkubb
added a commit
to salt-extensions/salt-extension-copier
that referenced
this pull request
Oct 23, 2026
macOS 15+ runner images block mDNS queries from non-root processes (Local Network Privacy), stalling each unresolvable hostname lookup ~35s. Salt's grains collection performs several unbounded serial lookups per process, exceeding the salt-factories start timeout, so the rendered projects' integration suites failed with FactoryNotStarted on macos-15/26. Root processes are exempt, hence run the test suite via sudo on macOS, both in this repo's CI and in the rendered test workflow. Refs: * actions/runner-images#14568 * actions/setup-python#1223 * saltstack/salt#69840 * Lightning-AI/pytorch-lightning#22010
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Pins CI runners to supported images:
ubuntu-24.04,macos-26,windows-2025.macOS-14retires on 2026-11-02 (with brownouts before) andubuntu-22.04is deprecated from 2026-09-17.ci-tests-pytorch,ci-tests-fabric,ci-pkg-install) use the new labels;DISABLE_MPSnow keys onstartsWith(matrix.os, 'macos').runs-on: ubuntu-22.04jobs moved toubuntu-24.04.sudo -E env "PATH=$PATH") to work around a ~35s DNS stall per process group on macOS 15+ runners (details below).hooks.rstdoctest no longer asserts the rich progress-bar width, which varied with runner speed and failed onubuntu-24.04.Why the macOS tests run as root
On the
macos-15/macos-26images, macOS Local Network Privacy blocks mDNS for non-root processes, so any hostname or reverse-DNS lookup (even of127.0.0.1) blocks for ~35s. torch's TCPStore callsgetnameinfo()on every client connect (socket.cpp), so every ddp_spawn/gloo test paid 35-70s and the macOS jobs hit their timeouts. Root is exempt.getnameinfo(127.0.0.1)test_default_process_groupRuled out (all still 35s+):
/etc/hostsentries,scutil --set HostName,/etc/resolveroverrides, running over SSH,sudo -u runner,USE_LIBUV=0,MASTER_ADDR=localhost. Not torch-version dependent.Upstream: actions/runner-images#14568, actions/runner-images#10924. The
sudoprefix can be dropped once the images are fixed.Other details
ubuntu-latestjobs and thenvidia/cuda ... ubuntu22.04containers (GPU images, not runners).required-jobs(.github/checkgroup.yml) only lists aggregate checks (pl-cpu-guardian,fabric-cpu-guardian, ...), so the OS rename needs no in-repo check-name changes.docs-make (*, linkcheck)(broken links on master, fixed by CI: update transformer engine & 1-bit adam DeepSpeed links in documentation and code comments #21978) andcreate-legacy-ckpts(stale AWS secret, failing on master since 2026-09-14).