Remove pytorch channel from conda env file and upper bound deepspeed - #394
Open
jnwei wants to merge 3 commits into
Open
Remove pytorch channel from conda env file and upper bound deepspeed#394jnwei wants to merge 3 commits into
jnwei wants to merge 3 commits into
Conversation
The conda CI job builds with BUILD_MODE=yaml, so environments/production-linux-64.yml is re-solved whenever the docker layer cache misses. That env asked for `pytorch::pytorch`, and the pytorch channel stopped publishing after 2.5.1, while `deepspeed` was unbounded. deepspeed 0.19.6 (2026-08-27) registers custom ops annotated `list[int]`, which torch.library.infer_schema only accepts from torch 2.7 onwards, so any fresh solve produced an environment where `import deepspeed` raises ValueError and every test errors at conftest import. - Take pytorch (and its CUDA runtime) from conda-forge via `pytorch-gpu`, matching feature.pytorch-conda-cuda in pixi.toml, and keep it in the CUDA 12 family. - Drop the now-unused pytorch and nvidia channels. - Bound deepspeed to >=0.19.2,<0.20 so an upstream release cannot silently change the required torch version again. - Export CONDA_OVERRIDE_CUDA while solving, as docker/Dockerfile.pixi already does, since conda-forge CUDA builds are gated on the __cuda virtual package that a GPU-less build machine does not advertise. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018n3563hHX7tvKSqBAiUtio
The PyPI route cannot be locked any more: deepspeed publishes sdists only, so conda-lock has to build its metadata, which pulls torch from PyPI, whose CUDA 13 dependency `nvidia-cublas` ships only `manylinux_2_27` wheels — a tag missing from conda-lock's MANYLINUX_TAGS, so the solve dies with "Unable to find installation candidates for nvidia-cublas". Taking deepspeed from conda-forge instead (as feature.deepspeed does in pixi.toml for linux-64) sidesteps that and is better anyway: those builds carry real pytorch run constraints, so the solver rejects an incompatible pair instead of letting it fail at import time. It resolves to deepspeed 0.19.6 + pytorch 2.10.0, the same torch the pixi environments lock. environments/production-linux-64.lock is regenerated from the updated environment file, so BUILD_MODE=lock and BUILD_MODE=yaml describe the same environment again: pytorch 2.5.1 (pytorch channel) -> 2.10.0 (conda-forge, cuda129), python 3.12 -> 3.13, deepspeed 0.19.2 (pip) -> 0.19.6 (conda-forge). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018n3563hHX7tvKSqBAiUtio
Contributor
Author
|
Paging @sdvillal for a quick review in case I've missed anything here |
jnwei
added a commit
that referenced
this pull request
Sep 3, 2026
…ound on CUDA runners, PR #394) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rw9iP39tQnStDVNmXRFsw9
The three integration workflows each hardcoded the checkpoint's S3 key and file name, and each gated the download on `cache-hit != 'true'`. That gate is unsound: a cache restore only helps if the archive lands in the directory that gets mounted into the container, and actions/cache stores paths relative to GITHUB_WORKSPACE, so an archive saved by the self-hosted AMD runner (workspace one level deeper than the AWS runners') unpacks somewhere else on the ephemeral runners. The step was skipped, the mount was empty, and every AWS job died with "Default checkpoint openbind-2025-06-30-174k not found in /root/.openfold3". Run `setup_openfold --non-interactive` inside the test image instead: - The checkpoint file name comes from OPENFOLD_MODEL_CHECKPOINT_REGISTRY rather than three copies in YAML, so a rename can no longer desync the workflows from the code. - download_model_parameters() skips a file that is already on disk, so a good cache restore still costs nothing and a bad one costs a download instead of a red run. - It runs in the container, so the host needs neither the AWS CLI nor credentials. That removes the reason the AMD job used curl, and all three paths now agree. - OpenFoldSetupConfig defaults download only the default checkpoint and leave run_integration_tests False, so setup does not run the suite; the next step does. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018n3563hHX7tvKSqBAiUtio
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A bad interaction between outdated pytorch from the pytorch conda channel and deepspeed is causing the unit test failures seen here: https://github.com/aqlaboratory/openfold-3/actions/runs/33709917312/job/100507354489?pr=393
pytorch:pytorcheffectivelyi pins pytorch to 2.5.1, whereas newer versions of deepspeed rely on newer ops available in 2.7.1The pixi build already fetches pytorch from conda-forge. Pinning deepspeed for now as we plan to deprecate it completely
Changes
setup_openfold --non-interactiveto streamline parameter downloadRelated Issues
This issue caused a failure on the CI tests for #393 and will likely break future conda CI tests.
Testing
Other Notes