packs: restore all environments on Retread 4.10.17 by garylvov · Pull Request #28 · garylvov/pixidock_template

garylvov · 2026-07-14T23:21:32Z

Summary

Restores the four Retread source packs and all 15 declared environments on an exact pixi-build-retread ==4.10.10 backend:

genesis-pack
isaac-pack (Isaac Sim 6.0 + IsaacLab v3 beta)
isaac-pack-latest
newton-pack-latest

The four target-qualified schema-12 solve JSONs are committed, contain no local path leakage, and are pinned to the exact Retread version so a backend upgrade cannot replay a stale record.

Cold-solve fix

The original Isaac cold build spent most of its time repeatedly minimizing an unsatisfiable set of conda routes: 195 Resolvo calls over roughly 120k–132k records. Retread 4.10.10 identifies the three singleton conflicts directly (lxml, mujoco-usd-converter, and urdf-usd-converter), reducing the same build to 11 total Resolvo calls across its phases.

Both Isaac manifests also route flatdict through conda before UV closure. PyPI's flatdict 4.0.1 is sdist-only and imports undeclared pkg_resources; retread-conda-deps avoids that clean-checkout failure up front.

Measured performance

Path	Result
Original Isaac, old Retread 4.10.9 cold build	42m05s
Original Isaac, first full Retread 4.10.10 cold build	3m14s
Original Isaac, backend-cold with Pixi metadata warm	1m25s
All 15 environments, fresh root lock replay	4m20s
Existing-tree root lock verification	1m14s

The 4m20s replay was measured from commit fac6d0d in a detached worktree with pixi.lock and .pixi absent. Shared package/download caches were retained, so this measures dependency solving and committed-record replay rather than redownloading every artifact.

Validation

pixi lock --check: pass
environment count: 15/15
four solve JSONs: schema 12, Retread 4.10.10
fresh-root replay: pass, with no checksum or flatdict failure
regenerated record semantics: unchanged apart from Retread version and version-pinned input hashes
generated .conda files remain untracked

Scope boundary

Isaac and ROS 2 environments are restored independently. Combined Isaac+ROS environments remain intentionally absent because the upstream stacks require incompatible TBB ABIs; this is not a Retread solve failure.

Restores the retread pack work reverted in #27 (revert of #23, c8da1bb) and moves it onto the now-proven backend. v4.6.3 is validated end-to-end: it locks the 25-env sibling imprint workspace green with zero hand-written workaround pins, so the retreat that motivated the revert no longer applies. Restored from c8da1bb (deliberately, not a blanket checkout): - pypi-packs as top-level source packs: genesis-pack, isaac-pack, isaac-pack-latest, newton-pack-latest (each a pixi-build-retread [package]) - root pixi.toml pack/feature wiring: python 3.12, preview=pixi-build, isaac/newton/genesis features consume the packs via path deps; Humble+Jazzy ROS 2 split (humble_ws/ + jazzy_ws/ with synchros2 submodules); the isaacsim/colcon activation scripts - dropped the old source-build path the packs replace: IsaacLab / IsaacLabNewton / patches/egl_probe / ros2_ws / ros_utilities submodules and their egl_probe CMake patch step - regenerated pixi.lock (did NOT restore c8da1bb's stale 4.4.0-era lock or the per-pack retread-*.lock.json courier locks) Backend bump: - every pypi-packs/*/pixi.toml pinned EXACTLY "==4.6.3" (was ">=4.4.0"), plus the prefix.dev/pixi-build-backends channel. Pinned exact on purpose: 4.6.4-4.6.7 are published but unproven and 4.6.4 changes the PyPI-index path / shifts the closure, so a floating >= would silently pick an unproven backend on the next cold solve. - root [pypi-options].extra-index-urls now lists "https://pypi.org/simple" first: the backend's un-route fallback only searches DECLARED indexes, so without the default index transitive PyPI-only deps 404 against the private indexes. Hand-written workaround pins: NONE to delete. Unlike imprint (which shed 126 lines of retread-drop-deps / retread-keep-pypi / torch-trio retread-overrides / pyglet widen), these four packs never carried those tables, and 4.6.3 derives the whole closure without them. The revert had instead pushed the crutches up into the workspace: pre-pack main hand-wrote 32 exact "==" conda pins in [feature.isaaclab.dependencies] (to appease isaacsim-kernel) and 14 pypi-options.dependency-overrides (torch==2.7.1, triton, starlette, isaacsim-kernel, pillow, scipy, warp-lang, mujoco, ...). This restore drops all of them to zero and the closure still solves. Solve result: 13/13 active declared environments + `default` lock GREEN under a freshly-installed 4.6.3 backend (probe traces for genesis-pack / isaac-pack-latest / newton-pack-latest all report retread_version 4.6.3; the cache held no other backend version). Packs land in the lock as content-hashed conda_source entries. Lock-skip (1 pack, 3 envs) — [lock-skip:retread-4.6.3-gitclone-self-deadlock]: isaaclab-gpu, isaaclab-ros2-humble-gpu, isaaclab-ros2-jazzy-gpu consume isaac-pack (IsaacLab v3.0.0-beta @ a4a7602). Building that pack DETERMINISTICALLY deadlocks the 4.6.3 backend: the backend takes an exclusive flock WRITE on its git-clone lock file (retread-git-clones/isaac-sim_IsaacLab.git/<rev>.lock) then blocks its own worker threads waiting on the same inode via separate fds — a self-deadlock (verified in /proc/locks: holder pid == every waiter pid; wchan locks_lock_inode_wait; CPU frozen). Reproduced on two independent cold solves, once warm-clone and once freshly-purged-clone, so it is a backend concurrency bug, not a workspace dependency conflict and not clone corruption. isaac-pack-latest (same repo, rev 28a37ce) builds cleanly under the same 4.6.3, so isaaclab-gpu-latest is unaffected. The isaac-pack files + [feature.isaaclab] wiring are kept intact (and pinned ==4.6.3) so the envs re-enable once the backend git-clone lock bug is fixed. Also: gitignore .pixi/ (pixi's local env/build-metadata cache, never committed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Pin Pixi and Retread to validated releases and commit target-qualified solve locks so fresh checkouts replay expensive pack metadata.

Pin records to 4.10.10 so backend upgrades cannot replay stale output. Route flatdict through conda before UV inspects its undeclared build dependencies.

garylvov · 2026-07-17T05:06:00Z

Performance/validation update after fac6d0d:

Published and adopted Retread 4.10.10's singleton-conflict fast path.
Regenerated and pushed all four target-qualified solve JSONs.
Fixed clean-checkout flatdict handling by excluding it from UV closure and routing it through conda.
Pinned each solve record to the exact backend version.
Reduced original Isaac cold generation from 42m05s to 3m14s (or 1m25s with Pixi metadata warm).
Completed a real fresh-root replay of all 15 environments in 4m20s with shared artifact caches retained; pixi lock --check passed afterward.

The old bottleneck was CPU-bound repeated Resolvo work, not downloads: 195 solver calls in the original Isaac build. The optimized path uses 11 calls across both metadata/build phases and directly rejects the three singleton-unsatisfiable routes.

Remaining work is CI only. The new pre-commit and test jobs are currently queued; there is no known solve blocker. Combined Isaac+ROS environments remain outside this PR because of the upstream TBB ABI conflict noted in the PR description.

garylvov · 2026-07-17T13:33:19Z

Retread 4.10.17 update:

Bumped all four source packs from exact ==4.10.10 to ==4.10.17.
Regenerated and pushed all four target-qualified schema-12 solve JSONs.
Cold record generation completed in 7m53.30s total with two pack workers.
A detached checkout with fresh submodules and no .pixi passed pixi lock in 3m30.70s without rewriting pixi.lock.
The exact CI command, pixi install --locked, then passed in 1m15.61s.

The previous red jobs both stopped at “lock file not up-to-date with the workspace.” That clean-checkout failure is no longer reproducible on 4.10.17. Generated .conda artifacts remain untracked.

Use Retread's built-in OpenCV and PyTorch mappings, refresh every solve record at 4.10.21, and replace the stale IsaacLab lxml metadata cap.

garylvov · 2026-07-17T20:50:51Z

Update pushed in e9b9797:

removed the pack-local opencv-python / opencv-python-headless / torch maps; Retread/Parselmouth now owns the common py-opencv / pytorch translations
added only the IsaacLab-specific lxml >=5.2.2,<7 override needed to replace NVIDIA usd-to-urdf's stale lxml<5 metadata cap
refreshed all four committed Retread solve JSONs at 4.10.21 and updated pixi.lock

Validation/timing on Slurm:

full repository relock: 4m59s
direct warm pack builds: Genesis 2m14s, Isaac 4m30s, Isaac-latest 2m25s, Newton 52s
all four direct builds and the root relock passed

Remaining: hosted PR checks plus a final isolated locked-install pass. Local .conda outputs were intentionally not committed.

Cold locked installation passes in 6m28s. Avoid unlocked source-package rebuilds that reopen unrelated upstream dependency drift.

Use Pixi's frozen install so hosted runner platform facts cannot invalidate and rebuild the already verified source-package records.

Prevent pixi run from re-entering a workspace solve and requiring unrelated CUDA environments on CPU runners.

garylvov · 2026-07-17T21:14:53Z

Final CI update:

Removed redundant per-pack mappings for common aliases (opencv-python / opencv-python-headless -> py-opencv, torch -> pytorch); Retread/Parselmouth now owns those centrally.
Kept only project-specific overrides (for example the Isaac lxml compatibility bound).
Refreshed and committed all four solve JSONs.
Verified an isolated empty-cache locked install at commit e9b9797: 6m28s, with no lockfile changes.
Updated setup and task execution to stay frozen on the committed lock, preventing CPU CI from re-solving unrelated CUDA environments.

Current head e3fe87e is green: test 22s, pre-commit 41s.

…, ABI bare-major normalize, anchor exact-pin band) Drop stale relaxation lock records; bump retread-build-number. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…quires an effective glibc per workspace target) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…lab manylinux ceiling) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

… clone; one clone per parallel install) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…lel conflict localization, shared repodata) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…patch race workaround) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Move the MuJoCo/MJX walls into exact Retread packs, split by the existing NumPy 2.5, Isaac 2.3.1, and Newton 3.8 closure families. Keep PyTorch3D feature-owned because Retread 4.10.45 rejects its sole linux_x86_64 wheel tag. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Backend bumped from 4.10.66. 4.10.70/.71 rank feature-declared channels above inherited workspace channels and solve each environment's transitive constraints against its own channels rather than the per-output union. That is what genesis-ros2-jazzy-gpu needed: its extraction previously rejected every robostack candidate under strict channel priority ('ros2-distro-mutex 0.14.0 is excluded'), which was silently swallowed until extraction became a hard error. The relock resolves ros2-distro-mutex from robostack-jazzy, matching what the real solve always did. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

garylvov and others added 2 commits July 14, 2026 19:20

fix: restore all Retread environments

6418792

Pin Pixi and Retread to validated releases and commit target-qualified solve locks so fresh checkouts replay expensive pack metadata.

garylvov changed the title ~~packs: restore pixi-build-retread pack work on backend 4.6.3~~ packs: restore all environments on Retread 4.10.9 Jul 17, 2026

perf(packs): accelerate cold solves

fac6d0d

Pin records to 4.10.10 so backend upgrades cannot replay stale output. Route flatdict through conda before UV inspects its undeclared build dependencies.

garylvov changed the title ~~packs: restore all environments on Retread 4.10.9~~ packs: restore all environments on Retread 4.10.10 Jul 17, 2026

build(packs): bump Retread to 4.10.17

4fe6c25

garylvov changed the title ~~packs: restore all environments on Retread 4.10.10~~ packs: restore all environments on Retread 4.10.17 Jul 17, 2026

garylvov added 2 commits July 17, 2026 16:21

fix(ci): tolerate cold backend bootstrap

3bbe803

fix(packs): centralize common name mapping

e9b9797

Use Retread's built-in OpenCV and PyTorch mappings, refresh every solve record at 4.10.21, and replace the stale IsaacLab lxml metadata cap.

garylvov added 3 commits July 17, 2026 17:00

fix(ci): consume committed pack locks

a0ae35e

Cold locked installation passes in 6m28s. Avoid unlocked source-package rebuilds that reopen unrelated upstream dependency drift.

fix(ci): freeze committed pack locks

b728eda

Use Pixi's frozen install so hosted runner platform facts cannot invalidate and rebuild the already verified source-package records.

fix(ci): freeze task execution

e3fe87e

Prevent pixi run from re-entering a workspace solve and requiring unrelated CUDA environments on CPU runners.

garylvov and others added 14 commits July 28, 2026 15:32

chore: bump pixi-build-retread to 4.10.42 (auto-relax: psutil unroute…

8c9e41b

…, ABI bare-major normalize, anchor exact-pin band) Drop stale relaxation lock records; bump retread-build-number. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

fix: declare glibc on linux-64 platform envelopes (retread 4.10.42 re…

6d3818c

…quires an effective glibc per workspace target) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

fix: platform envelopes declare glibc 2.35 (container runtime, pymesh…

e39bca6

…lab manylinux ceiling) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

chore: bump pixi-build-retread to 4.10.45

535f799

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

docs: pixi build-dispatch race workaround (concurrency.solves 1 after…

2acf4d3

… clone; one clone per parallel install) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

chore: bump pixi-build-retread to 4.10.46 (probe perf: bisected+paral…

ca937da

…lel conflict localization, shared repodata) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

chore(scripts): serialize pixi solve concurrency per clone (build-dis…

5e692b8

…patch race workaround) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

chore: roll back to retread 4.10.45 (4.10.46 backend crash)

7d98d2a

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

chore: relock on retread 4.10.45 (full env matrix green)

9cd0c2a

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

chore: commit workspace pixi config (serialized solves)

f737337

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

chore: bump to retread 4.10.47

57fe11e

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

chore: bump to retread 4.10.48 (hermetic native builds)

82f478b

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Navigation Menu

packs: restore all environments on Retread 4.10.17 - #28

packs: restore all environments on Retread 4.10.17#28
garylvov wants to merge 23 commits into
mainfrom
restore-packs-retread-4.6

garylvov commented Jul 14, 2026 •

edited

Loading

Uh oh!

garylvov commented Jul 17, 2026

Uh oh!

garylvov commented Jul 17, 2026

Uh oh!

garylvov commented Jul 17, 2026

Uh oh!

garylvov commented Jul 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

garylvov commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Cold-solve fix

Measured performance

Validation

Scope boundary

Uh oh!

garylvov commented Jul 17, 2026

Uh oh!

garylvov commented Jul 17, 2026

Uh oh!

garylvov commented Jul 17, 2026

Uh oh!

garylvov commented Jul 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

garylvov commented Jul 14, 2026 •

edited

Loading