Skip to content

fix(worker-image): resolve the nodetool-huggingface version after the PyPI wait - #1014

Merged
georgi merged 2 commits into
mainfrom
fix/worker-image-hf-version-race
Aug 24, 2026
Merged

fix(worker-image): resolve the nodetool-huggingface version after the PyPI wait#1014
georgi merged 2 commits into
mainfrom
fix/worker-image-hf-version-race

Conversation

@georgi

@georgi georgi commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

The race

docker.yml resolves HF_VERSION in the meta step — the first step of the job — as "latest nodetool-huggingface on PyPI". The "Wait for nodetool-core on PyPI" step runs after it.

Both repos publish on a tag push. Releasing them together (nodetool-core v0.7.10 and nodetool-huggingface v0.7.8, pushed minutes apart today) starts two wheel builds in parallel, and the image job reads "latest" before either has finished. The result is an image tagged :0.7.10 / :latest that bundles the previous nodetool-huggingface.

That is not hypothetical for this release: v0.7.8 carries the nunchaku Qwen rope shim and the FLUX variant routing fix, and an image built from 0.7.7 has neither.

Worse, it was invisible — the version only appeared in a build log line, nothing in the image recorded it.

The fix

  • Move the lookup into its own step, placed after the core-on-PyPI wait. That wait already polls for up to 30 minutes and normally costs a minute or more, by which time the sibling repo's wheel has landed.
  • Record both versions as OCI labels:
ai.nodetool.core-version
ai.nodetool.huggingface-version

so a pulled image can be checked with docker inspect instead of a rebuild.

workflow_dispatch with an explicit hf_version is unchanged and still wins.

Found while releasing core 0.7.10 and huggingface 0.7.8 together.

georgi added 2 commits August 24, 2026 17:07
… PyPI wait

Both repos publish on a tag push, so releasing nodetool-core and
nodetool-huggingface together starts two wheel builds in parallel. The
`meta` step read "latest nodetool-huggingface" at job start — before
either wheel existed — so the image pinned the *previous* release and
shipped without the fixes cut alongside it. Silently: nothing in the
build or the image recorded which version went in.

Move the lookup into its own step after "Wait for nodetool-core on
PyPI", which already blocks long enough for the sibling wheel to land,
and record both versions as OCI labels so a pulled image can be checked
without a rebuild.
The wait polled /pypi/<pkg>/<version>/json while uv installs from
/simple. They propagate independently: on the 0.7.10 release the JSON
endpoint answered 200 and the wait passed, then the build died with
"no version of nodetool-core==0.7.10" because /simple did not list it
yet. Poll the surface the installer actually reads, and confirm the
resolved nodetool-huggingface version is listed there too.
@georgi

georgi commented Aug 24, 2026

Copy link
Copy Markdown
Contributor Author

Both failure modes are now observed, not predicted. The v0.7.10 tag build (run 32742412809) failed, and its log shows them together:

RUN uv pip install "nodetool-core==0.7.10" "nodetool-huggingface==0.7.7"
  × No solution found when resolving dependencies:
  ╰─▶ Because there is no version of nodetool-core==0.7.10 …

1. nodetool-huggingface==0.7.7 — the race this PR opened for. nodetool-huggingface 0.7.8 was published 12 minutes later, so the image would have shipped without the nunchaku Qwen rope shim (#58) or the FLUX variant routing fix (#59).

2. nodetool-core==0.7.10 not found — a second race, now also fixed here. "Wait for nodetool-core on PyPI" passed, then the install failed on the same version. The wait polled pypi.org/pypi/<pkg>/<version>/json; uv installs from pypi.org/simple. Those are separate surfaces with independent propagation — /pypi/nodetool-core/0.7.10/json answered 200 while /simple/nodetool-core/ still did not list the wheel.

So the wait now polls the simple index, and the resolved nodetool-huggingface version is confirmed there too.

Checked against the live index, including that the check can fail and that the trailing dash anchors the version:

core 0.7.10 -> FOUND     hf 0.7.8 -> FOUND
core 0.7.9  -> FOUND     hf 0.7.7 -> FOUND
core 9.9.9  -> not listed   hf 9.9.9 -> not listed

$ echo 'nodetool_core-0.7.10-py3-none-any.whl' | grep -c "nodetool_core-0.7.1-"
0

Net effect: the 0.7.10 release produced no worker image at all. I have re-dispatched the build manually with version=0.7.10 hf_version=0.7.8.

@georgi
georgi merged commit 7bbbb42 into main Aug 24, 2026
5 checks passed
georgi added a commit that referenced this pull request Aug 24, 2026
… index (#1017)

The worker image build resolved "latest" from /pypi/<pkg>/json. That is a
different surface from /simple, with independent propagation, and it lags:
minutes after 0.7.9 was listed on /simple, info.version still answered 0.7.8.

The confirmation loop added in #1014 could not catch this. It verifies that
the chosen version is on /simple — and 0.7.8 is. So the step passed, and the
image built right after the 0.7.9 release carried the previous wheel while
looking correct. Found by reading the image labels rather than trusting the
green build.

Discovery now reads /simple too, so both halves use the surface uv installs
from. Sorting is numeric rather than lexical, which the JSON API never needed:
info.version was authoritative, but max() over parsed filenames would put
0.7.9 ahead of 0.7.10. Stdlib only — no packaging import to fail in the
release path.

Verified against the live index: picks 0.7.9, exits non-zero on an index with
no wheels, and orders 0.7.10 above 0.7.9.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant