Repository navigation
feat(python): build free-threaded wheels for CPython 3.14t and 3.15t - #174
Merged
Merged
Conversation
The module is declared with `gil_used = false`, so importing it on a free-threaded CPython leaves the GIL disabled; check_base_install.py checks that. The audit found no state two threads could corrupt: the test-only setters write atomics that each run reads once, and a parser or run raises "Already borrowed" when a second thread calls it during a call. Ir.value now releases its type cache's lock before it makes Python objects, so a thread waiting on the lock cannot stall a stop-the-world pause. A range's CRC-32C is now computed by the core (the crc32c crate, which rust/refkv-phkv uses too), and google-crc32c is no longer a dependency: it has no free-threaded wheels, and its pure-Python fallback is about 240 times slower. pyo3's abi3t-py315 feature lets a 3.15t build make a stable ABI wheel (PEP 803). The lock takes imagecodecs 2026.10.10, its first release with free-threaded wheels. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each platform now builds, beside the abi3 wheel, a cp314t wheel and a cp315 abi3t wheel, which covers 3.15t and later, and checks that each installs and runs on its Python. The free-threaded musllinux wheels are built but not run, as no official Alpine image has a free-threaded Python. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Linux job runs the tests on 3.14t, in the test environment minus the packages that have no free-threaded wheels. The prefetch budget test now takes an object and uncounts it in one step: take() frees the budget first, so a read could start in between and the test overcounted, which 3.14t showed. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…decs With pyo3's abi3t feature on, maturin in a Linux container built a wheel per interpreter and no abi3 one; the abi3 build there now names 3.12. The lock goes back to imagecodecs 2026.8.16, whose JPEG XR encoder the fixture generators reproduce; the 3.14t job installs 2026.10.10, its first release with free-threaded wheels, over the locked environment. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
refkv now installs and runs on free-threaded CPython: each platform's release build adds a
cp314-cp314twheel and acp315-abi3.abi3twheel (PEP 803's stable ABI, which covers 3.15t and later) beside the abi3 one, and importingrefkv._coreleaves the GIL disabled. A range's CRC-32C is now computed in the Rust core, sogoogle-crc32cis no longer a dependency.What changed
rust/refkv-python/src/lib.rs:#[pymodule(gil_used = false)]. The agent's audit of shared state found:_set_copy_threshold,_set_chunk_frames,_set_shard_frames) writeAtomicUsizes that each run reads once, into its facts. They stay process-wide, as they already were with the GIL.Nd2Parser,TiffParser,CziParser,RunandStoreRuntake&mut self. If a second thread calls one while another call on it is running, it raisesRuntimeError: Already borrowed. On 3.14t, 4 threads callingTiffParser.feedraised it 15 times out of 20, and the GIL stayed disabled. This could already happen with the GIL, because those calls detach while the core works.python/README.mdnow documents it.Iris read-only apart from its type cache, which sits behind aMutex.Ir.valueused to hold that lock while it built Python objects. On a free-threaded build, a second thread blocked on the lock while attached can hold up a stop-the-world GC pause that the lock holder is waiting for. The lock is now released beforeto_py.google-crc32chas no free-threaded (or musllinux) wheels, and its sdist falls back to pure Python. On 3.14t that fallback took 102 ms per MiB, against 0.42 ms for a native implementation, about 240× slower. Rather than switch to thecrc32cPyPI package (LGPL-2.1+), the agent added_core.crc32c, built on the Rustcrc32ccrate thatrust/refkv-phkvalready uses (MIT/Apache, hardware accelerated, runs detached).refkv.policy.crc32ccalls it.zarr3 still depends ongoogle-crc32c, so thezarrextra still pulls it in, but refkv's own checksums no longer go through it.release.yml: on Linux the abi3 build now names--interpreter 3.12. With the abi3t feature on, maturin in a container otherwise builds one wheel per interpreter and none of them abi3. A second maturin step with--interpreter 3.14t 3.15tper platform. setup-python installs 3.12, 3.14t and 3.15t, and each wheel is installed in its own venv on its own Python and runscheck_base_install.py base. That check now also asserts the GIL is still disabled at the end on a free-threaded Python. The free-threaded musllinux wheels are built but not run, since no official Alpine image has a free-threaded Python. The publish step's name check (refkv-$v[-.]*) matches the new names unchanged.ci.yml: a newpytest (ubuntu-latest, 3.14t)job. It runsuv sync --python 3.14twith--no-install-packagefor the packages that have no cp314t wheels:eccodeslib,eckitlib,astropy,pylibczirwandjpype1. The GRIB comparisons already skip without ecCodes. The two astropy FITS test files are left out. The job also asserts the GIL stays disabled withrefkv._core,refkv.zarr,imagecodecsandavimported.test_store_prefetch.py:test_bytes_held_ahead_are_boundedfailed about 1 run in 6 on 3.14t.take()frees the budget before the test decremented its counter, so a read could start in between and the test overcounted. The test now takes the object and decrements in one step under its lock. The race was in the test, not in_Prefetch. After the fix it passed 25 runs out of 25.python/RELEASING.mdhas a table of the three wheels per platform.python/README.mdlists the free-threaded wheels, the extras' coverage and the borrow behavior.Extras on free-threaded Python
Checked on macOS arm64 with
--only-binary :all:(except google-crc32c, allowed to build from its sdist):zarrimageszarrmediaobstorevirtualizarrgoogle-crc32ccomes in through zarr and installs from its sdist as pure Python, with aRuntimeWarning.Local verification
check_base_install.py basepassed on 3.14t and 3.15t, andzarrpassed on 3.14t. pip 26.2 and uv both install the abi3t wheel.python/testson 3.14t: 3072 passed, 58 skipped. The two failures were the prefetch race (fixed) and one test that needs the wasm build, which CI builds.just lint,just typecheckandjust sectionspass. On the GIL build,just testgave 3244 passed and 1 failed. The failure istest_every_output_declares_its_profile_where_the_convention_says, which runs node onjs/and needsnpm ciandjust js::wasm. CI runs both, and this worktree had not.Not verified locally
The manylinux, musllinux and Windows free-threaded builds rely on maturin-action's containers and on setup-python providing
python3.14tandpython3.15ton each runner (includingwindows-11-arm). This PR's CI is the first run. PyPI's acceptance of theabi3.abi3ttag set is also untested until a tag publishes.Stack bookkeeping
Based on
mainat af8f923. It does not block the v0.2.0 release PR (#171) and can merge before or after it.🤖 Generated with Claude Code