Skip to content

feat(python): build free-threaded wheels for CPython 3.14t and 3.15t - #174

Merged
d-v-b merged 4 commits into
mainfrom
free-threaded-wheels
Oct 31, 2026
Merged

d-v-b merged 4 commits into
mainfrom
free-threaded-wheels

Conversation

@d-v-b

@d-v-b d-v-b commented Oct 31, 2026 •

Copy link
Copy Markdown
Owner

🤖 AI text below 🤖

refkv now installs and runs on free-threaded CPython: each platform's release build adds a cp314-cp314t wheel and a cp315-abi3.abi3t wheel (PEP 803's stable ABI, which covers 3.15t and later) beside the abi3 one, and importing refkv._core leaves the GIL disabled. A range's CRC-32C is now computed in the Rust core, so google-crc32c is no longer a dependency.

What changed

  • rust/refkv-python/src/lib.rs: #[pymodule(gil_used = false)]. The agent's audit of shared state found:
    • The test-only setters (_set_copy_threshold, _set_chunk_frames, _set_shard_frames) write AtomicUsizes that each run reads once, into its facts. They stay process-wide, as they already were with the GIL.
    • Nd2Parser, TiffParser, CziParser, Run and StoreRun take &mut self. If a second thread calls one while another call on it is running, it raises RuntimeError: Already borrowed. On 3.14t, 4 threads calling TiffParser.feed raised it 15 times out of 20, and the GIL stayed disabled. This could already happen with the GIL, because those calls detach while the core works. python/README.md now documents it.
    • Ir is read-only apart from its type cache, which sits behind a Mutex. Ir.value used to hold that lock while it built Python objects. On a free-threaded build, a second thread blocked on the lock while attached can hold up a stop-the-world GC pause that the lock holder is waiting for. The lock is now released before to_py.
  • CRC-32C: google-crc32c has no free-threaded (or musllinux) wheels, and its sdist falls back to pure Python. On 3.14t that fallback took 102 ms per MiB, against 0.42 ms for a native implementation, about 240× slower. Rather than switch to the crc32c PyPI package (LGPL-2.1+), the agent added _core.crc32c, built on the Rust crc32c crate that rust/refkv-phkv already uses (MIT/Apache, hardware accelerated, runs detached). refkv.policy.crc32c calls it. zarr 3 still depends on google-crc32c, so the zarr extra still pulls it in, but refkv's own checksums no longer go through it.
  • release.yml: on Linux the abi3 build now names --interpreter 3.12. With the abi3t feature on, maturin in a container otherwise builds one wheel per interpreter and none of them abi3. A second maturin step with --interpreter 3.14t 3.15t per platform. setup-python installs 3.12, 3.14t and 3.15t, and each wheel is installed in its own venv on its own Python and runs check_base_install.py base. That check now also asserts the GIL is still disabled at the end on a free-threaded Python. The free-threaded musllinux wheels are built but not run, since no official Alpine image has a free-threaded Python. The publish step's name check (refkv-$v[-.]*) matches the new names unchanged.
  • ci.yml: a new pytest (ubuntu-latest, 3.14t) job. It runs uv sync --python 3.14t with --no-install-package for the packages that have no cp314t wheels: eccodeslib, eckitlib, astropy, pylibczirw and jpype1. The GRIB comparisons already skip without ecCodes. The two astropy FITS test files are left out. The job also asserts the GIL stays disabled with refkv._core, refkv.zarr, imagecodecs and av imported.
  • imagecodecs on 3.14t: the locked imagecodecs 2026.8.16 is abi3-only. The first release with cp314t and cp315t wheels is 2026.10.10, but its JPEG XR encoder writes different bytes, so 17 generated fixtures stopped reproducing. The lock therefore keeps 2026.8.16, and the 3.14t job installs 2026.10.10 over the locked environment.
  • test_store_prefetch.py: test_bytes_held_ahead_are_bounded failed about 1 run in 6 on 3.14t. take() frees the budget before the test decremented its counter, so a read could start in between and the test overcounted. The test now takes the object and decrements in one step under its lock. The race was in the test, not in _Prefetch. After the fix it passed 25 runs out of 25.
  • Docs: python/RELEASING.md has a table of the three wheels per platform. python/README.md lists the free-threaded wheels, the extras' coverage and the borrow behavior.

Extras on free-threaded Python

Checked on macOS arm64 with --only-binary :all: (except google-crc32c, allowed to build from its sdist):

extra 3.14t 3.15t
base ✅ ✅
zarr ✅ ❌ numcodecs has no cp315t wheel (builds from source)
images ✅ (imagecodecs ≥ 2026.10.10) ❌ via zarr
media ✅ ❌ PyAV has no cp315t wheel
obstore ✅ ✅
virtualizarr ✅ ❌ numcodecs

google-crc32c comes in through zarr and installs from its sdist as pure Python, with a RuntimeWarning.

Local verification

  • Wheels built with maturin for 3.12 (abi3), 3.14t (cp314t), and 3.15t and 3.15 (both cp315 abi3t). check_base_install.py base passed on 3.14t and 3.15t, and zarr passed on 3.14t. pip 26.2 and uv both install the abi3t wheel.
  • Full python/tests on 3.14t: 3072 passed, 58 skipped. The two failures were the prefetch race (fixed) and one test that needs the wasm build, which CI builds.
  • just lint, just typecheck and just sections pass. On the GIL build, just test gave 3244 passed and 1 failed. The failure is test_every_output_declares_its_profile_where_the_convention_says, which runs node on js/ and needs npm ci and just js::wasm. CI runs both, and this worktree had not.

Not verified locally

The manylinux, musllinux and Windows free-threaded builds rely on maturin-action's containers and on setup-python providing python3.14t and python3.15t on each runner (including windows-11-arm). This PR's CI is the first run. PyPI's acceptance of the abi3.abi3t tag set is also untested until a tag publishes.

Stack bookkeeping

Based on main at af8f923. It does not block the v0.2.0 release PR (#171) and can merge before or after it.

🤖 Generated with Claude Code

d-v-b and others added 4 commits October 31, 2026 13:01
The module is declared with `gil_used = false`, so importing it on a
free-threaded CPython leaves the GIL disabled; check_base_install.py
checks that. The audit found no state two threads could corrupt: the
test-only setters write atomics that each run reads once, and a parser or
run raises "Already borrowed" when a second thread calls it during a call.
Ir.value now releases its type cache's lock before it makes Python
objects, so a thread waiting on the lock cannot stall a stop-the-world
pause.

A range's CRC-32C is now computed by the core (the crc32c crate, which
rust/refkv-phkv uses too), and google-crc32c is no longer a dependency: it
has no free-threaded wheels, and its pure-Python fallback is about 240
times slower. pyo3's abi3t-py315 feature lets a 3.15t build make a stable
ABI wheel (PEP 803). The lock takes imagecodecs 2026.10.10, its first
release with free-threaded wheels.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each platform now builds, beside the abi3 wheel, a cp314t wheel and a
cp315 abi3t wheel, which covers 3.15t and later, and checks that each
installs and runs on its Python. The free-threaded musllinux wheels are
built but not run, as no official Alpine image has a free-threaded Python.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Linux job runs the tests on 3.14t, in the test environment minus the
packages that have no free-threaded wheels. The prefetch budget test now
takes an object and uncounts it in one step: take() frees the budget
first, so a read could start in between and the test overcounted, which
3.14t showed.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…decs

With pyo3's abi3t feature on, maturin in a Linux container built a wheel
per interpreter and no abi3 one; the abi3 build there now names 3.12. The
lock goes back to imagecodecs 2026.8.16, whose JPEG XR encoder the fixture
generators reproduce; the 3.14t job installs 2026.10.10, its first release
with free-threaded wheels, over the locked environment.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@d-v-b
d-v-b merged commit 1f5b7d1 into main Oct 31, 2026
33 checks passed
@d-v-b
d-v-b deleted the free-threaded-wheels branch October 31, 2026 13:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant