Consider an opt-in ABI3T wheel for Python 3.15+

[honglei]

Consider an opt-in ABI3T wheel for Python 3.15+

Python 3.15 and free-threaded Python 3.15 wheels are now covered by #619.
Separately, would you be interested in an opt-in Stable ABI for Free-Threaded
Builds (abi3t) artifact?

PEP 803 is now final. An ABI3T extension
can use one wheel per platform for both GIL-enabled and free-threaded CPython
3.15+, but the Stable ABI can be slower than the version-specific API. For
aiscat, I think the version-specific cp315t wheel should remain the preferred
high-throughput artifact; ABI3T would be an additional portability artifact,
not a replacement.

Prototype evidence

I have an unsubmitted, opt-in prototype based on current main. On Linux
x86_64 with CPython 3.15.0rc1 it:

• builds an extension named _core.abi3t.so in a wheel tagged
  cp315-abi3.abi3t;
• imports and passes the existing decoder tests on both GIL-enabled and
  free-threaded CPython 3.15;
• keeps the GIL disabled on the free-threaded interpreter;
• passes a multithreaded decoder stress test;
• exports PyModExport__core and declares Py_MOD_GIL_NOT_USED;
• compiles with Py_TARGET_ABI3T, with opaque object layout and public
  Stable-ABI calls in the ABI3T path;
• has no imported CPython symbols outside the CPython 3.15 Stable ABI
  manifest.

The normal version-specific source path and build remain unchanged.

Preliminary performance

I also replayed 5,000,000 real AIS input lines on one 64-logical-CPU Linux
host. File reading, multipart grouping, and payload allocation were excluded
from the timed interval. Each variant decoded the same 4,963,558 records with
the same message-type distribution.

Wheel / interpreter	1 worker	4 workers
cp315t / CPython 3.15.0rc1	0.919 M lines/s	3.178 M lines/s
ABI3T / CPython 3.15.0rc1	0.822 M lines/s	2.716 M lines/s
cp314t / CPython 3.14.7	0.713 M lines/s	2.912 M lines/s

Against cp315t on the same interpreter, the prototype was about 10.6% slower
with one worker and 14.5% slower with four workers. It still retained useful
parallel scaling. These are preliminary relative results from one host, not a
production capacity claim; I would repeat controlled runs before proposing a
PR.

Proposed boundary if this is wanted

A separate PR would:

• keep normal cp315/cp315t builds as the default and preserve their fast
  CPython-specific paths;
• make ABI3T an explicit opt-in build;
• limit changes to the Python binding, build configuration, tests, and ABI3T
  documentation;
• validate wheel metadata, extension naming, imported symbols, GIL state,
  functional equivalence, and single-/multi-thread performance;
• defer release-workflow changes until the ABI3T packaging path is reliable.

Known remaining gates are an actual Windows wheel/import test and an
ABI3T-aware binary audit. abi3audit currently cannot identify the new
PyModExport_* entry point; this is tracked in
pypa/abi3audit#215.

Would this additional portability artifact be useful to AIS-catcher, or would
you prefer to keep shipping the version-specific free-threaded wheels for now?

[jvde-github]

Thanks for the thorough prototype and the honest benchmarks — much appreciated, and the proposed boundary looks right.

My hesitation isn't the wheels, it's the source. The speed here comes from private APIs — _PyDict_NewPresized, GC untracking, PyObject_HEAD — so the ABI3T variant necessarily diverges exactly inside convert_object / convert_value / wrap_annotated, which is the hot loop I most expect to keep optimising. Worth noting too that ABI3T wouldn't replace anything for a long time: requires-python is >=3.9, so every version-specific wheel stays regardless, and this would be a second path alongside them rather than a simplification.

How did that look in your prototype — did the two paths separate cleanly, or does the conversion loop effectively fork? And how do you see the maintenance working over time: would you expect the fast path to stay the reference implementation with ABI3T following it, or something more unified? Your view on that would help me decide. A draft PR would be welcome if it's easy to share.