Add SQPOLL support (on by default when available) + deferred submission batching - #71
Merged
Conversation
…ters SQPOLL (linux_uring): - linux_uring.c probes at import time whether IORING_SETUP_SQPOLL is actually usable (kernel + capabilities) and exposes the result as SQPOLL_ALLOWED - Context(sqpoll=True) already fell back gracefully on its own either way, this is purely so callers can pick a default without constructing a throwaway Context. - linux_uring_asyncio.AsyncioContext now defaults sqpoll to SQPOLL_ALLOWED instead of False (an explicit sqpoll= kwarg still wins). - Fixed a real, reproducible race this surfaced: under SQPOLL, the kernel thread dequeues SQEs on its own schedule, not synchronously within submit()/flush() - a slot the asyncio semaphore just released (because its op's completion was observed) can still show as occupied from the ring's own head/tail accounting for a brief window. Requests one spare ring entry from the kernel unconditionally in AIOContext_init (self->max_requests, the caller-visible property, stays exactly what was asked for) - confirmed via a 1000-iteration stress repro that this eliminates the "io_uring SQ ring full" OverflowError previously reproducible in roughly 1-in-25 runs at max_requests=2. - linux_uring_asyncio.py's _create_context now accepts **kwargs and forwards them (it silently dropped sqpoll= before this). Deferred submission batching (linux_aio, linux_uring): - New deferred=False constructor kwarg on AsyncioContextBase. When True, linux_aio/linux_uring's adapters batch multiple submissions queued within the same event-loop iteration into one syscall (io_submit()/io_uring_enter()) via loop.call_soon(), instead of flushing/submitting eagerly after every single op - nothing between submit()'s isinstance check and the submit call itself ever awaits, so concurrent submits previously always paid one syscall each with no batching despite N being ready at once. - Measured (hot page-cache workload, concurrency=64): +71% throughput for linux_uring, +3.7% for linux_aio - linux_uring's flush() is a full io_uring_enter() syscall per call where linux_aio's io_submit() is already a single tight syscall, so there's much less to batch away. Real-disk cold-cache workload: -7%/-3% respectively (deferring adds one event-loop round-trip that isn't recovered when ops are already I/O-latency-bound, not syscall-count-bound). - deferred is a no-op whenever linux_uring actually negotiated SQPOLL: flush() there only calls io_uring_enter() at all when the kernel thread has gone idle, so eager per-op flush() is already close to syscall-free - measured no benefit batching a cost that's mostly already gone. - linux_aio's deferred path had its own real bug, fixed before landing: _deferred_submit runs as a bare call_soon callback, not inside any pending coroutine - a synchronous exception from context.submit(*ops) (e.g. EBADF on a closed fd) used to just get logged by asyncio's default handler and orphan every future in the batch forever. Now caught and routed to every pending future in the batch (io_submit() rejects a batch wholesale on this class of error, confirmed via linux_aio.c's AIOContext_submit rollback logic - "the whole batch failed together" is accurate here, not just a prefix). Test coverage: - tests/conftest.py builds test-only variants (functools.partial over the real backends, no subclassing) forcing sqpoll/deferred on, added to the existing cross-backend fixture parametrization - so the ordinary suite exercises these code paths too instead of only the untested-by-default state. sqpoll+deferred together isn't tested given the no-op finding above. - New async_context fixture: a ready, already-entered AsyncioContext for the common case (no custom constructor kwargs needed) - migrated the asyncio adapter tests that don't need one off async_context_maker (kept for the one test needing a non-default max_requests). - Two subprocess-spawning tests in test_raw_low_level.py now skip cleanly for synthetic (non-caio.*-importable) backend variants, rather than generating an import statement that can't work in a fresh subprocess. 238 passed / 0 failed / 9 skipped on the Linux VM (5 consecutive full- suite runs, plus 1000 iterations of a dedicated ring-overflow stress repro with zero failures), 75 passed / 0 failed on macOS.
- Removed benchmark/bench_go.go (the Go goroutine baseline) - no longer maintained alongside the 4 caio backends. - bench_runner.py's CSV merge step now validates every per-backend CSV against a single expected column schema before merging, rejecting a mismatched file with a clear message instead of silently producing a column-shifted bench_all.csv. - plot_results.py updates to match. - Track benchmark/*.png via Git LFS (.gitattributes) instead of committing chart images as regular blobs, and add the corresponding .gitignore entry for local results/ output directories.
linux_uring[deferred=True]'s test variant let SQPOLL_ALLOWED's smart default silently take over, so it never actually exercised the deferred-flush call_soon path it was named for - force sqpoll=False. Add a cancel-before-submit test to cover context.cancel() on an op the backend never reached. Auto-retry the one known-flaky timing test via pytest-rerunfailures.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
linux_uring.cprobesIORING_SETUP_SQPOLLusability once at import time (linux_uring.SQPOLL_ALLOWED);linux_uring_asyncio.AsyncioContextdefaults itssqpollkwarg to that instead ofFalse. A dedicated kernel thread polls the SQ ring instead of everysubmit()/flush()needing anio_uring_enter()syscall. An explicitsqpoll=kwarg still wins either way, and unsupported kernels/permissions still fall back to a plain ring exactly as before (pre-existingflag_table_sqpollbehavior, unchanged).deferred=FalseonAsyncioContextBase, implemented forlinux_aio/linux_uring. N concurrently-scheduled submits in the same event-loop iteration used to each pay their own syscall;deferred=Truedefers the actual submit/flush to the next_run_once()pass vialoop.call_soon()so one syscall covers the whole ready batch. Automatically a no-op once SQPOLL is actually negotiated onlinux_uring(itsflush()is already close to syscall-free there — measured no benefit combining the two).Bugs found along the way
master, just never triggered until this PR flipped the default:linux_uring.c'sio_uring_setup()sized the ring to exactlymax_requests, with no headroom for the window between a completion being observed and the SQPOLL kernel thread's own ring bookkeeping catching up. That gap has always existed for any caller passingsqpoll=Trueexplicitly — nothing onmasterdid, so it never fired. Surfaced here as an intermittentOverflowError: io_uring SQ ring full(~1-in-25 runs oftest_max_requests_backpressure). Fixed with one spare ring slot at theio_uring_setup()call site only (self->max_requests + 1) — the public.max_requestsproperty still reports exactly what was requested. Verified with a dedicated 1000-iteration stress repro: 0 failures after, reliably failing within ~30 before.masterbug):linux_aio's new deferred-submit path runs as a barecall_sooncallback. A synchronous exception fromsubmit()(e.g. a bad fd) used to just get logged by asyncio's default exception handler, leaving every future in that batch unresolved forever. Fixed by catching it and routing it to every pending future in the batch.Benchmark
Linux VM,
max_requests-only construction (no explicit kwargs — what an existing caller gets automatically after upgrading), 5 runs each, before =master/ after = this branch:linux_uringpulling ahead oflinux_aioat moderate concurrency shows up in the fuller sweep too:Test coverage
tests/conftest.pyaddssqpoll/deferred-forced test-only variants oflinux_uring/linux_aio(viafunctools.partial, no subclassing) into the existing cross-backend parametrization, so the whole suite exercises these paths, not just dedicated tests.sqpoll+deferredtogether is deliberately not generated (no-op, per above).Test plan
uv run pytest --cov=caio --cov-report=term-missing -sv tests— green on Linux VM (238 passed / 0 failed / 9 skipped, 5x) and macOS (75 passed)mypy/ruffclean on both platforms.max_requestsconfirmed unchanged (padding not leaked to callers)