[NOGIL] Restrict sharing of Consumer instances across threads - #2322
Open
Ojasva Jain (ojasvajain) wants to merge 2 commits into
Open
[NOGIL] Restrict sharing of Consumer instances across threads#2322Ojasva Jain (ojasvajain) wants to merge 2 commits into
Ojasva Jain (ojasvajain) wants to merge 2 commits into
Conversation
…s protection Introduce a unified gate_owner/gate_depth mechanism in Consumer.c that rejects concurrent cross-caller access to a single Consumer instance while still allowing legitimate re-entrant calls (e.g. a rebalance/commit callback calling back into the Consumer that triggered it). The gate is shared between the sync Consumer (thread ID as identity) and AIOConsumer (a Python-generated logical-caller identity, since the owning caller may move across ThreadPoolExecutor worker threads). Gate enforcement (CFL_CONSUMER_GATE_ENABLED in confluent_kafka.h) only applies to Python versions we've never shipped a wheel for -- 3.15+, or 3.14 built free-threaded -- so existing users on <=3.14 GIL-based Python see zero behavior change on upgrade. Also fixes Consumer__pause_internal/Consumer__resume_internal, which never checked self->rk before calling into librdkafka, causing pause()/resume() after close() to segfault instead of raising RuntimeError.
Ojasva Jain (ojasvajain)
requested review from
a team and
Matthew Seal (MSeal)
as code owners
August 6, 2026 13:33
|
🎉 All Contributor License Agreements have been signed. Ready to merge. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Introduces a reentrancy gate for the sync Consumer and AIOConsumer that rejects illegal concurrent access to a single Consumer instance from a different caller, while still allowing legitimate re-entrant calls — e.g. a rebalance or commit callback calling back into the Consumer/AIOConsumer that triggered it (on_assign calling assign(), on_commit calling commit(), etc.).
or 3.14 built free-threaded. On every version already shipped (≤3.14, GIL-based), the gate is compiled out entirely, so existing users see zero behavior change on upgrade.
TODO -> Add unit test cases for AIOConsumer