Skip to content

[NOGIL] Restrict sharing of Consumer instances across threads - #2322

Open
Ojasva Jain (ojasvajain) wants to merge 2 commits into
dev_producer_no_gilfrom
dev_consumer_no_gil
Open

[NOGIL] Restrict sharing of Consumer instances across threads#2322
Ojasva Jain (ojasvajain) wants to merge 2 commits into
dev_producer_no_gilfrom
dev_consumer_no_gil

Conversation

@ojasvajain

@ojasvajain Ojasva Jain (ojasvajain) commented Aug 6, 2026

Copy link
Copy Markdown
Member

Introduces a reentrancy gate for the sync Consumer and AIOConsumer that rejects illegal concurrent access to a single Consumer instance from a different caller, while still allowing legitimate re-entrant calls — e.g. a rebalance or commit callback calling back into the Consumer/AIOConsumer that triggered it (on_assign calling assign(), on_commit calling commit(), etc.).

  • A unified gate_owner/gate_depth mechanism (Handle_gate_enter()/Handle_gate_exit() in Consumer.c) is shared between both consumer types: the sync Consumer presents its own OS thread ID as the gate identity; AIOConsumer presents a Python-generated logical-caller identity instead, since the "owning" caller may move across ThreadPoolExecutor worker threads between the initial call and any re-entrant calls made from within a callback it triggers.
  • Every gated Consumer method (assign, subscribe, poll, consume, commit, close, etc.) was refactored into an _internal entry point that accepts the caller's identity explicitly, plus a thin public wrapper that supplies its own thread ID. AIOConsumer calls the same _internal methods directly with its generated identity, so both consumer types are gated identically in C.
  • Backward compatibility: gate enforcement is controlled by CFL_CONSUMER_GATE_ENABLED (confluent_kafka.h) and only applies to Python versions this project has never shipped a wheel for — 3.15+ (any build),
    or 3.14 built free-threaded. On every version already shipped (≤3.14, GIL-based), the gate is compiled out entirely, so existing users see zero behavior change on upgrade.

TODO -> Add unit test cases for AIOConsumer

…s protection

Introduce a unified gate_owner/gate_depth mechanism in Consumer.c that
rejects concurrent cross-caller access to a single Consumer instance
while still allowing legitimate re-entrant calls (e.g. a rebalance/commit
callback calling back into the Consumer that triggered it). The gate is
shared between the sync Consumer (thread ID as identity) and AIOConsumer
(a Python-generated logical-caller identity, since the owning caller may
move across ThreadPoolExecutor worker threads).

Gate enforcement (CFL_CONSUMER_GATE_ENABLED in confluent_kafka.h) only
applies to Python versions we've never shipped a wheel for -- 3.15+, or
3.14 built free-threaded -- so existing users on <=3.14 GIL-based Python
see zero behavior change on upgrade.

Also fixes Consumer__pause_internal/Consumer__resume_internal, which
never checked self->rk before calling into librdkafka, causing
pause()/resume() after close() to segfault instead of raising
RuntimeError.
@confluent-cla-assistant

Copy link
Copy Markdown

🎉 All Contributor License Agreements have been signed. Ready to merge.
Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant