fix(streams): transaction probe must not lease an AR connection - #395
Conversation
Stream#current_open_transaction probed via ActiveRecord::Base.connection, which on Rails 7.2+ takes a sticky, executor-scoped lease. Two leaks: - On a non-executor thread (the Coalescer's flush thread, app worker threads) the lease is never released — one AR pool connection pinned per thread for its lifetime. - Inside connection_pool.with_connection the sticky flag defeats the block-exit release, so the CALLER's connection leaks when its thread dies (pool reports it dead-but-in-use). Both variants only fire on the durable path — ephemeral broadcasts early-return before the probe — which is why flipping streams_default_broadcast_mode to :durable surfaced it: a downstream fan-out spec with an exactly-sized pool exhausted deterministically (5 workers + 1 coalescer flush = whole pool dead in one run). Probe connection_pool.active_connection? instead: the existing lease or nil, never a checkout. Semantics unchanged — a transaction is per-lease, so a thread holding no connection has no open transaction to defer on; the old fresh checkout always answered nil anyway, at the price of the leak.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository UI (base), Organization UI (inherited) Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
📝 WalkthroughWalkthrough
ChangesActiveRecord transaction probing
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
Suggested labels: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
Summary
Stream#current_open_transactionprobes for an open transaction viaActiveRecord::Base.connection. On Rails 7.2+ that call takes a sticky, executor-scoped lease, which leaks pool connections in two ways:Concurrent::ScheduledTaskflush thread, any app worker thread not wrapped by the Rails executor): the lease is never released — one AR pool connection pinned per thread for its lifetime.connection_pool.with_connection: the sticky flag defeats the block-exit release, so the caller's connection leaks when its thread dies. The pool then reports it dead-but-in-use (stat→dead: N), and those slots are gone.Both variants only fire on the durable path — ephemeral broadcasts
return broadcast_ephemeral(wrapped) unless use_durablebefore the probe. That's why flippingstreams_default_broadcast_mode = :durablesurfaced it downstream: Zazu's bulk-payment fan-out spec sizes its worker threads to the exact AR pool, and one run left the pool at{connections: 6, busy: 1, dead: 5, waiting: 6}— 5 dead leases from the workers'with_connectionblocks plus one live lease pinned to the coalescer flush thread. DeterministicActiveRecord::ConnectionTimeoutErroron every run.Production impact isn't limited to specs: every coalesced/durable broadcast flushed on the coalescer thread pins an AR connection in web/worker processes.
Fix
Probe
connection_pool.active_connection?instead — returns the calling thread/fiber's existing lease or nil, never a checkout. Semantics are unchanged: a transaction is per-lease, so a thread holding no connection has no open transaction to defer on. The old code's fresh checkout always answerednilin that situation anyway — at the price of the leak.Tests
spec/pgbus/streams/transaction_probe_spec.rb(real AR via the dummy app):active_connection?returns the leaked lease)with_connectiondoesn't defeat its release (red on main: dead-owner in-use connection)Existing AR stubs in
streams_spec.rbupdated to the newconnection_pool.active_connection?contract. Full streams suite: 383 green. (spec/rubocop/cop/pgbus/no_ruby_timeout_spec.rbfails to load on a clean main checkout locally — pre-existing, unrelated.)Summary by CodeRabbit
Bug Fixes
Tests