fix: stop leaking client supervisor children on failed and closed connections#554
Open
mvanhorn wants to merge 2 commits into
Open
fix: stop leaking client supervisor children on failed and closed connections#554mvanhorn wants to merge 2 commits into
mvanhorn wants to merge 2 commits into
Conversation
Collaborator
|
@mvanhorn Some tests are failing, could you please check them? |
The connection tests asserted DynamicSupervisor active == 0 on the shared GRPC.Client.Supervisor, which fails under the full suite where other tests leave children behind (CI showed active: 44). Capture the pre-existing count in setup and assert the count returns to that baseline.
Contributor
Author
|
The OTP-matrix failures were the connection tests asserting |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Short-lived client connections no longer leak
GRPC.Client.Supervisorchildren. Orchestrator processes that fail to finalize a connection are terminated instead of being left under the DynamicSupervisor, and disconnected connections are fully cleaned up, so the supervisor's binary memory stays flat under connect/disconnect churn.Why this matters
Reported in #531: a user running hundreds of short-lived connections (
GRPC.Stub.connect/2thenGRPC.Stub.disconnect/1) on 1.0.0-rc.1 seesGRPC.Client.Supervisortop:recon.proc_count(:memory, 10)and:recon.bin_leak(10)with persistent growth; downgrading to 0.10.2 (no DynamicSupervisor in the client path) eliminates it, and a collaborator confirmed the diagnosis and labeled it a bug. The regression is in the 0.11+/1.0 client connection architecture:GRPC.Client.Connection.connect/2leaves the orchestrator child running whenfinalize_connection/2returns{:error, :no_connection}, and the disconnect path leaves per-connection state behind.Changes
connect/2now terminates the orchestrator child when connection finalization fails, so error paths cannot accumulate supervised processes.GRPC.Client.Supervisor.Testing
Added regression tests that drive repeated connect/disconnect cycles and connect-failure cycles, asserting
DynamicSupervisor.count_children/1onGRPC.Client.Supervisorreturns to baseline after each cycle. Ran the client connection test files with mix test.Fixes #531