Skip to content

fix: stop leaking client supervisor children on failed and closed connections#554

Open
mvanhorn wants to merge 2 commits into
elixir-grpc:masterfrom
mvanhorn:fix/531-grpc-client-supervisor-memory-leak
Open

fix: stop leaking client supervisor children on failed and closed connections#554
mvanhorn wants to merge 2 commits into
elixir-grpc:masterfrom
mvanhorn:fix/531-grpc-client-supervisor-memory-leak

Conversation

@mvanhorn

Copy link
Copy Markdown
Contributor

Summary

Short-lived client connections no longer leak GRPC.Client.Supervisor children. Orchestrator processes that fail to finalize a connection are terminated instead of being left under the DynamicSupervisor, and disconnected connections are fully cleaned up, so the supervisor's binary memory stays flat under connect/disconnect churn.

Why this matters

Reported in #531: a user running hundreds of short-lived connections (GRPC.Stub.connect/2 then GRPC.Stub.disconnect/1) on 1.0.0-rc.1 sees GRPC.Client.Supervisor top :recon.proc_count(:memory, 10) and :recon.bin_leak(10) with persistent growth; downgrading to 0.10.2 (no DynamicSupervisor in the client path) eliminates it, and a collaborator confirmed the diagnosis and labeled it a bug. The regression is in the 0.11+/1.0 client connection architecture: GRPC.Client.Connection.connect/2 leaves the orchestrator child running when finalize_connection/2 returns {:error, :no_connection}, and the disconnect path leaves per-connection state behind.

Changes

  • connect/2 now terminates the orchestrator child when connection finalization fails, so error paths cannot accumulate supervised processes.
  • The disconnect path releases the connection's supervised resources instead of leaving the child alive under GRPC.Client.Supervisor.

Testing

Added regression tests that drive repeated connect/disconnect cycles and connect-failure cycles, asserting DynamicSupervisor.count_children/1 on GRPC.Client.Supervisor returns to baseline after each cycle. Ran the client connection test files with mix test.

Fixes #531

@sleipnir

Copy link
Copy Markdown
Collaborator

@mvanhorn Some tests are failing, could you please check them?

The connection tests asserted DynamicSupervisor active == 0 on the shared GRPC.Client.Supervisor, which fails under the full suite where other tests leave children behind (CI showed active: 44). Capture the pre-existing count in setup and assert the count returns to that baseline.
@mvanhorn

Copy link
Copy Markdown
Contributor Author

The OTP-matrix failures were the connection tests asserting active == 0 on the shared GRPC.Client.Supervisor — under the full suite other tests leave children behind (CI showed active: 44), so the assertions failed even though the lifecycle fix works. 5a6f752 captures the supervisor's pre-existing child count in setup and asserts the count returns to that baseline instead. All 14 connection tests pass locally.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GRPC.Client.Supervisor persistent memory growth

2 participants