WIP: Harden Connection Management to Prevent Zombie Connections and Indefinite Blocking

[lucksus]

In the process of diagnosing this flaky test holochain/holochain#4174, I've experimented with these changes in hope to avoid and/or mitigate loss of connection in some cases.

I'm posting this PR for visibility - stopping work on this for now in favor of the iroh transport implementation.

Summary

This PR tries to fix critical connection management issues that caused threads to block indefinitely on failed connections and allowed zombie connections to remain in the peer map, leading to cascading timeout failures. It also fixes a channel capacity bug that caused "closed" errors under concurrent load.

Problem

The existing implementation had several critical issues:

1. Indefinite Blocking: wait_for_ready() would block indefinitely on a semaphore when waiting for peer connections, causing threads to hang forever when connections failed
2. Zombie Connection Reuse: Failed connections remained in peer_map and were reused by subsequent sends, causing repeated 20-second timeouts
3. Channel Saturation: Event channel was hardcoded to 32 slots despite config specifying 1024, causing channel saturation and "closed" errors under concurrent load
4. No Failed State Detection: No mechanism to proactively detect and clean up failed peer connections before reuse attempts

Changes

1. Add Timeout Parameter to wait_for_ready() (715648d)

• Wrap semaphore acquire in tokio::time::timeout to prevent indefinite blocking
• Add timeout parameter to MaybeReady::wait_for_ready() and Peer::wait_for_ready()
• Update all call sites to pass config.timeout
• Prevents threads from blocking indefinitely on failed connections

2. Add Failed Connection Detection (2a92be6)

• Add MaybeReady::is_failed() to check MaybeReadyState::Failed
• Add Peer::is_failed() wrapper method
• Enables proactive cleanup of zombie connections before reuse attempts

3. Fix Channel Capacity Bug (2b2dc39)

Critical fix: Endpoint event channel was hardcoded to 32 despite config specifying 1024

• ep.rs: Use config.internal_event_channel_size (32→1024, 32x increase)
• hub.rs: Increase hub command channel from 32→256 (8x increase)
• framed.rs: Increase cmd and msg channels from 32→256 (8x increase)
• Prevents channel saturation during concurrent connection establishment and high message throughput

4. Add Aggressive Peer Cleanup (0c9d81f)

Prevents failed connections from remaining in peer_map and being reused, which would cause repeated timeouts

• Add stale peer check in connect_peer() - remove failed peers before reuse
• Add early cleanup in peer task() when connection fails during negotiation
• Add cleanup in send() when wait_for_ready() times out
• Add debug logging for peer lifecycle events

Impact

• Prevents indefinite thread blocking on failed connections
• Eliminates zombie connection reuse that caused cascading timeout failures
• Fixes channel saturation under concurrent load (32x increase for main event channel)
• Enables proactive failed connection detection and cleanup
• Improves system reliability during high-concurrency scenarios and network failures

Testing

While this changed the signature of flaky test failures, it didn't (yet?) result in 100% stable connections.

[cocogitto-bot]

✔️ 715648d...0c9d81f - Conventional commits check succeeded.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

✨ Finishing touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[EthosengineBot]

Even though this is paused in favor of iroh — flagging that conductors still pinned to tx5 (Holochain 0.6.x via kitsune2 0.3.x) hit the zombie-connection problem in production as an off-heap OOM, not just test flakiness.

We root-caused our prod leak to the Evt::State(_) => () gap in go_pion.rs (dead-peer PeerConnections never freed → monotonic Go-runtime mmap growth → OOM), and confirmed #194's teardown fix by building it: the conn_dropped_on_peer_connection_state_* tests pass with #194/#199 and time out without. The broader peer_map cleanup + wait_for_ready timeout + the 32-vs-1024 channel-capacity fix in this PR are complementary on the same surface. If a minimal #194/#199 tx5 patch release is feasible before iroh fully lands, it would unblock everyone on 0.6.x. Thanks for documenting the mechanics here — it matched our independent trace closely.

[EthosengineBot]

Correction to my earlier comment (#issuecomment-4732920114). I said we'd root-caused our prod OOM to the Evt::State(_) => () gap in go_pion.rs (dead-peer PeerConnections never freed). That root-cause attribution was wrong and I want to set it straight here.

We deployed #194 + #199 fleet-wide, binary-verified, and the off-heap OOM persisted unchanged — so the zombie-PeerConnection path was not what was driving our leak. Reading per-VMA smaps showed the growing anon living in glibc malloc secondary arenas (the 0x77xx mmap region) serving the conductor's native Rust/C allocations; the Go heap (0xc0…) stayed flat ~52 MB. My "flat Rust heap ⇒ Go" inference was the mistake — glibc arenas are anon mmap classified other (never the brk [heap]), and Rust-on-Linux uses glibc malloc by default, so off-heap anon there is normal. Switching the conductor's global allocator to jemalloc (with unprefixed-malloc interposition) flattened it for many hours past the prior OOM cadence — pointing to reclaimable glibc-pinned retention, an allocator-fit issue, rather than a never-freed code leak in tx5.

None of this diminishes the work in this PR. The peer_map cleanup, wait_for_ready timeout, and the 32-vs-1024 channel-capacity fix are real connection-management hardening, and #194/#199 are a worthwhile liveness fix — we're keeping them downstream. I just can't honestly attach a memory-OOM justification to them anymore, so I'll leave prioritization to you. Apologies for the noise on the original attribution.