Skip to content

Namespace-join open_stream timeout leaves the dial running in the background (uncancelled) #3111

Description

@chefsale

Summary

When a namespace-join stream open times out, the in-flight libp2p dial/stream-open is not cancelled — it keeps running in the background on the NetworkManager arbiter until it resolves on its own. This is a low-severity inefficiency (the orphan self-cleans, see below), split out from a batch of async/concurrency hardening fixes so the rest could ship.

Where

  • crates/node/src/sync/manager/namespace_join.rs:181time::timeout(open_timeout, sync_network.open_stream(*peer))

The same pattern applies to any caller that wraps NetworkClient::open_stream in a timeout.

Trace / root cause

NetworkClient::open_stream (crates/network/primitives/src/client.rs:154) sends NetworkMessage::OpenStream { request, outcome: tx } to the NetworkManager actor, then awaits rx. The message is dispatched through forward_handler (crates/utils/actix/src/adapters.rs:127), which drives the handler's ResponseFuture (crates/network/src/handlers/commands/open_stream.rs) on the actor independently of the caller's rx.

So when the caller's open_stream future is dropped on timeout:

  • Only the caller-side rx is dropped.
  • The actor's spawned ResponseFuture keeps running stream_control.open_stream(peer_id, CALIMERO_STREAM_PROTOCOL).await to completion — the dial is not cancelled and is bounded only by libp2p's own dial resolution.

Why it's low severity (not a leak)

When the background dial finally resolves, the opened Stream is constructed and then immediately dropped, because the tx.send(...) fails (the receiver is gone). Dropping the Stream closes it. So the orphan self-cleans on completion — it is a lingering background future, not a permanent resource leak.

Options considered

  1. Handler-side dial timeout — wrap stream_control.open_stream(...) in a tokio::time::timeout(...) inside the OpenStream handler so a hung dial self-cancels at the network layer. Contained, but: (a) affects all open_stream callers, and (b) picking a value is largely redundant with libp2p's configured dial timeout and would be a guess.
  2. Propagate caller cancellation — have the handler race the dial against outcome.closed() (tokio oneshot Sender::closed()) and abort the dial when the caller drops rx. This is the "correct" fix but needs custom actix response plumbing (the current ResponseFuture path doesn't expose the sender's closed signal to the handler).

Recommendation

Defer unless we see this actually biting (e.g. background dials piling up on the NetworkManager arbiter under churn). If we do act, prefer option 2 (propagate cancellation) over a guessed timeout. Worth measuring libp2p's effective dial timeout first before adding a second bound.

Context

Sibling async-hardening fixes landed on branch fix/async-hardening. This item was intentionally left out pending a decision.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions