NCBC-4280: Keep a usable node when one KV connection fails to connect

[davidkelly]

Motivation

The Capella private-endpoint situational suite failed its scored ramp
window with tens of thousands of errors per scenario, every one of them
"Cluster has not yet bootstrapped".

For a window after an AWS PrivateLink endpoint reports available, its
data path is lossy rather than down. Two pool connections open to the
same address; one completes TCP, TLS and Helo while its sibling never
gets a SYN through and dies at KvConnectTimeout. AddConnectionsAsync
keeps each connection that succeeds, but the Task.WhenAll it ends with
rethrows if any failed, so InitializeAsync threw and the caller
discarded the whole node along with its authenticated, working
connection. With no nodes IsBootstrapped stayed false and every
operation in the 28-second ramp failed.

Two further defects hid the cause. Bootstrapper.Execute cleared
DeferredExceptions unconditionally after BootStrapAsync returned, but
Cluster.BootStrapAsync records most failures there rather than throwing,
so the clear erased the failure that had just happened and, because
IsBootstrapped derives from that list, both stopped the retry loop and
left the reported exception carrying no inner exceptions. That same
method also described any failure as ignorable on 6.5 or earlier, which
excuses a global-config failure and nothing else.

Modification

Tolerate a partial pool when at least one connection came up, and let
the scale controller backfill the rest as it already does when a
connection drops. An empty pool is still a failure. Both pools are
fixed, though ChannelConnectionPool is the one in use because
Experiments.ChannelConnectionPools defaults to true. That pool also now
rejects a pool left empty by connections returning IsDead, which
AddConnectionsAsync drops without throwing; DataFlowConnectionPool
tolerates the same case deliberately and recovers in SendAsync.

This puts .NET on the behaviour Java and Go already had. Both treat one
connected connection as a usable node, and both default to one KV
connection per node where .NET uses two, which is why .NET was the only
one of the three exposed to a sibling connection failing at all.

Bootstrapper now discards only the failures present before an attempt,
removing them by identity because subjects clear the list themselves on
success. Cluster.BootStrapAsync warns on entering the failed state and
drops to Debug while it persists, rather than warning on every 2.5s
poll, and ConnectAsync warns when it returns an unbootstrapped cluster.
The address a KV connection is made to is now logged at Debug; on .NET
Core and later it was never recorded anywhere, and that gap is what made
the original analysis inconclusive.

Results

op-capella-pe-sit-lite against server 8.0 passed 5 of 5 scenarios with
zero failures and zero errors, against a baseline of 3 of 5. An earlier
run of the same suite hit the partial-pool path in every scenario, 12
times in total, each pool backfilling to full within about 30 seconds
once the degraded window passed.

op-onprem-func-release on 8.0 and 7.6 exercised the new paths about 1270
times per leg without either warning ever firing, so healthy clusters
are unaffected.

Each fix has a unit test confirmed to fail against the previous
behaviour.

[Copilot]

Pull request overview

This PR addresses cold-start bootstrap failures in lossy-network windows (notably AWS PrivateLink) by ensuring nodes aren’t discarded when at least one KV connection successfully initializes, and by improving bootstrap failure retention and diagnostics so failures are visible and retries continue appropriately.

Changes:

• Allow connection pools to complete initialization with a partial pool when at least one connection succeeds; correct initialization-size logging.
• Fix bootstrap retry behavior by preserving deferred bootstrap failures across attempts and adjusting cluster bootstrap logging to be actionable without becoming noisy.
• Add unit tests covering partial-pool behavior (both pool implementations) and bootstrap deferred-failure retention.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/Couchbase.UnitTests/Core/IO/Connections/DataFlow/DataFlowConnectionPoolTests.cs	Adds tests ensuring DataFlow pool tolerates partial success but throws on total failure.
tests/Couchbase.UnitTests/Core/IO/Connections/Channels/ChannelConnectionPoolTests.cs	Adds tests ensuring Channel pool tolerates partial success and rejects empty/dead-only pools.
tests/Couchbase.UnitTests/Core/BootstrapperTests.cs	Adds coverage to ensure deferred bootstrap failures are not cleared and retries continue.
src/Couchbase/Core/IO/Connections/DataFlow/DataFlowConnectionPool.cs	Swallows failures during initial connection creation when Size > 0; logs partial/actual pool size.
src/Couchbase/Core/IO/Connections/ConnectionFactory.cs	Adds redactor null-check; logs resolved address prior to connecting for better diagnostics.
src/Couchbase/Core/IO/Connections/Channels/ChannelConnectionPool.cs	Swallows failures during initial connection creation when Size > 0; rejects empty pool; logs partial/actual pool size.
src/Couchbase/Core/Bootstrapping/Bootstrapper.cs	Preserves deferred failures recorded during a bootstrap attempt; only removes pre-existing failures after the attempt.
src/Couchbase/Cluster.cs	Adds bootstrap failure state tracking for warning noise control; warns on ConnectAsync returning unbootstrapped cluster; improves deferred bootstrap logging.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.