TCP M: connection pool — concurrent operations over one client

[alex-clickhouse]

Stacked on #559 (tcp/epic-n9-poco-write) — review that one first; this PR's diff is the pool only.

TCP epic M: a real connection pool. ConnectionPool replaces the interim SingleConnectionSource behind the same IConnectionSource seam, so the client changed by one line — but the behaviour it exposes is new: a shared client now runs operations concurrently instead of serializing them onto one connection.

await using var client = new ClickHouseTcpClient("Host=localhost;MaxPoolSize=20");

// 50 queries, 20 connections. The rest queue.
await Task.WhenAll(queries.Select(q => client.ExecuteAsync(q)));

Knobs

On ClickHouseTcpClientOptions and the connection-string builder. TimeSpan.Zero disables the two limits. SweepInterval is the one knob whose absence means something rather than nothing.

	Default
MaxPoolSize	20	Cap on connections, and so on concurrent operations
MinPoolSize	0	Connections kept open when possible, in use as well as idle
PoolTimeout	30s	How long a caller waits for a free connection
MaxConnectionLifetime	30min	How long a connection may be reused after opening
IdleTimeout	5min	How long a connection may sit unused before it is retired
SweepInterval	derived	How often the pool looks for connections to retire, and holds the floor
PoolReusePolicy	Lifo	Which idle connection is handed out next

Decisions worth a look

Validation is always on, client-side, with no knob. The design sketch had a ValidateOnBorrow switch, but a switch to turn off correctness is only a way to get corruption. ClickHouseTcpConnection.IsReusable is Ready + nothing in our read buffer + a non-blocking Socket.Poll(SelectRead); both buffers have to be checked, since the poll sees the kernel's and BufferedBytes sees ours. A readable idle socket means either the peer closed or bytes are waiting that the last operation did not consume — neither allows reuse. No Ping option: a round-trip per checkout is the wrong default and the wrong opt-in.

Age and idleness are the same kind of limit. Both are read at checkout, on return, and by the sweep, through one predicate, and both override MinPoolSize.

The sweep period is derived, with an override. A quarter of the shortest limit in force, held between 1 and 30 seconds, so the delay in noticing an expiry stays proportional to the limit being enforced — one fixed period cannot do that for limits that range from seconds to hours. At the defaults that is 30s, so a flat 5s would be six times the wake-ups for nothing, while a 2s IdleTimeout would be enforced three and a half times late. SweepInterval overrides it for a workload the derivation does not suit, and is used unclamped: silently changing an explicit value is the failure mode the knob exists to avoid. Zero is invalid rather than meaning "no sweep" — null is how to go back to deriving — because zero alongside a MinPoolSize would be a floor nothing ever fills, the sweep being the only thing that tops it up. And nothing-to-sweep still wins over the override: with both limits off and no floor, no timer is created at all, which is what keeps an undisposed client collectable.

Idleness started as a resource limit only — swept, honouring the floor, never barring a checkout — on the reasoning that IdleTimeout exists to release sockets nobody is using, not to judge whether one works. That reasoning misses what actually happens to an idle connection: a proxy or load balancer between client and server drops it on its own schedule, and such a drop can arrive without a FIN, in which case IsReusable still says yes and the operation sent over it stalls until TCP gives up — on Linux, about fifteen minutes. So idleness is a liveness limit as much as a resource one, and the docs now say to set it below the shortest idle timeout on the path to the server.

Two consequences worth spelling out.

Neither limit yields to MinPoolSize, and that is what makes the floor useful. Reaping only to the floor while a checkout refuses an expired connection is the worst of both: the pool holds MinPoolSize sockets nobody can use, and the next burst still pays connect and handshake. Reaping everything expired and letting the top-up re-dial makes it a floor of fresh connections. The cost is MinPoolSize dials per IdleTimeout on a pool carrying no traffic at all — with the defaults, none, the floor being 0, but the two knobs multiply, so a floor of 10 against a 5-second idle limit is 10 connects and handshakes every 5 seconds from an application issuing no queries. The MinPoolSize doc now says so with that example, and a test pins the rotation over two sweeps. Npgsql and HikariCP both stop at the floor instead, and HikariCP then needs a separate keepaliveTime to validate what it kept; one limit does both here.

The idle clock has to stop while a connection is out. It used to keep running, which was harmless when only the sweep read it — the sweep walks the idle set, so the value was meaningless for a leased connection either way. Once the return path reads it, that stops being harmless: a query longer than IdleTimeout would report its own duration as idleness and retire the connection that had just run it successfully. With the 5-minute default, every query over five minutes. PooledConnection now tracks whether it is out and reports zero idleness while it is.

That also simplified the sweep to one indexed walk over the idle set, which removed the two-pass structure behind the idle[0] bug below rather than only fixing it.

MinPoolSize is held, not just respected. The sweep tops the pool back up to the floor after reaping, so it does not decay — without that, nothing refilled it and after MaxConnectionLifetime of inactivity the pool was empty whatever the floor said. It counts open connections, in use as well as idle: the name says pool size, and counting only spare ones would over-provision a pool under steady light load. The top-up takes a permit per connection exactly as a checkout does, which is what keeps the floor inside MaxPoolSize, and takes it with a zero timeout so it never queues ahead of a real caller. One runs at a time; a failed dial ends the round quietly and the next sweep retries; disposal cancels a dial in flight. What is still M10 is the eager half — filling the floor at construction, so the very first burst does not pay connect and handshake.

No jitter on connection lifetime. Connections opened in one burst age out together, so they are reaped together and re-dialled together; across a fleet deployed at once, every instance rotates at the same time. Npgsql, Go's database/sql and clickhouse-go all accept this and HikariCP is the outlier that jitters. Matching the majority, and recorded in the TODO as a choice rather than an oversight — worth revisiting if reconnect spikes show up against an expensive auth path, where the herd actually costs something.

No retry counter for M9. The checkout loop discards unusable idle candidates until one passes or the set is empty, then dials once. A failure to dial that connection is reported rather than retried, so a server that is down is reported as down instead of being hidden behind PoolTimeout.

Waiters are not ordered, and the pool says so. SemaphoreSlim makes no promise and a fair queue is not worth building here: two operations that overlap on different connections have no ordering at the server either.

The cap, and what a permit actually bounds

Review found the pool could open more connections than MaxPoolSize, twice, by the same mistake — worth spelling out because the reasoning is not obvious.

Backpressure is a SemaphoreSlim of MaxPoolSize permits, and it is tempting to read that as the cap. It is not. A permit is held by a checkout, not by a connection: an idle connection sits in the pool holding none. So permits bound the concurrent operations, and something else has to bound the sockets.

That something is a count, and it was reading low in two places where a connection belongs to no set:

1. A dial in flight. With min=max=3, a top-up finishes one connection and reassesses while two checkouts are still dialing. It sees 1 against a floor of 3 and opens two more: five sockets, cap of three.
2. A reused connection between the two sets. A checkout lifts its connection out of idle before it records it in leased. With one leased and two idle, a checkout taking one leaves the count reading 2, and a top-up dials for a fourth.

Both are now covered by a slot reserved under the same lock that decides a connection may be opened, and given back under the same lock that files it, so the count never dips in between. The slot is held for the whole checkout rather than around the dial, which means a checkout that has not yet decided whether to reuse or dial is counted before it holds anything — the figure is an upper bound, not a census. That is the safe direction: a top-up declines to dial rather than overshooting.

Both are pinned by tests that fail on the parent commit. The first holds every dial open by hand so the bad interleaving is the one that runs; the second needs a seam inside a checkout, and the clock is one — the checkout reads it to test the connection's age, between the two sets — so ControlledTimeProvider grew a one-shot hook and the test sweeps at exactly that moment.

Worth noting the earlier stress test — 400 checkouts against a cap of 3 with 200 sweeps racing them — passes on both broken versions. A scripted dial finishes too fast to overlap reliably, so it only samples the bad interleaving by luck. Gating the dials is what turns it from a test that might stumble into the bug to one that always runs it.

The sweep, and a swallow that hid a bug

The trim used to be a second pass, whose floor test counts the connections that are out, so it passes with nothing idle whenever more are leased than the floor asks for — and it then read idle[0]. With the default floor of zero that is any sweep at all while one connection is checked out, which is the commonest state a sweep can land in.

On the timer SweepQuietly swallowed the ArgumentOutOfRangeException, so the sweep silently stopped trimming and stopped holding the floor for the rest of the pool's life. Worse, the connections already reaped for age had left the idle set by then, and the throw skipped the close that follows the lock — leaving them open with nothing able to reach them.

Every existing sweep test either had something idle or a floor at least as large as the number of leases, which is why none of them caught it. Two tests now cover it: the bare state, and the reaping-then-empty case that asserts the reaped connection really is closed.

Worth noting as a general point about that swallow, which exists so a failed socket close cannot kill a thread-pool thread: it hides a bug just as readily. The TODO now says so next to the design note.

Both tests are kept, but the structure that made the bug possible is gone: with the floor test dropped, the two passes collapsed into one indexed walk, which cannot reach an entry that is not there.

One sweep at a time, and one buffer

The sweep allocated a List every time it retired anything. It now fills a reusable field — but a plain field is not safe here, because the closes deliberately happen outside the lock. Two overlapping sweeps would share the buffer, and the second would clear it while the first was still iterating it. Timer callbacks can overlap, a close being able to outlast a period as short as 1s. So a sweep now admits one caller at a time, through the same Interlocked guard the top-up already used, which also publishes the clear to the next sweep. Skipping is correct: the sweep already running reads the same state the skipped one would.

A test re-enters Sweep from inside a close, which is that window made deterministic. Without the guard it fails with Collection was modified and leaves the rest of the batch open — the same shape of bug as the swallowed throw above.

Disposal cannot lose a socket

A checkout still dialing is the one connection disposal could not reach: it is not in leased, so the abort at the end of the drain does not see it, and the dial observed only the caller's token. Disposal waited out PoolTimeout, found no straggler, and returned while the socket lived on for up to DialTimeout — 30s by default, and RequireUsableTimeout accepts up to int.MaxValue ms. Checkout dials now run on the pool's shutdown token too, reported as ObjectDisposedException; a caller's own cancellation still surfaces as a cancellation.

Concurrent disposal shares one teardown, too. A second caller used to see the flag and return at once, reporting a pool whose connections were still open — and might yet be aborted by the first caller — as closed. It now awaits the same completion.

The other side of that race is a dial that had already finished when the cancellation arrived, so it hands back a connection to a pool that is already shut. Both the checkout and the top-up close such a connection themselves, since nothing else can reach it — that is why the disposed check shares a lock with the add to the set.

M12 was built, then dropped

The TODO called for auto max_execution_time = remaining lifetime − 5s, "like in Go". I implemented it, and review found the premise wrong on three counts:

1. The Go precedent describes something else. clickhouse-go derives max_execution_time from the caller's context deadline, not ConnMaxLifetime, and adds +5s rather than subtracting — so the client's own deadline fires first and the server limit is a backstop. It sends nothing when no deadline is set. One assignment in the library, context.go:240. No other official client (Java client-v2, clickhouse-connect, clickhouse-rs) derives a query limit from connection lifetime.
2. There is nothing to prevent. The pool reaches a connection only through the idle set, which a checkout removes it from, so a running query is never interrupted for age — exactly as Go's is not. The cap bought only a bound on lifetime overshoot.
3. It cost far more than that. With the 30-minute default the injected value ranged over [10s, 1795s]: a checkout landing in a connection's last ~35s capped that query at 10s, so the same query failed on ~0.8% of checkouts with a TIMEOUT_EXCEEDED naming a limit the user never set. Every query over ~30 minutes failed outright. And against a profile that constrains max_execution_time, ClickHouse throws rather than clamps (TCPHandler.cpp clamps only for SECONDARY_QUERY, never a client), so every operation would fail with error 452/164.

The defensible version is Go's — derive from a caller-supplied deadline, plus a margin, only when one exists. Filed against Q3/Q4, where ReadTimeout and cancellation live; it needs a way for a caller to express a deadline, which a bare CancellationToken does not carry.

Teardown, which is where the sharp edges were

Two rounds of review landed here, both worth reading in the diff:

• Disposal can reach a leased connection. Nothing but the lease-holder references a checked-out connection, so a lease that is never disposed used to leave its socket open with nothing able to close it — not even DisposeAsync. The pool now tracks the leased set, waits PoolTimeout for it to drain, and then aborts what is left.
• It aborts, it does not terminate. Terminate is documented unsafe to call concurrently with an operation, and it was: racing the operation's own unwinding could return one pooled buffer to ArrayPool twice and hand the same memory to two unrelated callers. AbortTransport closes the socket only — which is what frees an operation parked on a read that will never arrive — and leaves the buffers to the operation itself.
• And the buffers really do come back. Review caught that the first cut of this leaked them unconditionally: AbortTransport sets the state, and Terminate had an early return for an already-terminated connection, so the operation's own Terminate() became a no-op. Fixed rather than documented — ClickHouseBinaryReader.Dispose and ClickHouseBinaryWriter.Dispose now guard with Interlocked.Exchange, making the release exactly-once under a race, and Terminate drops its early return since every step in it is idempotent and the release has to run even when the state was set elsewhere. Only a genuinely abandoned operation leaks now, which is what the comment claims.
• A close that throws takes nothing else down. Swallowed once in PooledConnection.Close rather than at the four call sites: each batch is emptied from its set before anything is closed, so an escaping exception used to strand the rest, and on the return and checkout paths it surfaced as a failure of an unrelated caller's operation.
• The permit is released in a finally, so a throwing teardown cannot shrink the pool by one slot for the rest of the process.
• The sweep timer's callback swallows, since a timer callback faulting on a thread-pool thread ends the process rather than the sweep.

One build note: CI builds Release, where CHTCP0001 is an error rather than a warning — so nameof(ClickHouseTcpClient) inside the internal pool failed every matrix job while building clean locally in Debug. Fixed, and Release is now part of the local check.

Known and documented, not fixed: the sweep timer keeps the pool reachable, so a client that is never disposed holds its sockets for the life of the process. Every pooling library with a pruning timer has this; the client's doc comment now says to dispose it. And IsReusable cannot detect a connection dropped without a FIN. IdleTimeout now covers the common case of that, an intermediary dropping a connection nobody was using; what is left is a drop that strikes a connection in active use, and the answer to that is the idle read deadline of Q3, which is parsed but not yet enforced. The doc comment says exactly that rather than claiming a defence that does not exist.

Testing

2161 tests pass on net9 against a real server (Release, matching CI). ConnectionPool.cs is at 98% line coverage — the remainder is the sweep timer's defence-in-depth catch, which no path can now reach because each individual close already swallows, and the top-up's re-test of the floor once it holds a permit — and PooledConnection.cs, IConnectionFactory.cs and ClickHouseTcpClientOptions.cs are at 100%.

Split by what each layer can prove:

• ConnectionPoolTests runs the pool over scripted connections and a hand-driven TimeProvider, so age and idleness advance by minutes with no waiting and the sweep is invoked directly rather than raced against its timer. Covers reuse, both discard paths, both reuse policies, both expiry limits at their boundary exactly, queueing, exhaustion, cancellation, drain, a throwing close on each of the four paths, and the floor: topped up to it, not past it, counting connections in use, skipped when every slot is busy, never started without a floor, retried after a failed dial, and not started twice at once. Each expiry test disables the limit it is not about, since both now bar a checkout and either could otherwise explain the result. The derived sweep period is pinned at each boundary — both clamps, the shorter of two limits, the floor-only case and the no-timer case — and the override adds five more: that it replaces the derived period, that it is not clamped in either direction, that it replaces the floor-only period, and that no-timer still beats it. Plus a concurrency stress case — 800 checkouts over 32 tasks against a cap of 4 — asserting no two leases ever hold one connection, the cap is never exceeded, and every slot comes back.
• TcpConnectionFactoryTests uses a loopback TcpListener for the two things needing a real socket: DialTimeout bounding connect plus handshake (a server that accepts and then says nothing), and the Socket.Poll branch of IsReusable (a server that hangs up while the connection is idle).
• The abort path is proven with a real operation, not just a rented connection: a loopback listener completes the handshake and then goes silent, a QueryAsync blocks on the reply, and disposal past PoolTimeout has to release it. Asserted with Task.WhenAny rather than WaitAsync, whose own timeout would have read as success.
• ConnectionPoolIntegrationTests covers only what needs a server: that four concurrent 1s sleeps really do overlap, that 16 concurrent results are not crossed between connections, that concurrent inserts all land, and that an unfinished enumerator exhausts a pool of one — which is the case the timeout message calls out. Plus retirement end to end: ConnectionPoolTests drives a hand-held TimeProvider whose timers do nothing, so nothing there shows an over-idle connection retired without a test calling Sweep itself. A temporary table is the marker, the server scoping one to the connection that made it. It does not say which mechanism did it — the sweep timer and the checkout would both refuse that connection — only that the caller gets a working connection and not the stale one.

Every test that pins the new behaviour fails when it is reverted, checked by reverting each part in turn: the checkout and return checks, the idle clock stopping while a connection is out, and the floor override — restoring the old floor test fails three sweep tests while correctly leaving the age-only one green.

A stress harness found no pool defects. Not committed — it lives in the gitignored ignored-docs/ — but worth recording what it covers: eleven scenarios against a real server, including bursts after the whole pool has gone over-idle, 400 callers against a cap of 4, deliberate exhaustion, partial reads, cancellation, disposal racing live operations, floor churn, a 3s lifetime under continuous load, both reuse policies, and a 200ms SweepInterval. Each scenario ends with a leak check against the server's own TCPConnection metric and /proc/self/fd, plus a capacity check that holds every slot at the same time, since a permit lost or released twice is invisible to a throughput test. Worth recording the first run too: it passed while three scenarios proved nothing, the verified query being sub-millisecond, so 60 callers drained through 2 connections without ever reaching a 150ms PoolTimeout. Positive controls now fail a scenario whose intended path did not run — with them, exhaustion really happens (22 timed out, 2 got through), both cancellation paths fire (48 cancelled while queued, 10 mid-stream), and the floor is observed held at 4.

No CHANGELOG/RELEASENOTES entry: no TCP epic PR has one, the assembly being unreleased and [Experimental]. The epic gets a single entry when it ships.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Introduces concurrent TCP operations through a configurable, bounded connection pool.

Changes:

• Replaces the single-connection source with pooled connection leasing.
• Adds lifetime, idle-timeout, reuse-policy, and teardown handling.
• Adds extensive pool, socket, configuration, and integration tests.

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
ClickHouseTcpConnection.cs	Adds reuse validation and transport abort.
ClickHouseBinaryReader.cs	Exposes buffered-byte count.
SingleConnectionSource.cs	Removes serialized connection source.
PooledConnection.cs	Adds pooled connection bookkeeping.
IConnectionSource.cs	Updates leasing contract documentation.
IConnectionFactory.cs	Adds timeout-aware connection factory.
ConnectionPool.cs	Implements pooling, pruning, queueing, and disposal.
ClickHouseTcpPoolReusePolicy.cs	Adds FIFO/LIFO policies.
ClickHouseTcpConnectionStringBuilder.cs	Adds pool settings.
ClickHouseTcpClientOptions.cs	Adds pool options and validation.
ClickHouseTcpClient.cs	Switches the client to pooling.
PoolTestDoubles.cs	Adds deterministic pool test infrastructure.
ClickHouseTcpConnectionTests.cs	Tests connection reuse checks.
ConnectionPoolIntegrationTests.cs	Tests concurrent server operations.
TcpConnectionFactoryTests.cs	Tests dialing and socket liveness.
SingleConnectionSourceTests.cs	Removes obsolete source tests.
ConnectionPoolTests.cs	Covers pool lifecycle and concurrency.
ClickHouseTcpConnectionStringBuilderTests.cs	Tests pool setting parsing.
ClickHouseTcpClientOptionsTests.cs	Tests pool defaults and validation.

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 2 comments.

Suppressed comments (2)

ClickHouse.Driver.Tcp/Client/ConnectionPool.cs:306

• This indexes idle[0] even when the lifetime pass just removed every idle connection but leased.Count remains above the floor. The timer wrapper swallows the resulting exception before CloseAll(reaped), so those removed connections are never closed. Guard idle.Count > 0 before indexing.

            while (options.IdleTimeout > TimeSpan.Zero
                && idle.Count + leased.Count > options.MinPoolSize
                && idle[0].IdleFor >= options.IdleTimeout)

ClickHouse.Driver.Tcp/Client/ClickHouseTcpConnectionStringBuilder.cs:95

• This public description says the floor is a count of idle connections, but the pool and ClickHouseTcpClientOptions.MinPoolSize count idle plus in-use connections. That distinction changes capacity planning under steady load; align this connection-string API documentation with the implemented semantics.

    /// <summary>The number of idle connections kept rather than closed for inactivity. Defaults to 0.</summary>

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit d25b41d. Configure here.}