Skip to content

Commit 284ce13

Browse files
authored
feat(bigtable): BigtableDataClientFactory session support (#13829)
This PR extends the session-based API path to support `BigtableDataClientFactory`. Now the factory creates a single shared `ChannelPool` and `ClientConfigurationManager` once and hands lightweight child clients a reference to both. The bulk of the implementation work is a rewrite of `ChannelPoolDpImpl` to correctly handle multiple tenants (different instances/app profiles) sharing a single channel pool. ### `ChannelPoolDpImpl`: New Placement Model #### Old design The old pool organized channels into `AfeChannelGroup` objects — one deque of channels per observed AFE. When a channel's first session revealed its AFE (via `onBeforeSessionStart`), the channel was rehomed from the temporary `startingGroup` into the matching `AfeChannelGroup`, creating one if it didn't exist. New streams were then placed by picking the group with the fewest sessions. This design had a problem for the multi-tenant factory case: AFE affinity was tracked per channel, but for factory children different tenants on the same channel can be routed to different AFEs by RLS. A channel in group "AFE-7" might route tenant A to AFE-7 but tenant B somewhere else entirely. #### New design The new pool replaces the nested group structure with a flat `channels` list plus a `routeObservations` map keyed on `(channelId, tenantKey) → lastObservedAfeId`. ##### Data structures: - `channels: List<ChannelWrapper>` — all channels in ACTIVE or DRAINING state - `routeObservations: Map<RouteKey, AfeId>` — the last AFE seen for a given (channel, tenant) pair, populated on every `onBeforeSessionStart` and self-healing as RLS routes drift over time - `sessionsPerAfeId: Multiset<AfeId>` — global count of live sessions per AFE, used to find underloaded targets - `TenantKey` (new value type) — stamped onto `CallOptions` by `TableBase` so the pool can distinguish tenants within the same shared pool ##### Placement (`newStream`): 1. Capacity mode — find the AFE with the smallest session count that is still below `softMaxPerGroup`: - 3a — Preferred: pick the ACTIVE channel whose last observed route for this tenant was that AFE, choosing the least-loaded among candidates. This is the steady-state path: RLS is stable per (channel, tenant), so repeat sessions for the same tenant land on the same AFE. - 3b — Unobserved-tenant fallback: if no route observation exists for this tenant on any channel, pick any ACTIVE channel with remaining capacity (least-loaded). The actual AFE is unknown until `onBeforeSessionStart` fires; `routeObservations` is updated then for future placements. 2. Diversity mode — entered when all known AFEs are at cap, or the pool is empty: prefer an ACTIVE channel with fewer than `softMaxPerGroup / 2` outstanding streams. If none exists, open a new channel and absorb whichever AFE it lands on. ##### Scale-in (DRAINING state): The old "thinning" logic would call `channel.shutdown()` during serviceChannels(). The new approach introduces a two-state channel lifecycle (`ACTIVE` / `DRAINING`). When `serviceChannels()` decides to shrink the pool it marks excess channels DRAINING — they stop accepting new streams but continue serving existing ones. A DRAINING channel is removed only when its `numOutstanding` reaches zero (in `releaseChannel`). Drain candidates are chosen by `pickDrainCandidates`, which prefers idle channels first, then least-recently-used, then oldest. ##### Route observation cleanup: `removeChannel` purges all `routeObservations` entries for the removed channel's ID, preventing the map from growing unboundedly as channels cycle. ### Other changes #### `BigtableDataClientFactory` / `BigtableClientContext`: - `BigtableDataClientFactory.create()` now calls `BigtableClientContext.createForFactory()` (a new entry point) instead of manually disabling sessions. The factory context initializes a shared `ShimImpl` (channel pool + config manager) that is reused by all children. - `BigtableClientContext.createChild()` now creates a lightweight `ShimImpl` child via `ShimImpl.createForFactoryChild()` rather than sharing a `DisabledShim`. #### `ShimImpl` / `Client`: - `Client.channelPool` changed from a bare `ChannelPool` reference to `Resource<ChannelPool>` so the factory-child constructor can hold a shared (non-closing) reference and the standard constructor holds an owned one. - `ShimImpl.configManager` similarly wrapped in `Resource<>` so `close()` is a no-op for factory children (the parent owns the manager's lifecycle).
1 parent ccd13eb commit 284ce13

9 files changed

Lines changed: 463 additions & 238 deletions

File tree

java-bigtable/google-cloud-bigtable/src/main/java/com/google/cloud/bigtable/data/v2/BigtableDataClientFactory.java

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -74,10 +74,6 @@ public final class BigtableDataClientFactory implements AutoCloseable {
7474
*/
7575
public static BigtableDataClientFactory create(BigtableDataSettings defaultSettings)
7676
throws IOException {
77-
BigtableDataSettings.Builder builder = defaultSettings.toBuilder();
78-
builder.stubSettings().setSessionsEnabled(false);
79-
defaultSettings = builder.build();
80-
8177
BigtableClientContext sharedClientContext =
8278
BigtableClientContext.create(defaultSettings.getStubSettings());
8379
ClientOperationSettings perOpSettings = defaultSettings.getStubSettings().getPerOpSettings();

java-bigtable/google-cloud-bigtable/src/main/java/com/google/cloud/bigtable/data/v2/internal/api/Client.java

Lines changed: 54 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -100,7 +100,7 @@ public class Client implements AutoCloseable {
100100
private final BigtableTimer sessionTimer;
101101

102102
private final CallOptions defaultCallOptions;
103-
private final ChannelPool channelPool;
103+
private final Resource<ChannelPool> channelPool;
104104
private final Resource<Metrics> metrics;
105105
private final Resource<ClientConfigurationManager> configManager;
106106

@@ -185,22 +185,26 @@ public static Client create(ClientSettings settings) throws IOException {
185185
return new Client(
186186
featureFlags,
187187
clientInfo,
188-
settings.getChannelProvider(),
189188
Resource.createOwned(metrics, metrics::close),
190189
Resource.createOwned(configManager, configManager::close),
191190
Resource.createOwned(backgroundExecutor, backgroundExecutor::shutdown),
192191
Resource.<Executor>createOwned(
193-
userCallbackExecutor, () -> shutdownAndAwait(userCallbackExecutor)));
192+
userCallbackExecutor, () -> shutdownAndAwait(userCallbackExecutor)),
193+
settings.getChannelProvider());
194194
}
195195

196+
/**
197+
* Standard constructor used by non-factory clients. Builds an owned {@link SwitchingChannelPool}
198+
* from the provided {@link ChannelProvider}.
199+
*/
196200
public Client(
197201
FeatureFlags featureFlags,
198202
ClientInfo clientInfo,
199-
ChannelProvider channelProvider,
200203
Resource<Metrics> metrics,
201204
Resource<ClientConfigurationManager> configManager,
202205
Resource<ScheduledExecutorService> bgExecutor,
203-
Resource<Executor> userCallbackExecutor)
206+
Resource<Executor> userCallbackExecutor,
207+
ChannelProvider channelProvider)
204208
throws IOException {
205209
this.featureFlags = featureFlags;
206210
this.clientInfo = clientInfo;
@@ -224,13 +228,38 @@ public Client(
224228
// TODO: consider localizing this for large reads
225229
.maxInboundMessageSize(256 * 1024 * 1024));
226230

227-
channelPool =
231+
SwitchingChannelPool switchingPool =
228232
new SwitchingChannelPool(
229233
configuredChannelProvider,
230234
configManager.get(),
231235
metrics.get(),
232236
backgroundExecutor.get());
233-
channelPool.start();
237+
switchingPool.start();
238+
this.channelPool = Resource.createOwned(switchingPool, switchingPool::close);
239+
}
240+
241+
/**
242+
* Factory-child constructor. Uses a pre-built, shared {@link ChannelPool} and {@link
243+
* ClientConfigurationManager}. The pool and other resources should be created with
244+
* Resource.createShared() and closed when the factory closes.
245+
*/
246+
public Client(
247+
FeatureFlags featureFlags,
248+
ClientInfo clientInfo,
249+
Resource<Metrics> metrics,
250+
Resource<ClientConfigurationManager> configManager,
251+
Resource<ScheduledExecutorService> bgExecutor,
252+
Resource<Executor> userCallbackExecutor,
253+
Resource<ChannelPool> sharedChannelPool) {
254+
this.featureFlags = featureFlags;
255+
this.clientInfo = clientInfo;
256+
this.metrics = metrics;
257+
this.configManager = configManager;
258+
this.backgroundExecutor = bgExecutor;
259+
this.userCallbackExecutor = userCallbackExecutor;
260+
this.channelPool = sharedChannelPool;
261+
this.sessionTimer = new HashedWheelTimer("bigtable-session-timer");
262+
defaultCallOptions = CallOptions.DEFAULT;
234263
}
235264

236265
@Override
@@ -329,7 +358,7 @@ public TableAsync openTableAsync(String tableId, Permission permission) {
329358
featureFlags,
330359
clientInfo,
331360
configManager.get(),
332-
channelPool,
361+
channelPool.get(),
333362
defaultCallOptions,
334363
tableId,
335364
permission,
@@ -353,7 +382,7 @@ public AuthorizedViewAsync openAuthorizedViewAsync(
353382
featureFlags,
354383
clientInfo,
355384
configManager.get(),
356-
channelPool,
385+
channelPool.get(),
357386
defaultCallOptions,
358387
tableId,
359388
viewId,
@@ -378,7 +407,7 @@ public MaterializedViewAsync openMaterializedViewAsync(
378407
featureFlags,
379408
clientInfo,
380409
configManager.get(),
381-
channelPool,
410+
channelPool.get(),
382411
defaultCallOptions,
383412
viewId,
384413
permission,
@@ -391,6 +420,21 @@ public MaterializedViewAsync openMaterializedViewAsync(
391420
}
392421
}
393422

423+
/** Returns the underlying channel pool (e.g. for sharing with factory children). */
424+
public ChannelPool getChannelPool() {
425+
return channelPool.get();
426+
}
427+
428+
/** Returns the feature flags (e.g. for sharing with factory children). */
429+
public FeatureFlags getFeatureFlags() {
430+
return featureFlags;
431+
}
432+
433+
/** Returns the underlying user callback executor (e.g. for sharing with factory children). */
434+
public Executor getUserCallbackExecutor() {
435+
return userCallbackExecutor.get();
436+
}
437+
394438
public static class Resource<T> {
395439
private final T value;
396440
private final Runnable closer;

java-bigtable/google-cloud-bigtable/src/main/java/com/google/cloud/bigtable/data/v2/internal/api/TableBase.java

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,8 @@
2222
import com.google.bigtable.v2.SessionReadRowRequest;
2323
import com.google.bigtable.v2.SessionReadRowResponse;
2424
import com.google.cloud.bigtable.data.v2.internal.channels.ChannelPool;
25+
import com.google.cloud.bigtable.data.v2.internal.channels.ChannelPoolOptions;
26+
import com.google.cloud.bigtable.data.v2.internal.channels.TenantKey;
2527
import com.google.cloud.bigtable.data.v2.internal.csm.Metrics;
2628
import com.google.cloud.bigtable.data.v2.internal.csm.attributes.ClientInfo;
2729
import com.google.cloud.bigtable.data.v2.internal.csm.tracers.VRpcTracer;
@@ -66,14 +68,20 @@ static <ReqT extends Message> TableBase createAndStart(
6668
Executor backgroundExecutor,
6769
Executor userCallbackExecutor) {
6870

71+
// Stamp the tenant key so ChannelPoolDpImpl can make tenant-aware placement decisions.
72+
CallOptions stamped =
73+
callOptions.withOption(
74+
ChannelPoolOptions.TENANT_KEY_OPTION,
75+
new TenantKey(clientInfo.getInstanceName(), clientInfo.getAppProfileId()));
76+
6977
SessionPool<ReqT> sessionPool =
7078
new SessionPoolImpl<>(
7179
metrics,
7280
featureFlags,
7381
clientInfo,
7482
configManager,
7583
channelPool,
76-
callOptions,
84+
stamped,
7785
sessionDescriptor,
7886
sessionPoolName,
7987
timer,

0 commit comments

Comments
 (0)