Skip to content

Commit 90f95ef

Browse files
committed
feat(bigtable): BigtableDataClientFactory session support
This commit extends the session-based API path to support BigtableDataClientFactory. Now the factory creates a single shared ChannelPool and ClientConfigurationManager once and hands lightweight child clients a reference to both. The bulk of the implementation work is a rewrite of ChannelPoolDpImpl to correctly handle multiple tenants (different instances/app profiles) sharing a single channel pool. --- ChannelPoolDpImpl: New Placement Model Old design The old pool organized channels into AfeChannelGroup objects — one deque of channels per observed AFE. When a channel's first session revealed its AFE (via onBeforeSessionStart), the channel was rehomed from the temporary startingGroup into the matching AfeChannelGroup, creating one if it didn't exist. New streams were then placed by picking the group with the fewest sessions. This design had a problem for the multi-tenant factory case: AFE affinity was tracked per channel, but for factory children different tenants on the same channel can be routed to different AFEs by RLS. A channel in group "AFE-7" might route tenant A to AFE-7 but tenant B somewhere else entirely. New design The new pool replaces the nested group structure with a flat channels list plus a routeObservations map keyed on (channelId, tenantKey) → lastObservedAfeId. Data structures: - channels: List<ChannelWrapper> — all channels in ACTIVE or DRAINING state - routeObservations: Map<RouteKey, AfeId> — the last AFE seen for a given (channel, tenant) pair, populated on every onBeforeSessionStart and self-healing as RLS routes drift over time - sessionsPerAfeId: Multiset<AfeId> — global count of live sessions per AFE, used to find underloaded targets - TenantKey (new value type) — stamped onto CallOptions by TableBase so the pool can distinguish tenants within the same shared pool Placement (newStream): 1. Capacity mode — find the AFE with the smallest session count that is still below softMaxPerGroup: - 3a — Preferred: pick the ACTIVE channel whose last observed route for this tenant was that AFE, choosing the least-loaded among candidates. This is the steady-state path: RLS is stable per (channel, tenant), so repeat sessions for the same tenant land on the same AFE. - 3b — Unobserved-tenant fallback: if no route observation exists for this tenant on any channel, pick any ACTIVE channel with remaining capacity (least-loaded). The actual AFE is unknown until onBeforeSessionStart fires; routeObservations is updated then for future placements. 2. Diversity mode — entered when all known AFEs are at cap, or the pool is empty: prefer an ACTIVE channel with fewer than softMaxPerGroup / 2 outstanding streams. If none exists, open a new channel and absorb whichever AFE it lands on. Scale-in (DRAINING state): The old "thinning" logic would call channel.shutdown() during serviceChannels(). The new approach introduces a two-state channel lifecycle (ACTIVE / DRAINING). When serviceChannels() decides to shrink the pool it marks excess channels DRAINING — they stop accepting new streams but continue serving existing ones. A DRAINING channel is removed only when its numOutstanding reaches zero (in releaseChannel). Drain candidates are chosen by pickDrainCandidates, which prefers idle channels first, then least-recently-used, then oldest. Route observation cleanup: removeChannel purges all routeObservations entries for the removed channel's ID, preventing the map from growing unboundedly as channels cycle. --- Other changes BigtableDataClientFactory / BigtableClientContext: - BigtableDataClientFactory.create() now calls BigtableClientContext.createForFactory() (a new entry point) instead of manually disabling sessions. The factory context initializes a shared ShimImpl (channel pool + config manager) that is reused by all children. - BigtableClientContext.createChild() now creates a lightweight ShimImpl child via ShimImpl.createForFactoryChild() rather than sharing a DisabledShim. ShimImpl / Client: - Client.channelPool changed from a bare ChannelPool reference to Resource<ChannelPool> so the factory-child constructor can hold a shared (non-closing) reference and the standard constructor holds an owned one. - ShimImpl.configManager similarly wrapped in Resource<> so close() is a no-op for factory children (the parent owns the manager's lifecycle).
1 parent 5ff7a0f commit 90f95ef

9 files changed

Lines changed: 477 additions & 242 deletions

File tree

java-bigtable/google-cloud-bigtable/src/main/java/com/google/cloud/bigtable/data/v2/BigtableDataClientFactory.java

Lines changed: 1 addition & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -74,12 +74,8 @@ public final class BigtableDataClientFactory implements AutoCloseable {
7474
*/
7575
public static BigtableDataClientFactory create(BigtableDataSettings defaultSettings)
7676
throws IOException {
77-
BigtableDataSettings.Builder builder = defaultSettings.toBuilder();
78-
builder.stubSettings().setSessionsEnabled(false);
79-
defaultSettings = builder.build();
80-
8177
BigtableClientContext sharedClientContext =
82-
BigtableClientContext.create(defaultSettings.getStubSettings());
78+
BigtableClientContext.createForFactory(defaultSettings.getStubSettings());
8379
ClientOperationSettings perOpSettings = defaultSettings.getStubSettings().getPerOpSettings();
8480
return new BigtableDataClientFactory(sharedClientContext, perOpSettings);
8581
}

java-bigtable/google-cloud-bigtable/src/main/java/com/google/cloud/bigtable/data/v2/internal/api/Client.java

Lines changed: 46 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -73,7 +73,7 @@ public class Client implements AutoCloseable {
7373
private final Resource<ScheduledExecutorService> backgroundExecutor;
7474

7575
private final CallOptions defaultCallOptions;
76-
private final ChannelPool channelPool;
76+
private final Resource<ChannelPool> channelPool;
7777
private final Resource<Metrics> metrics;
7878
private final Resource<ClientConfigurationManager> configManager;
7979

@@ -147,19 +147,23 @@ public static Client create(ClientSettings settings) throws IOException {
147147
return new Client(
148148
featureFlags,
149149
clientInfo,
150-
settings.getChannelProvider(),
151150
Resource.createOwned(metrics, metrics::close),
152151
Resource.createOwned(configManager, configManager::close),
153-
Resource.createOwned(backgroundExecutor, backgroundExecutor::shutdown));
152+
Resource.createOwned(backgroundExecutor, backgroundExecutor::shutdown),
153+
settings.getChannelProvider());
154154
}
155155

156+
/**
157+
* Standard constructor used by non-factory clients. Builds an owned {@link SwitchingChannelPool}
158+
* from the provided {@link ChannelProvider}.
159+
*/
156160
public Client(
157161
FeatureFlags featureFlags,
158162
ClientInfo clientInfo,
159-
ChannelProvider channelProvider,
160163
Resource<Metrics> metrics,
161164
Resource<ClientConfigurationManager> configManager,
162-
Resource<ScheduledExecutorService> bgExecutor)
165+
Resource<ScheduledExecutorService> bgExecutor,
166+
ChannelProvider channelProvider)
163167
throws IOException {
164168
this.featureFlags = featureFlags;
165169
this.clientInfo = clientInfo;
@@ -181,13 +185,34 @@ public Client(
181185
// TODO: consider localizing this for large reads
182186
.maxInboundMessageSize(256 * 1024 * 1024));
183187

184-
channelPool =
188+
SwitchingChannelPool switchingPool =
185189
new SwitchingChannelPool(
186190
configuredChannelProvider,
187191
configManager.get(),
188192
metrics.get(),
189193
backgroundExecutor.get());
190-
channelPool.start();
194+
switchingPool.start();
195+
this.channelPool = Resource.createOwned(switchingPool, switchingPool::close);
196+
}
197+
198+
/**
199+
* Factory-child constructor. Uses a pre-built, shared {@link ChannelPool} and {@link
200+
* ClientConfigurationManager}. The pool is already started and must not be closed by this client.
201+
*/
202+
public Client(
203+
FeatureFlags featureFlags,
204+
ClientInfo clientInfo,
205+
Resource<Metrics> metrics,
206+
Resource<ClientConfigurationManager> configManager,
207+
Resource<ScheduledExecutorService> bgExecutor,
208+
Resource<ChannelPool> sharedChannelPool) {
209+
this.featureFlags = featureFlags;
210+
this.clientInfo = clientInfo;
211+
this.metrics = metrics;
212+
this.configManager = configManager;
213+
this.backgroundExecutor = bgExecutor;
214+
this.channelPool = sharedChannelPool;
215+
defaultCallOptions = CallOptions.DEFAULT;
191216
}
192217

193218
@Override
@@ -200,7 +225,7 @@ public void close() {
200225
.setDescription("Client closing")
201226
.build()));
202227
metrics.close();
203-
channelPool.close();
228+
channelPool.close(); // no-op when Resource.createShared (factory child)
204229
configManager.close();
205230
backgroundExecutor.close();
206231
}
@@ -211,7 +236,7 @@ public TableAsync openTableAsync(String tableId, Permission permission) {
211236
featureFlags,
212237
clientInfo,
213238
configManager.get(),
214-
channelPool,
239+
channelPool.get(),
215240
defaultCallOptions,
216241
tableId,
217242
permission,
@@ -228,7 +253,7 @@ public AuthorizedViewAsync openAuthorizedViewAsync(
228253
featureFlags,
229254
clientInfo,
230255
configManager.get(),
231-
channelPool,
256+
channelPool.get(),
232257
defaultCallOptions,
233258
tableId,
234259
viewId,
@@ -246,7 +271,7 @@ public MaterializedViewAsync openMaterializedViewAsync(
246271
featureFlags,
247272
clientInfo,
248273
configManager.get(),
249-
channelPool,
274+
channelPool.get(),
250275
defaultCallOptions,
251276
viewId,
252277
permission,
@@ -256,6 +281,16 @@ public MaterializedViewAsync openMaterializedViewAsync(
256281
return viewAsync;
257282
}
258283

284+
/** Returns the underlying channel pool (e.g. for sharing with factory children). */
285+
public ChannelPool getChannelPool() {
286+
return channelPool.get();
287+
}
288+
289+
/** Returns the feature flags (e.g. for sharing with factory children). */
290+
public FeatureFlags getFeatureFlags() {
291+
return featureFlags;
292+
}
293+
259294
public static class Resource<T> {
260295
private T value;
261296
private Runnable closer;

java-bigtable/google-cloud-bigtable/src/main/java/com/google/cloud/bigtable/data/v2/internal/api/TableBase.java

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,8 @@
2222
import com.google.bigtable.v2.SessionReadRowRequest;
2323
import com.google.bigtable.v2.SessionReadRowResponse;
2424
import com.google.cloud.bigtable.data.v2.internal.channels.ChannelPool;
25+
import com.google.cloud.bigtable.data.v2.internal.channels.ChannelPoolOptions;
26+
import com.google.cloud.bigtable.data.v2.internal.channels.TenantKey;
2527
import com.google.cloud.bigtable.data.v2.internal.csm.Metrics;
2628
import com.google.cloud.bigtable.data.v2.internal.csm.attributes.ClientInfo;
2729
import com.google.cloud.bigtable.data.v2.internal.csm.tracers.VRpcTracer;
@@ -63,14 +65,20 @@ static <ReqT extends Message> TableBase createAndStart(
6365
Metrics metrics,
6466
ScheduledExecutorService executor) {
6567

68+
// Stamp the tenant key so ChannelPoolDpImpl can make tenant-aware placement decisions.
69+
CallOptions stamped =
70+
callOptions.withOption(
71+
ChannelPoolOptions.TENANT_KEY_OPTION,
72+
new TenantKey(clientInfo.getInstanceName(), clientInfo.getAppProfileId()));
73+
6674
SessionPool<ReqT> sessionPool =
6775
new SessionPoolImpl<>(
6876
metrics,
6977
featureFlags,
7078
clientInfo,
7179
configManager,
7280
channelPool,
73-
callOptions,
81+
stamped,
7482
sessionDescriptor,
7583
sessionPoolName,
7684
executor);

0 commit comments

Comments
 (0)