feat(gax): support transparent retries during mTLS certificate rotations - #13993
Closed
macastelaz wants to merge 19 commits into
Closed
feat(gax): support transparent retries during mTLS certificate rotations#13993macastelaz wants to merge 19 commits into
macastelaz wants to merge 19 commits into
Conversation
…eapis#13931) Remove unused python setup step in workflows. They are unused since googleapis#13912 and causing problems in clean up steps. Fixes googleapis/librarian#7096
…fault via Conscrypt (googleapis#13853)
…#13932) Remove PYTHONPATH exports and deleted template paths from release-please configs. Fixes googleapis/librarian#7064
Updated googleapis commitish in librarian.yaml to googleapis/googleapis@b8486a2 💡 **Note:** If this PR is still open when the daily update workflow runs next, it will be closed and replaced with a new PR containing the latest updates.
…eapis#13935) b/538631816 This PR integrates OpenTelemetry tracing context into the JDBC driver's local file logging system, bringing it to parity with the existing Google Cloud Logging export integration. **Key Changes:** * **`BigQueryJdbcRootLogger.java`**: Updated the root formatter to extract the active `TraceId` and `SpanId` from the current OpenTelemetry thread context. When a valid trace is active, local logs will now automatically include a `[trace_id=... span_id=...]` block. * **`BigQueryJdbcRootLoggerTest.java`**: Added `testFormatterWithOpenTelemetrySpan` to verify the formatter correctly extracts and formats IDs when a span is active.
Fixes: googleapis#13925 Output of `zizmor --fix=all --gh-token=$(gh auth token) ${YOUR_GH_DIRECTORY}/.github/workflows/` From: go/github-zizmor-help
…sts (googleapis#13950) Exposes BIGQUERY_ENDPOINT and BIGQUERY_STORAGE_ENDPOINT environment variables to override endpoints in integration tests, making it possible to target regional canary endpoints like us-east7. b/472499857
…e stream initial query. (googleapis#13885)
…keep-alive (googleapis#13897) - Enable `setRemoveOnCancelPolicy(true)` on `KEEP_ALIVE_SERVICE` so canceled tasks are immediately purged from `DelayedWorkQueue`. - Use a `WeakReference<ReadWriteTransaction>` in `KeepAliveRunnable` to prevent scheduled tasks from retaining strong references to transaction instances. - Use `abortedLock.tryLock()` in `KeepAliveRunnable` so the shared executor thread does not block when a transaction is active or retrying. - Remove duplicate `maybeScheduleKeepAlivePing` listener registration on keep-alive query completion. - Add unit tests in `ReadWriteTransactionTest` verifying task removal on cancel, weak reference retention, non-blocking lock handling, and single ping scheduling on completion.
🤖 I have created a release *beep* *boop* --- <details><summary>1.89.0</summary> ## [1.89.0](googleapis/google-cloud-java@v1.88.0...v1.89.0) (2026-07-29) ### Features * **agentidentity:** onboard v1 and v1beta API versions ([googleapis#13796](googleapis#13796)) ([1b1300d](googleapis@1b1300d)) * **auth:** add JSpecify Null annotations to Auth ([googleapis#13842](googleapis#13842)) ([f6f24d4](googleapis@f6f24d4)) * **bigquery-jdbc:** add `SSLTrustStoreType` and `SSLTrustStoreProvider` connection properties ([googleapis#13858](googleapis#13858)) ([9449be1](googleapis@9449be1)) * **bigquery-jdbc:** add otel trace and span IDs to local logs ([googleapis#13935](googleapis#13935)) ([2801fbd](googleapis@2801fbd)) * **bigquery-jdbc:** implement BigQueryParameterMetaData and dynamic type mappings ([googleapis#13812](googleapis#13812)) ([3d67dba](googleapis@3d67dba)) * **bigquery-jdbc:** implement parameter setters in PreparedStatement ([googleapis#13792](googleapis#13792)) ([94f7404](googleapis@94f7404)) * **bigquery-jdbc:** Migrate `getImportedKeys` and `getCrossReference` to BQ API ([googleapis#13692](googleapis#13692)) ([082b046](googleapis@082b046)) * **bigquery-jdbc:** migrate `getPrimaryKeys` to use BQ API ([googleapis#13691](googleapis#13691)) ([1951f49](googleapis@1951f49)) * **bigquery-jdbc:** OpenTelemetry integration in BQ JDBC ([googleapis#12902](googleapis#12902)) ([af18f65](googleapis@af18f65)) * **bigquery-jdbc:** optimize memory footprint for JSON result set streaming ([googleapis#13660](googleapis#13660)) ([11f26d3](googleapis@11f26d3)) * **bigquery-jdbc:** standardize parameter handling and calendar defensive copying across statement interfaces ([googleapis#13805](googleapis#13805)) ([ccd13eb](googleapis@ccd13eb)) * **bigtable:** add view_parameters support to BoundStatement ([googleapis#13673](googleapis#13673)) ([d5cc437](googleapis@d5cc437)) * **bigtable:** BigtableDataClientFactory session support ([googleapis#13829](googleapis#13829)) ([284ce13](googleapis@284ce13)) * **commerceproducer:** onboard v1beta API ([googleapis#13814](googleapis#13814)) ([61b89b3](googleapis@61b89b3)) * Default to least-in-flight balancing for Bigtable unary clients ([googleapis#13802](googleapis#13802)) ([ac9ccd1](googleapis@ac9ccd1)) * **firestore:** Add support for 16MB documents ([googleapis#13478](googleapis#13478)) ([1b7c2e0](googleapis@1b7c2e0)) * **gapic-generator:** add JSpecify Null annotations to the generator classes ([googleapis#13769](googleapis#13769)) ([843bd7c](googleapis@843bd7c)) * **gapic-generator:** Add Nullable annotation to generated classes ([googleapis#13558](googleapis#13558)) ([e3e9d0b](googleapis@e3e9d0b)) * **gapic-generator:** Add NullMarked annotation to generated classes ([googleapis#13584](googleapis#13584)) ([b7a8504](googleapis@b7a8504)) * **gax-httpjson:** Add Post Quantum Cryptography (PQC) Support by default via Conscrypt ([googleapis#13853](googleapis#13853)) ([550df81](googleapis@550df81)) * **gax-java:** add JSpecify Null annotations to gax ([googleapis#13799](googleapis#13799)) ([65aee08](googleapis@65aee08)) * **google/cloud/sql:** onboard a new library ([googleapis#13864](googleapis#13864)) ([38e272e](googleapis@38e272e)) * **google/maps/navconnect/v1:** onboard a new library ([googleapis#13927](googleapis#13927)) ([2256394](googleapis@2256394)) * **maps-isochrones:** onboard v1 API ([googleapis#13817](googleapis#13817)) ([3037ab3](googleapis@3037ab3)) * port secure_context testing support to executor proxy ([googleapis#13522](googleapis#13522)) ([0f81bf0](googleapis@0f81bf0)) * **productregistry:** onboard v1 API ([googleapis#13816](googleapis#13816)) ([9517313](googleapis@9517313)) * **storage:** allow checksum on appendable upload finalization ([googleapis#13833](googleapis#13833)) ([ddf9add](googleapis@ddf9add)) * **storage:** enable App-Centric Observability (ACO) support in Otel ([googleapis#13248](googleapis#13248)) ([4329896](googleapis@4329896)) ### Bug Fixes * **bigquery-jdbc:** Add PerConnectionHandler to list of excempted logging classes ([googleapis#13888](googleapis#13888)) ([50b3c24](googleapis@50b3c24)) * **bigquery-jdbc:** add preferIPv4Stack to argLine for Kokoro reliability ([googleapis#13923](googleapis#13923)) ([d5e33d6](googleapis@d5e33d6)) * **bigquery-jdbc:** add service resource transformer for standalone IT ([googleapis#13893](googleapis#13893)) ([dc80fe8](googleapis@dc80fe8)) * **bigquery-jdbc:** align metadata methods error handling with spec ([googleapis#13793](googleapis#13793)) ([d85fb10](googleapis@d85fb10)) * **bigquery-jdbc:** fix WriteAPI when running in restricted environment ([googleapis#13856](googleapis#13856)) ([66ba925](googleapis@66ba925)) * **bigquery-jdbc:** refine temporal timezone coercion and PreparedStatement parameter setters ([googleapis#13813](googleapis#13813)) ([6f68c4d](googleapis@6f68c4d)) * **bigquery-jdbc:** resolve `ITOpenTelemetryTest` pipeline and trace validation failures ([googleapis#13898](googleapis#13898)) ([c18141d](googleapis@c18141d)) * **bigquery-jdbc:** resolve failing otel IT in nightly ([googleapis#13915](googleapis#13915)) ([ac79713](googleapis@ac79713)) * **bigquery:** resultSet.getLong() does not truncate for large int64 values ([googleapis#13718](googleapis#13718)) ([bc19822](googleapis@bc19822)) * **bigquery:** support optional fields in BigLakeConfiguration to prevent NPE on Iceberg/Lakehouse tables ([googleapis#13733](googleapis#13733)) ([e2cca4d](googleapis@e2cca4d)) * **bigtable:** add materialized view routing param to ReadRows and Sa… ([googleapis#13918](googleapis#13918)) ([4ddf250](googleapis@4ddf250)) * **bigtable:** bound SessionPoolImpl lock to prevent pod-wide wedge ([googleapis#13890](googleapis#13890)) ([ed87a68](googleapis@ed87a68)) * **bigtable:** fix session creation leaks ([googleapis#13887](googleapis#13887)) ([d586d07](googleapis@d586d07)) * **bigtable:** prevent ClientConfigurationManagerTest from wedging on… ([googleapis#13907](googleapis#13907)) ([725086d](googleapis@725086d)) * **bigtable:** stop installing DirectpathEnforcer on the directpath pool ([googleapis#13880](googleapis#13880)) ([5f2e056](googleapis@5f2e056)) * **bom:** make release-note-generation Java 8 compatible ([googleapis#13837](googleapis#13837)) ([bc18390](googleapis@bc18390)) * **ci:** fix java-cloud-bom release-notes workflow errors ([googleapis#13682](googleapis#13682)) ([b679835](googleapis@b679835)) * deprecate resource detector ([googleapis#13844](googleapis#13844)) ([aba4f01](googleapis@aba4f01)) * **deps:** align logback versions and add java8 profile in storage ([googleapis#13678](googleapis#13678)) ([7e57092](googleapis@7e57092)) * do not start stream with direct executor ([googleapis#13945](googleapis#13945)) ([630e790](googleapis@630e790)) * fix java-cloud-bom README update workflow after monorepo migration ([googleapis#13892](googleapis#13892)) ([5b8e295](googleapis@5b8e295)) * **oauth2_http:** Avoid retrying on 4xx errors during GCE metadata ping ([googleapis#13715](googleapis#13715)) ([537c16c](googleapis@537c16c)) * regenerate ([googleapis#13714](googleapis#13714)) ([8a72860](googleapis@8a72860)) * regenerate libraries ([googleapis#13703](googleapis#13703)) ([a29ea79](googleapis@a29ea79)), refs [googleapis#13690](googleapis#13690) * **release:** handle missing release tags gracefully in release-note-generation ([googleapis#13795](googleapis#13795)) ([43005fb](googleapis@43005fb)) * **release:** resolve first-party-dependencies SNAPSHOT in libraries-bom ([googleapis#13790](googleapis#13790)) ([d5bfe21](googleapis@d5bfe21)) * **spanner:** avoid data race on DIRECTPATH_CHANNEL_CREATED by using volatile ([googleapis#13727](googleapis#13727)) ([1e05ea5](googleapis@1e05ea5)) * **spanner:** prevent fastpath tablet routing flaps ([googleapis#13803](googleapis#13803)) ([4dabdac](googleapis@4dabdac)) * **storage:** BidiAppendableUpload Takeover operation fixes ([googleapis#13776](googleapis#13776)) ([f2d0474](googleapis@f2d0474)) * **storage:** correctly insert explicit nulls for json patch updates ([googleapis#13716](googleapis#13716)) ([4fb3f4b](googleapis@4fb3f4b)) * update group id mapping ([googleapis#13698](googleapis#13698)) ([50a72e1](googleapis@50a72e1)) * use a new managed channel builder when creating channels ([googleapis#13684](googleapis#13684)) ([a049999](googleapis@a049999)) ### Performance Improvements * **bigquery-jdbc:** optimize getExportedKeys performance using hybrid metadata lookup ([googleapis#13734](googleapis#13734)) ([c9738ea](googleapis@c9738ea)) ### Dependencies * Add Conscrypt to shared-deps ([googleapis#13838](googleapis#13838)) ([05ce6ce](googleapis@05ce6ce)) * move conscrypt from third-party-dependencies POM to gax-java POM ([googleapis#13948](googleapis#13948)) ([1e634ec](googleapis@1e634ec)) * **shared-deps:** migrate awaitility to shared-dependencies ([googleapis#13671](googleapis#13671)) ([abb91ef](googleapis@abb91ef)) * **shared-deps:** switch conscrypt shared dependency to conscrypt-openjdk-uber ([googleapis#13845](googleapis#13845)) ([2e3f208](googleapis@2e3f208)) * Update gRPC-Java to v1.82.2 ([googleapis#13877](googleapis#13877)) ([da228d8](googleapis@da228d8)) * Update http-client to v2.2.0 ([googleapis#13854](googleapis#13854)) ([334a2c5](googleapis@334a2c5)) * Update Protobuf-Java to v4.33.6 ([googleapis#13876](googleapis#13876)) ([5c6478c](googleapis@5c6478c)) * Upgrade Guava to v33.6.0-jre ([googleapis#13875](googleapis#13875)) ([41a7a52](googleapis@41a7a52)) ### Documentation * add ErrorProne and NullAway integration guide and JSpecify migration playbook ([googleapis#13882](googleapis#13882)) ([d5ea739](googleapis@d5ea739)) * add JSpecify nullness guidelines to AGENTS.md ([googleapis#13881](googleapis#13881)) ([54b846b](googleapis@54b846b)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
… export (googleapis#13741) ## Summary Adds **Client Metrics**: a new, opt-in feature that exports Spanner's client-side metrics to a caller-provided OpenTelemetry pipeline (OTLP, Prometheus, any exporter), including on Spanner Omni where the existing Cloud Monitoring metrics are unavailable. Fully opt-in and fully decoupled from the existing built-in (Cloud Monitoring) metrics. Default behavior is unchanged: without a client-metrics provider, nothing new happens, and built-in metrics continue to export to Cloud Monitoring exactly as before. > **Scope note:** This change was split out of a larger PR into two focused PRs. This PR contains **only** the Client Metrics export feature. The optional `endpoint` metric attribute for location-aware routing is a separate follow-up: googleapis#13740. ## Two independent features - **Built-in metrics** (existing, unchanged): export to Google Cloud Monitoring. Controlled by `setBuiltInMetricsEnabled` and the `SPANNER_DISABLE_BUILTIN_METRICS` environment variable, exactly as today. Not available on Spanner Omni. - **Client metrics** (new): export the client instruments to a caller-owned OpenTelemetry. Controlled **solely** by `setClientMetricsProvider(...)` — a `CustomOpenTelemetryMetricsProvider` turns it on; `NoopMetricsProvider` (or no provider) turns it off. Works on all instance types, Omni included. The two are decoupled: `setBuiltInMetricsEnabled` and the env var affect only the built-in Cloud Monitoring sink and have no effect on client metrics; the client-metrics provider affects only the caller-owned sink and has no effect on built-in metrics. Under the hood these are the same client instruments, exported under the distinct `spanner/client` namespace — the difference is the export path, not the metrics. ## Emulator handling Client metrics are not recorded when Spanner is pointed at the emulator. To make that reliable when the emulator is configured programmatically (via `setEmulatorHost(...)`) and not only through the `SPANNER_EMULATOR_HOST` environment variable, this PR broadens emulator detection: `SpannerOptions.isEmulatorEnabled()` now also recognizes the builder-configured emulator host, and the connection-check error message is generalized to describe both the environment-variable and programmatic configuration paths. This gates both the client and Cloud Monitoring metrics sinks off against the emulator using a single detection predicate. Called out explicitly here for reviewer visibility. ## Motivation Client metrics currently export only to Cloud Monitoring, which is unavailable on Spanner Omni. Customers running on Omni (or who standardize on their own observability stack) had no way to receive these metrics. This lets them route the metrics to any OpenTelemetry exporter, independently of the built-in Cloud Monitoring configuration. ## API ```java SdkMeterProviderBuilder meterProviderBuilder = SdkMeterProvider.builder(); SpannerMetrics.configureMeterProviderBuilder(meterProviderBuilder); // ... attach your exporter to meterProviderBuilder ... OpenTelemetry otel = OpenTelemetrySdk.builder().setMeterProvider(meterProviderBuilder.build()).build(); SpannerOptions options = SpannerOptions.newBuilder() .setClientMetricsProvider(new CustomOpenTelemetryMetricsProvider(otel)) .build(); ```
The "java-shared-config downstream (dependencies) / flatten-plugin-check" is currently failing on release PR googleapis#13934 It [passes](https://github.com/googleapis/google-cloud-java/actions/runs/30405348254/job/90429578990) on this PR that updates kokoro/java-storage-expected-flattened-dependencies.txt, accounting for recent version changes (googleapis#13693, googleapis#13799, googleapis#13842, googleapis#13853, googleapis#13854, googleapis#13875, googleapis#13876, googleapis#13877).
- Add CertificateBasedAccess and WorkloadCertificateUtils for SPIFFE and custom certificate loading - Implement RefreshingHttpJsonChannel and ChannelPool mTLS certificate fingerprint tracking and rotation - Enable transparent retries for retryable UnauthenticatedExceptions in ApiResultRetryAlgorithm and AttemptCallable - Add override delegation for getEndpoint, getHttpTransport, and getExecutor to preserve SLF4J MDC logging in Showcase tests
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request prepares the repository for the 1.89.0 release by bumping SNAPSHOT versions to release versions across multiple Java libraries, BOMs, and configuration files. It updates the changelog, release manifest, expected flattened dependencies, and version constants in stub classes. I have no feedback to provide as these are standard automated release updates.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
13 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR introduces robust dynamic mTLS certificate rotation capabilities for
HTTP/JSON and gRPC transport channels, ensuring that certificates can be
rotated in long-lived environments without prematurely severing active, in-
flight RPCs or streams.
🚀 Core Features & Architectural Updates
• Dynamic Certificate Rotation: Implemented RefreshingHttpJsonChannel and
overhauled the gRPC ChannelPool to support dynamic, thread-safe, hot-swapping
of the underlying active transport channels whenever workload certificates
rotate dynamically on the filesystem.
• Preemptive Drop Mitigation: Refactored the internal channel rotation
pipeline (via refreshAll() and refreshSafely()) so that newly formed
connections are seamlessly brought online while preceding active streams are
cleanly drained and gracefully retired. This mitigates GFE connection drop
errors that previously occurred during hard resource refreshes.
• Core Retry Integration: Aligned streaming algorithm Callables and Retry
mechanisms with the dynamic refresh paradigm to ensure transparent retry
policies are respected, avoiding double-wrapped exceptions when traversing
rotated transports.
🔒 System Hardening & Bug Fixes
During the development of these features, several deep-dive reviews were
conducted over the GAX codebase, resulting in the following critical fixes:
• HTTP/JSON Teardown Thread-Safety: Fixed a race condition in
RefreshingHttpJsonChannel.java where shutdown() was calculating state
dynamically from underlying sub-channels without a lock. This allowed a
concurrent refresh() to spawn completely new channels after teardown began,
permanently leaking the channel pool.
• Outstanding RPC Memory Leak (ChannelPool.java): Fixed an uncontrolled
exception escape hatch in ReleasingClientCall.start(). If a pre-existing
cancellation exception was detected, the method aborted forcefully. This
bypassed onClose and never executed entry.release(), leaving the sub-channel
permanently trapped with an outstanding RPC count and preventing graceful
cleanup during rotations.
• Transport Channel Override Drops: Fixed merge() operations in
GrpcCallContext and HttpJsonCallContext that intentionally dropped custom
outer transportChannel references in favor of strict this.transportChannel
defaults. Context overrides now safely propagate custom overrides.
• Cross-Platform Compatibility: Fixed naively concatenated pathing for
certificates (Windows compatibility) and properly escaped JSON strings inside
CertificateBasedAccess.
mTLS Fail-Open Security Fix (CertificateBasedAccess.java):
JSON), the system previously swallowed the I/O exception, failed-open to
a null filepath, and allowed a standard non-mTLS auth connection without
notifying the developer. The system now correctly fails-closed (crashing
startup by throwing an IllegalStateException) upon parsing failure,
preventing unintentional security downgrade rollbacks.
Infinity Timeout Boundary Enforcement (GrpcCallContext & HttpJsonCallContext):
user limits into downstream libraries). However, a logical flaw permitted
bypassing this if a downstream caller submitted an unconstrained/infinite
timeout limit (represented as null), quietly erasing strict prior
deadlines. Override evaluations now properly reject null expansion
boundaries.
🧪 Testing
• Added and updated comprehensive unit-tests reflecting the thread-safety
fixes inside ChannelPoolTest.java and RefreshingHttpJsonChannelTest.java.
• Corrected edge case test configurations to leverage realistic mocked X.509
certificates to properly exercise deep WorkloadCertificateUtils.
getCertificateFingerprint() filesystem caching mechanisms.