Summary
DatastoreOpenTelemetryOptions exposes an exportBuiltinMetricsToGoogleCloudMonitoring flag that defaults to false. The clear intent is to allow users to opt out of the built-in Cloud Monitoring metrics export. However, BuiltInDatastoreMetricsProvider.createOpenTelemetry() never reads this flag, so the PeriodicMetricReader is unconditionally created for any application using real credentials.
The result is an unavoidable "Exporter failed" warning logged every 60 seconds by io.opentelemetry.sdk.metrics.export.PeriodicMetricReader for any application that either:
- runs locally with ADC credentials that lack roles/monitoring.metricWriter, or
- intentionally does not want to export Datastore metrics to Cloud Monitoring.
Affected version
google-cloud-datastore:2.41.0
Steps to reproduce
- Use google-cloud-datastore:2.41.0 with real ADC credentials (e.g. a service account without roles/monitoring.metricWriter)
- Do not set DATASTORE_EMULATOR_HOST or use NoCredentials
- Start the application
Observed: PeriodicMetricReader is created and logs "Exporter failed" every ~60 seconds to monitoring.googleapis.com.
Expected: Since exportBuiltinMetricsToGoogleCloudMonitoring defaults to false, no PeriodicMetricReader should be created and no export should be attempted.
Root cause
In DatastoreImpl, the constructor unconditionally calls:
BuiltInDatastoreMetricsProvider.INSTANCE.createOpenTelemetry(options)
Inside BuiltInDatastoreMetricsProvider.createOpenTelemetry(), the only conditions that return OpenTelemetry.noop() are:
- DATASTORE_EMULATOR_HOST system property / env var is set
- credentials are NoCredentials
- MetricServiceClient creation throws IOException
The flag options.getOpenTelemetryOptions().isExportBuiltinMetricsToGoogleCloudMonitoring() is never checked. The fix should be straightforward:
// BuiltInDatastoreMetricsProvider.java
public OpenTelemetry createOpenTelemetry(DatastoreOptions options) {
// NEW: respect the opt-out flag
if (!options.getOpenTelemetryOptions().isExportBuiltinMetricsToGoogleCloudMonitoring()) {
return OpenTelemetry.noop();
}
// ... existing logic
} Workaround
Until this is fixed, the only available workaround is to set the DATASTORE_EMULATOR_HOST system property to a non-empty dummy value before the DatastoreImpl is instantiated, then override spring.cloud.gcp.datastore.host (or equivalent) with the real endpoint to prevent the gRPC connection from being misdirected:
// In main() before application start
System.setProperty("DATASTORE_EMULATOR_HOST", "metrics-disabled"); This exploits the emulator detection path as a side effect, which is fragile and unintuitive.
Impact
This is particularly painful for local development and CI environments where ADC credentials exist but Cloud Monitoring IAM permissions are absent.
The warning fires every minute, clutters logs, and there is no documented, supported way to suppress it without resorting to the emulator side-channel.
Summary
DatastoreOpenTelemetryOptions exposes an exportBuiltinMetricsToGoogleCloudMonitoring flag that defaults to false. The clear intent is to allow users to opt out of the built-in Cloud Monitoring metrics export. However, BuiltInDatastoreMetricsProvider.createOpenTelemetry() never reads this flag, so the PeriodicMetricReader is unconditionally created for any application using real credentials.
The result is an unavoidable "Exporter failed" warning logged every 60 seconds by io.opentelemetry.sdk.metrics.export.PeriodicMetricReader for any application that either:
Affected version
google-cloud-datastore:2.41.0
Steps to reproduce
Observed: PeriodicMetricReader is created and logs "Exporter failed" every ~60 seconds to monitoring.googleapis.com.
Expected: Since exportBuiltinMetricsToGoogleCloudMonitoring defaults to false, no PeriodicMetricReader should be created and no export should be attempted.
Root cause
In DatastoreImpl, the constructor unconditionally calls:
BuiltInDatastoreMetricsProvider.INSTANCE.createOpenTelemetry(options)
Inside BuiltInDatastoreMetricsProvider.createOpenTelemetry(), the only conditions that return OpenTelemetry.noop() are:
The flag options.getOpenTelemetryOptions().isExportBuiltinMetricsToGoogleCloudMonitoring() is never checked. The fix should be straightforward:
Workaround
Until this is fixed, the only available workaround is to set the DATASTORE_EMULATOR_HOST system property to a non-empty dummy value before the DatastoreImpl is instantiated, then override spring.cloud.gcp.datastore.host (or equivalent) with the real endpoint to prevent the gRPC connection from being misdirected:
This exploits the emulator detection path as a side effect, which is fragile and unintuitive.
Impact
This is particularly painful for local development and CI environments where ADC credentials exist but Cloud Monitoring IAM permissions are absent.
The warning fires every minute, clutters logs, and there is no documented, supported way to suppress it without resorting to the emulator side-channel.