java-datastore: Error messages from OpenTelemetry

[hwellmann]

Issue Details

In our logs, we see lots of messages

i.o.s.m.e.PeriodicMetricReader$Scheduled:275 - Exporter failed

Environment

• OS Type and Version: cos-121-18867-381-125
• Java Version and JDK Vendor: eclipse-temurin:25.0.3_9-jdk

Dependencies

- Libraries-Bom ([com.google.cloud:libraries-bom](https://central.sonatype.com/artifact/com.google.cloud/libraries-bom)): 26.83.0

Reproducer

Caused by creating a Datastore:

        var builder = DatastoreOptions.newBuilder()
            .setRetrySettings(Gcloud.getDefaultRetrySettings())
            .setProjectId(gcloudProjectId)
            .setDatabaseId(databaseId)
            .setCredentials(Gcloud.DEFAULT_CREDENTIALS.createCredentials());
        var result = builder.build().getService();

Even with

   setOpenTelemetryOptions(
                DatastoreOpenTelemetryOptions.newBuilder()
                    .setTracingEnabled(false)
                    .setMetricsEnabled(false)
                    .setOpenTelemetry(ourOwnInstance)
                    .build());

the issue persists.

It seems the reason is the built-in OpenTelemetry instance which gets created even when the DatastoreOpenTelemetryOptions are turned off.

A rather clumsy workaround is setting an environment variable

DATASTORE_EMULATOR_HOST=https://datastore.googleapis.com 

There should be an option to turn off OpenTelemetry completely, or it should be opt-in and turned off by default.

Logs and Stack Trace

The export failures are initially caused by

"com.google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: One or more TimeSeries could not be written: timeSeries[0] (metric.type="otel.sdk.metric_reader.collection.duration", metric.labels={"client_uid": "0002bd", "otel_component_type": "periodic_metric_reader", "otel_component_name": "periodic_metric_reader/0", "service": "datastore.googleapis.com"}, resource.type="global", resource.labels={"project_id": "nd-staging"}): The metric type must be a URL-formatted string with a domain and non-empty path."