grpc connector errors when using "java-pubsub-group-kafka-connector 1.3.2" after upgrading from CP 7.8.2 to CP 8.1.3

[erctru]

We are getting grpc connector errors when using "java-pubsub-group-kafka-connector 1.3.2" after upgrading from CP 7.8.2 to CP 8.1.3

Environment details

1. java-pubsub-group-kafka-connector 1.3.2
2. OS type and version: Red Hat Enterprise Linux release 8.10
3. Java version: jdk-17.0.2
4. version(s): Confluent Platform 8.1.3 (upgrading from CP 7.8,2)

Steps to reproduce

1. Upgrade from CP 7.8.2 to CP 8.1.3
2. Restart Connect worker.
3. Error shows when the connector status is checked.

Code example

On CP 7.8.2 we used - plugin.path=/apps/osp/kafka/connector/pubsub-1.3.2/target,
On CP 8.1.3 we used - plugin.path=/apps/osp/kafka/connector/pubsub-1.3.2_CP-8.1.3/target,

On CP 7.8.2 this was the grpc jars we used -
/apps/osp/kafka/connector/pubsub-1.3.2/target/:
grpc-netty-1.70.0.jar
grpc-rls-1.70.0.jar
pubsub-group-kafka-connector-1.3.2.jar

On CP 8.1.3 this was the grpc jars we used -
/apps/osp/kafka/connector/pubsub-1.3.2_CP-8.1.3/target/
pubsub-group-kafka-connector-1.3.2.jar
grpc-netty-1.75.0.jar
grpc-rls-1.75.0.jar

Stack trace ERROR

"version":"1.3.2","trace":"java.util.ServiceConfigurationError: io.grpc.LoadBalancerProvider: io.grpc.rls.RlsLoadBalancerProvider not a subtype\n\tat java.base/java.util.ServiceLoader.fail(ServiceLoader.java:593)\n\tat java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1244)\n\tat java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1273)\n\tat java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1309)\n\tat java.base/java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1393)\n\tat io.grpc.ServiceProviders.loadAll(ServiceProviders.java:67)\n\tat io.grpc.LoadBalancerRegistry.getDefaultRegistry(LoadBalancerRegistry.java:102)\n\tat io.grpc.internal.AutoConfiguredLoadBalancerFactory.(AutoConfiguredLoadBalancerFactory.java:51)\n\tat io.grpc.internal.ManagedChannelImpl.(ManagedChannelImpl.java:581)\n\tat io.grpc.internal.ManagedChannelImplBuilder.build(ManagedChannelImplBuilder.java:731)\n\tat io.grpc.ForwardingChannelBuilder2.build(ForwardingChannelBuilder2.java:278)\n\tat com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createSingleChannel(InstantiatingGrpcChannelProvider.java:742)\n\tat com.google.api.gax.grpc.ChannelPool.(ChannelPool.java:106)\n\tat com.google.api.gax.grpc.ChannelPool.create(ChannelPool.java:84)\n\tat com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createChannel(InstantiatingGrpcChannelProvider.java:353)\n\tat com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.getTransportChannel(InstantiatingGrpcChannelProvider.java:346)\n\tat com.google.api.gax.rpc.ClientContext.create(ClientContext.java:230)\n\tat com.google.cloud.pubsub.v1.stub.GrpcSubscriberStub.create(GrpcSubscriberStub.java:287)\n\tat com.google.pubsub.kafka.source.CloudPubSubSourceConnector.verifySubscription(CloudPubSubSourceConnector.java:305)\n\tat com.google.pubsub.kafka.source.CloudPubSubSourceConnector.start(CloudPubSubSourceConnector.java:146)\n\tat org.apache.kafka.connect.runtime.WorkerConnector.doStart(WorkerConnector.java:231)\n\tat org.apache.kafka.connect.runtime.WorkerConnector.start(WorkerConnector.java:260)\n\tat org.apache.kafka.connect.runtime.WorkerConnector.doTransitionTo(WorkerConnector.java:420)\n\tat org.apache.kafka.connect.runtime.WorkerConnector.doTransitionTo(WorkerConnector.java:401)\n\tat org.apache.kafka.connect.runtime.WorkerConnector.doRun(WorkerConnector.java:177)\n\tat org.apache.kafka.connect.runtime.WorkerConnector.run(WorkerConnector.java:137)\n\tat org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$0(Plugins.java:376)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\n"},"tasks":[],"type":"source"}

Any additional information below

Even with 1.3.3 the same error shows up.

Thanks!

[hbartti]

We see the same ERROR in a Strimzi environment after upgrading from Strimzi 1.0.0 to 1.1.0.

[klasvo]

For Strimzi, the issue was probably caused by an updated of a grpc dependency from 1.75.0 to 1.79.0:
strimzi/strimzi-kafka-operator@4c509f7