[Enhancement]: Allow disabling the Kafka Connect API ClusterIP service

[laughingman7743]

Related problem

Every KafkaConnect cluster gets two Services: the already-headless <name>-connect, and <name>-connect-api, which allocates a ClusterIP.

When many KafkaConnect clusters run on a single Kubernetes cluster, those ClusterIPs accumulate. A cluster's Service CIDR is fixed when the cluster is created and generally cannot be resized afterwards, so a busy shared cluster can approach exhaustion. Once the range is full, creation of every new Service and Ingress on that cluster fails with failed to allocate a serviceIP: range is full, which affects all workloads on the cluster, not just Kafka Connect.

There is currently no configuration a user can reach for in that situation: spec.template.apiService is an InternalServiceTemplate, which exposes only metadata, ipFamilyPolicy and ipFamilies.

Suggested solution

Let users opt into creating <name>-connect-api as a headless Service (spec.clusterIP: None) — for example a field on InternalServiceTemplate, or a boolean on the Connect spec.

The Cluster Operator's own reconciliation should be unaffected, because it addresses the REST API by DNS name rather than by ClusterIP: KafkaConnectResources.qualifiedServiceName() returns <name>-connect-api.<namespace>.svc, which KafkaConnectAssemblyOperator uses for connector reconciliation and plugin listing. For a headless Service with a selector, that name resolves to the Connect pod IPs.

One caveat worth noting: a headless name has no A record while no Connect pod is Ready, so clients would get NXDOMAIN rather than a connection error during rollouts. That argues for keeping this opt-in rather than changing the default.

Alternatives

• Consolidating many small Connect clusters into fewer large ones — this conflicts with isolating connectors per team or per source database.
• Managing connectors outside of KafkaConnector CRs so that the API Service is not needed — this gives up the CR-based workflow.

Additional context

Flink's Kubernetes integration may be a useful reference, since it solves the same problem and the implementation is small.

It exposes Headless_ClusterIP as one value of kubernetes.rest-service.exposed.type (KubernetesConfigOptions.ServiceExposedType). HeadlessClusterIPService extends ClusterIPService and overrides nothing but the two Service builders, setting spec.clusterIP: None; the Service name, DNS record and port stay the same. Crucially it does not override getRestEndpoint(), which returns the namespaced DNS name — that is what makes the headless variant transparent to callers, and it mirrors how the Cluster Operator already addresses the Connect API here.

The Flink Kubernetes Operator keeps ClusterIP as the default and only switches when the user sets the option (FlinkConfigBuilder.applyFlinkConfiguration()), i.e. it is strictly opt-in — which seems like the right shape here too.

Checked against main: InternalServiceTemplate currently has only the three fields listed above. Happy to open a PR if the approach sounds acceptable.

[scholzj]

The ClusterIP and Headless services have each different roles and they work differently. The Cluster IP service is used to reach the REST API. The headless service is used to give a DNS names to the Pods. So no, you cannot choose. It might be possible to use the headless API to reach the REST API. But it routes the traffic differently and might cause different behaviours. Also, if there is any flag to disable the ClusterIP service it has to be opt-out and not opt-in for backwards compatibility.

[laughingman7743]

Thanks — and sorry, my wording was misleading. I am not suggesting reusing or repurposing <name>-connect; I agree the two services have distinct roles. The ask is narrower: that <name>-connect-api itself could optionally be created with spec.clusterIP: None, keeping its name, selector and port.

On the routing difference, you are right, and I should correct something I implied. This is not limited to trivial single-worker setups — our own production Connect clusters run multiple replicas, so it would apply to us too.

For the record, as far as I can tell the difference is not which worker gets the request: through the ClusterIP service a request already lands on an arbitrary ready worker. What changes is that the client resolves the name itself, so it can cache an address and keep using a pod that has gone away, and there is no per-connection balancing for a long-lived client.

That seems like exactly the reason to make it an opt-out flag rather than a default, as you say. The operator keeps the safe behaviour, and an administrator who is running out of Service IPs on a shared cluster can take that trade-off knowingly — and it can be documented as such.

If that direction seems acceptable I would be happy to write it up, including a proposal first if you'd want one for a CRD API change. If you'd rather not carry the extra surface at all, that is a fair answer too and I will stop here.

[scholzj]

I'm not sure why we would make the service optionally headless. Why can't the existing headless service be used for the same?

If that direction seems acceptable I would be happy to write it up, including a proposal first if you'd want one for a CRD API change. If you'd rather not carry the extra surface at all, that is a fair answer too and I will stop here.

The issue should go through the triage first. That is where we decide if this is something we want to do or not. (The triage usually happens on the bi-weekly community calls.)

[laughingman7743]

You are right, and I apologise for the confusion I caused. My second comment reframed the request as making <name>-connect-api headless, which was misleading and is not what is needed.

The existing headless service <name>-connect already exposes the REST API port, so it can serve the same purpose. What I actually need is only an opt-out flag that stops the operator from creating the <name>-connect-api ClusterIP service. When the flag is set, the operator would reach the REST API through <name>-connect instead, and status.url would reflect that.

I have updated the issue title to match this. I understand the issue goes through triage first, and I will wait for that outcome.