[kube-prometheus-stack] Every kubelet is scraped twice on dual-stack clusters since chart v88 (prometheus-operator v0.93.0)

[bootc]

Describe the bug

Since kube-prometheus-stack v88.0.0 (prometheus-operator v0.92.1 → v0.93.0), every kubelet on a dual-stack cluster is scraped twice — once via its IPv4 InternalIP and once via its IPv6 one. With the chart's defaults there is nothing filtering the duplicate out, so kubelet/cAdvisor ingestion doubles and the bundled cAdvisor recording rules silently double-count.

What's your helm version?

v4.2.3+g43e8b7f

What's your kubectl version?

Client v1.36.3, Server v1.36.2+rke2r1 (RKE2, dual-stack)

Which chart?

kube-prometheus-stack

What's the chart version?

88.0.1 (regression introduced in 88.0.0; 87.21.0 is unaffected)

What happened?

prometheus-operator v0.93.0 includes prometheus-operator#8682 ("Create IPv6 EndpointSlice for the kubelet Service on dual-stack clusters", fixing #8681). That PR changed nodeAddress() → nodeAddresses() in pkg/kubelet/controller.go, so getNodeAddresses() now returns every address of the chosen family group per node rather than just the first.

That result feeds both sync paths (controller.go#L400-L427):

addresses, errs := c.getNodeAddresses(nodes)
...
if c.manageEndpoints {
    c.syncEndpoints(ctx, addresses)          // one flat EndpointSubset — no family separation
}
if c.manageEndpointSlice {
    c.syncEndpointSlice(ctx, svc, addresses) // splits per family, so Prometheus can dedupe
}

syncEndpointSlice() deliberately separates addresses by family (if a.ipv4 != (epsl[i].AddressType == discoveryv1.AddressTypeIPv4)), because Prometheus's role: endpointslice skips slices whose addressType doesn't match svc.Spec.IPFamilies[0]. That's the dedup the PR was written around.

syncEndpoints() has no equivalent — it puts every address into a single EndpointSubset, and Prometheus's role: endpoints has no address-family filter to compensate.

The chart defaults put you squarely on the path with no dedup:

prometheusOperator:
  kubeletEndpointsEnabled: true       # --kubelet-endpoints=true
  kubeletEndpointSliceEnabled: false  # --kubelet-endpointslice=false
prometheus:
  prometheusSpec:
    serviceDiscoveryRole: ""          # => Endpoints

So on an 8-node dual-stack cluster:

$ kubectl -n kube-system get endpoints kube-prometheus-stack-kubelet -o json | jq '.subsets[0].addresses | length'
16      # 8 IPv4 + 8 IPv6 in one subset — was 8 before the upgrade

and Prometheus turns each into a target. Measured across the upgrade on our cluster:

	87.21.0	88.0.1
kubelet targets	24 (8 nodes × 3 paths)	48
kubelet samples/scrape	132,407	267,104
total samples/scrape	668,568	801,377 (+20%)
2h TSDB block	~250 MB	~325 MB

Two knock-on effects, both silent:

1. The bundled cAdvisor recording rules double-count. node_namespace_pod_container:* aggregates by (cluster, namespace, pod, container), so two instance series per node are summed together. Cluster container working set read 160.5 GB against a true 78.7 GB — a clean 2×. Every dashboard built on those rules was wrong, with nothing to indicate it.

2. It can fill the Prometheus volume. Ours were sized with ~6% headroom for the old rate. A 20% ingestion increase filled both replicas 24h after the upgrade, and Prometheus then deadlocks: head compaction needs free space to write a block, and until that succeeds the WAL can't be truncated, so it never recovers on its own.

err="write to WAL: log samples: write /prometheus/wal/00099904: no space left on device"
err="compact head: persist head block: mkdir /prometheus/01KZ....tmp-for-creation: no space left on device"

Worth flagging for anyone hitting this: rule evaluation keeps running against an empty TSDB, so every absent()-based alert fires at once (KubeAPIDown, CoreDNSDown, Watchdog aside — all of them) while no up == 0 alert fires at all, and /api/v1/targets shows every target health: up because the scrape succeeds and it's the commit that fails. It reads like a total cluster outage when nothing is actually wrong.

What you expected to happen?

Each kubelet scraped once, as before v88 — an ingestion increase of this size shouldn't arrive as a silent side effect of a patch-level operator bump, and the recording rules shipped by the chart shouldn't double-count on a supported cluster topology.

How to reproduce it?

1. A dual-stack cluster whose nodes have both IPv4 and IPv6 InternalIP addresses.
2. Install kube-prometheus-stack 88.x with default prometheusOperator and kubelet values.
3. kubectl -n kube-system get endpoints kube-prometheus-stack-kubelet → 2 addresses per node.
4. Prometheus → Targets → serviceMonitor/<ns>/kube-prometheus-stack-kubelet/0 shows 2 entries per node.
5. count(up{job="kubelet"}) returns 2× the node count × 3 paths.

Enter the changed values of values.yaml?

NONE relevant — reproduced with chart defaults for prometheusOperator.* and kubelet.*.

Anything else we need to know?

Workaround (verified in production — restores 24 targets, halves kubelet ingestion, corrects the recording rules, and applies by config reload with no pod restart). Note that overriding these keys replaces the chart defaults, so the metrics_path relabeling has to be carried over:

kubelet:
  serviceMonitor:
    relabelings: &kubeletRelabelings
    - action: drop
      sourceLabels: [__address__]
      regex: \d+\.\d+\.\d+\.\d+:\d+   # drops IPv4, keeping IPv6; invert with \[.+\]:\d+
    - action: replace
      sourceLabels: [__metrics_path__]
      targetLabel: metrics_path
    cAdvisorRelabelings: *kubeletRelabelings
    probesRelabelings: *kubeletRelabelings

Whether the operator should publish one address per node or all of them is arguable — a complete Endpoints object listing every address the kubelet answers on isn't unreasonable in itself. But "scrape each node once" is scrape policy, and that lives in this chart's kubelet ServiceMonitor, so it seems like the right place to resolve it. Two options:

• Filter in the shipped ServiceMonitor, roughly as above, keyed on the cluster's primary family. Small and immediate, but it needs a sensible default for single-stack clusters of either family.
• Default to the EndpointSlice path — prometheusOperator.kubeletEndpointSliceEnabled: true, kubeletEndpointsEnabled: false, prometheus.prometheusSpec.serviceDiscoveryRole: EndpointSlice. Prometheus's addressType vs IPFamilies[0] filter then dedupes natively, which is what prometheus-operator#8682 was designed around. It also moves off the v1.Endpoints API, which is deprecated as of Kubernetes 1.33 (kubectl warns on every read). Bigger change, but it's the direction of travel and doesn't need a family heuristic.

Happy to open a PR for either if you have a preference.

Related: prometheus/prometheus#17193 (duplicate scraping with role: endpointslice and ipFamilyPolicy: PreferDualStack) covers the neighbouring case in Prometheus SD.

One smaller thing on the operator side, which I can file separately if it's useful: syncEndpoints() only sets endpointslice.kubernetes.io/skip-mirror when manageEndpointSlice is true, so on chart defaults the kubelet Endpoints object is left unannotated and the EndpointSlice mirroring controller derives per-family slices from it. Harmless as things stand — nothing reads them — but it means both representations exist while only one is authoritative.

[bootc]

It looks like this may be fixed by prometheus-operator/prometheus-operator#8739; it sounds likely, but I haven't tested it.