[kube-prometheus-stack] Built-in ServiceMonitors use deprecated file-based credentials, breaking under deny-by-default collectors

[filippobuletto]

Describe the bug a clear and concise description of what the bug is.

The chart's built-in Kubernetes component ServiceMonitors authenticate with file-based credentials:

bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
tlsConfig:
  caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

Both fields are rejected by consumers that block arbitrary filesystem access from ServiceMonitors - Prometheus objects with spec.arbitraryFSAccessThroughSMs.deny: true, and, since v1.19.0, Grafana Alloy's prometheus.operator.servicemonitors component by default. In both cases the whole ServiceMonitor is dropped, so all control-plane targets disappear and the kubernetes-system-* alerts start firing.

For four of the seven components - kubelet, kube-api-server, kube-controller-manager and kube-scheduler - the paths are hardcoded in the templates, so there is no way to change them without disabling the ServiceMonitor and re-creating it via prometheus.additionalServiceMonitors. core-dns, kube-etcd and kube-proxy do read bearerTokenFile from values (kube-etcd also caFile/certFile/keyFile), but blanking those only removes the field - it leaves the endpoint unauthenticated rather than switching it to a secret-based credential.

What's your helm version?

version.BuildInfo{Version:"v4.2.4", GitCommit:"3900f434fd3ef2b84065dc04508df48f288dba00", GitTreeState:"clean", GoVersion:"go1.26.5", KubeClientVersion:"v1.36"}

What's your kubectl version?

Client Version: v1.37.0

Which chart?

kube-prometheus-stack

What's the chart version?

88.6.2

What happened?

With arbitraryFSAccessThroughSMs.deny: true, the Prometheus Operator rejects the entire ServiceMonitor object on the first offending endpoint, so all control-plane targets are lost.

Grafana Alloy ≥ v1.19.0 behaves the same way in effect. It emits one warning per offending endpoint:

level=warn msg="serviceMonitor endpoint references an arbitrary file from Alloy's filesystem" component_path=xxx component_id=xxx kind=serviceMonitor namespace=prometheus name=prometheus-kubelet endpoint=2 field=bearerTokenFile mitigation="remove the file reference or set allow_arbitrary_file_access to true to opt out"

but addServiceMonitor then breaks out of the endpoint loop on the first generation error and returns without applying any of the endpoints it had already collected, so the whole ServiceMonitor is discarded too.

Affected ServiceMonitors, verified against 88.6.2 with default values:

Component	bearerTokenFile	caFile	Source
kubelet	3	3	hardcoded in kube-prometheus-stack.kubelet.authConfig
kube-api-server	1	1	hardcoded
core-dns	1	0	values
kube-etcd	1	0	values
kube-controller-manager	1	1	hardcoded
kube-scheduler	1	1	hardcoded
kube-proxy	1	0	values

kubelet accounts for three of each because kubelet, cAdvisor and probes all default to true and each pulls in the same auth helper (resource defaults to false). kube-controller-manager and kube-scheduler already emit caFile at defaults: both default to https: null, which the insecureScrape helper resolves to secure on Kubernetes ≥ 1.22 and ≥ 1.23 respectively. core-dns has no tlsConfig at all, kube-proxy defaults to https: false, and kube-etcd defaults to scheme: http with an empty caFile so those three trip the guard on bearerTokenFile alone unless etcd is switched to HTTPS with a CA path set.

(kubeDns.serviceMonitor.bearerTokenFile exists as well, but that component is disabled by default.)

Because Alloy v1.19.0 (2026-08-21) made deny-by-default the behaviour of prometheus.operator.servicemonitors, a user who scrapes this chart's ServiceMonitors via Alloy silently loses all control-plane targets on an Alloy minor upgrade, with no change on the chart side, unless they opt back out of the guard.

What you expected to happen?

The credential source for these endpoints should be configurable, so secret-based auth can be used instead of file paths. For example, per-component:

kubelet:
  serviceMonitor:
    authorization:
      type: Bearer
      credentials:
        name: prometheus-sa-token
        key: token
    tlsConfig:
      ca:
        configMap:
          name: kube-root-ca.crt
          key: ca.crt

When set, the template would emit authorization / tlsConfig.ca instead of bearerTokenFile / tlsConfig.caFile. Defaults would stay as they are today, so this is backwards compatible. These are also the replacements Alloy itself points at (bearerTokenSecret or authorization, tlsConfig.ca, tlsConfig.cert, tlsConfig.keySecret).

A global toggle (e.g. kubernetesServiceMonitors.authorization) applied to all built-in ServiceMonitors would also work and would avoid repeating the same block seven times.

Separately, bearerTokenFile on a ServiceMonitor endpoint is deprecated in the monitoring.coreos.com/v1 API, the field is annotated Deprecated: use authorization instead, so these templates rely on a deprecated field regardless of the guard.

How to reproduce it?

The file references can be confirmed from the rendered templates alone:

$ for t in kubelet kube-api-server core-dns kube-etcd \
         kube-controller-manager kube-scheduler kube-proxy; do
  printf '%-26s %s\n' "$t" \
    "$(helm template release-name prometheus-community/kube-prometheus-stack \
         --version 88.6.2 \
         --show-only "templates/exporters/$t/servicemonitor.yaml" 2>/dev/null \
       | grep -c bearerTokenFile)"
done
kubelet                    3
kube-api-server            1
core-dns                   1
kube-etcd                  1
kube-controller-manager    1
kube-scheduler             1
kube-proxy                 1
 
$ for t in kubelet kube-api-server core-dns kube-etcd \
         kube-controller-manager kube-scheduler kube-proxy; do
  printf '%-26s %s\n' "$t" \
    "$(helm template release-name prometheus-community/kube-prometheus-stack \
         --version 88.6.2 \
         --show-only "templates/exporters/$t/servicemonitor.yaml" 2>/dev/null \
       | grep -c caFile)"
done
kubelet                    3
kube-api-server            1
core-dns                   0
kube-etcd                  0
kube-controller-manager    1
kube-scheduler             1
kube-proxy                 0

To see the resulting target loss:

1. Install chart 88.6.2 with default values.
2. Have the ServiceMonitors scraped either by a Prometheus object with spec.arbitraryFSAccessThroughSMs.deny: true, or by Grafana Alloy ≥ v1.19.0 with allow_arbitrary_file_access unset.
3. Check the targets page: the control-plane targets are gone, and the kubernetes-system-* alerts fire. The Operator logs skipping servicemonitor; Alloy logs the per-endpoint warning above followed by error generating scrapeconfig from serviceMonitor.

Enter the changed values of values.yaml?

NONE

Enter the command that you execute and failing/misfunctioning.

helm template release-name prometheus-community/kube-prometheus-stack --version 88.6.2

Anything else we need to know?

This also matters outside of self-managed Prometheus instances: OpenShift's openshift-user-workload-monitoring Prometheus has deny: true hardcoded in the cluster-monitoring-operator asset that manages it, so users cannot change it.

Alternatives considered:

• Setting kubernetesServiceMonitors.enabled: false and re-declaring all control-plane ServiceMonitors under prometheus.additionalServiceMonitors. This works, but means maintaining a copy of seven templates, including the relabelings the chart's shipped dashboards and rules depend on, outside the chart.
• Blanking bearerTokenFile for core-dns, kube-etcd and kube-proxy via values. Possible today, but it removes authentication rather than replacing it, and does nothing for the four components with hardcoded paths.
• Leaving arbitraryFSAccessThroughSMs.deny unset, or setting allow_arbitrary_file_access = true in Alloy. Not viable where the guard is required by policy, or where the Prometheus object is managed by the platform.

Versions and references:

• Chart version: reproduced on 88.6.2
• Prometheus Operator API: ArbitraryFSAccessThroughSMsConfig, added in prometheus-operator v0.34.0 (2019-10-31)
• API reference: https://prometheus-operator.dev/docs/api-reference/api/
• Grafana Alloy v1.19.0 breaking change (deny-by-default): https://github.com/grafana/alloy/releases/tag/v1.19.0

[jkroepke]

Feel free to setup an PR.