Skip to content

fix(k8s-infra): grant persistentvolumes to otel-agent ClusterRole - #907

Open
haidaraM wants to merge 1 commit into
SigNoz:mainfrom
haidaraM:fix/k8s-infra-agent-pv-rbac
Open

fix(k8s-infra): grant persistentvolumes to otel-agent ClusterRole#907
haidaraM wants to merge 1 commit into
SigNoz:mainfrom
haidaraM:fix/k8s-infra-agent-pv-rbac

Conversation

@haidaraM

@haidaraM haidaraM commented Aug 12, 2026

Copy link
Copy Markdown

What

Add persistentvolumes to the otel-agent ClusterRole rules.

Why

Since #889 (released in k8s-infra 0.17.0), the kubeletstats receiver is configured with k8s_api_config: {auth_type: serviceAccount}. With API access configured, the receiver collects detailed PVC labels: it follows each PersistentVolumeClaim to its bound PersistentVolume to resolve the underlying volume type. When that lookup fails, the receiver skips metric collection for the volume entirely.

The ClusterRole grants persistentvolumeclaims but not persistentvolumes, so the PV lookup is forbidden and all k8s.volume.* metrics for PVC-backed volumes are dropped after upgrading to 0.17.0. The otel-agent logs the following for every PVC-backed volume on every scrape:

Failed to gather additional volume metadata. Skipping metric collection.
failed to set extra labels from metadata: failed to set labels from volume claim:
persistentvolumes "pvc-97589824-bdcb-48ee-8957-ff49e8959bd0" is forbidden:
User "system:serviceaccount:observability:signoz-k8s-infra-otel-agent"
cannot get resource "persistentvolumes" in API group "" at the cluster scope

Ref: the receiver's detailed PVC labels setter calls GetPersistentVolume (kubeletstatsreceiver metadata).

@haidaraM
haidaraM requested a review from a team as a code owner August 12, 2026 10:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant