Skip to content

[Feature] Native Resource Exclusions for AKS Recommended Prometheus Alerts (Label-Based Opt-Out) #5915

Description

Is your feature request related to a problem? Please describe.
Yes. Azure recommended AKS Prometheus alerts currently do not provide a built-in, first-class exclusion behavior for specific Kubernetes resources.
In real operations, teams often need to suppress known-noisy workloads, non-production namespaces, or specific PVC scenarios without disabling recommended alerts globally or forking all alert rules.

Today, to get this behavior, operators must:

  • Clone or override recommended alert rule groups
  • Add custom PromQL exclusion clauses
  • Ensure kube-state-metrics label collection is configured correctly
  • Maintain these customizations over time as Microsoft updates baseline recommended rules

This creates drift, adds operational risk, and makes recommended alerts harder to adopt at scale.

Describe the solution you'd like
Add native exclusion support to Azure recommended AKS alerts, so customers can safely opt out specific resources without replacing the full rule set.

Requested behavior:

  • Support a standard exclusion label convention for Kubernetes objects, for example:
    • key: excluded_from_alerts
    • value: true
  • Apply this across relevant recommended alert scopes where dimensions exist:
    • pod-scoped alerts
    • namespace-scoped alerts
    • PVC-scoped alerts
  • Expose exclusion behavior as a supported configuration option in Azure:
    • AKS/Monitor onboarding settings
    • CLI and ARM/Bicep
    • Portal experience
  • Ensure Microsoft-managed recommended rule updates preserve this exclusion behavior automatically.
  • Document exactly which recommended alerts support exclusions and at which scope.

Describe alternatives you've considered

  • Customizing Prometheus rule groups manually with PromQL unless clauses.
  • Deploying and maintaining custom rule templates instead of Microsoft recommended defaults.
  • Disabling noisy alerts entirely, which reduces coverage and signal quality.
  • Namespace-level alert routing/suppression downstream in action handling systems rather than at rule-evaluation level.

All alternatives increase maintenance burden and can break when recommended rule baselines evolve.

Additional context
This is especially valuable for enterprise AKS environments with:

  • Shared clusters (prod and non-prod workloads)
  • Temporary test workloads
  • Platform teams managing many clusters with centralized standards

A native exclusion capability would improve adoption of recommended alerts, reduce customization drift, and preserve signal quality while still allowing practical suppression for known exceptions.

Metadata

Metadata

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions