Skip to content

Add metrics to monitor PVCs stuck in resize - #309

Draft
RadaBDimitrova wants to merge 2 commits into
gardener:masterfrom
RadaBDimitrova:enh/stuck-in-resize-metrics
Draft

Add metrics to monitor PVCs stuck in resize#309
RadaBDimitrova wants to merge 2 commits into
gardener:masterfrom
RadaBDimitrova:enh/stuck-in-resize-metrics

Conversation

@RadaBDimitrova

Copy link
Copy Markdown
Contributor

How to categorize this PR?

/area monitoring
/kind enhancement

What this PR does / why we need it:
Adds a per-PVC Gauge that exposes when an in-flight resize started, giving operators visibility into stuck or long-running resizes. This way you can set up alerts to signal if a resize gets stuck for a long time, by using the new resize_started_timestamp_seconds Gauge labelled by namespace and persistentvolumeclaim) that reports the unix timestamp at which the current in-progress resize started. It is unset for PVCs that are not mid-resize.

The PVC autoscaler sets the gauge whenever a resize is observed in progress - both for PVCs being actively scaled and for PVCs at max capacity whose resize is still in flight by using the recommendation's LastResizeTime and falling back to now when unset. It resets the Gauge at the start of each reconcile sweep (and when there are no PVCAs to reconcile), so series are cleared once a resize completes and no longer report the PVC.

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:
This has been developed on top of #308 in a common effort to improve metrics for the pvc-autoscaler. The first commit will be dropped once #308 gets merged.
/cc @Kostov6 @plkokanov

Release note:

Add a new `resize_started_timestamp_seconds` Gauge reporting, per PVC currently being resized, the unix timestamp at which the in-flight resize started.

@gardener-prow

gardener-prow Bot commented Jul 27, 2026

Copy link
Copy Markdown

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@gardener-prow
gardener-prow Bot requested review from Kostov6 and plkokanov July 27, 2026 12:47
@gardener-prow gardener-prow Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 27, 2026
@gardener-prow

gardener-prow Bot commented Jul 27, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign plkokanov for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@gardener-prow gardener-prow Bot added area/monitoring Monitoring (including availability monitoring and alerting) related kind/enhancement Enhancement, improvement, extension cla: yes Indicates the PR's author has signed the cla-assistant.io CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jul 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/monitoring Monitoring (including availability monitoring and alerting) related cla: yes Indicates the PR's author has signed the cla-assistant.io CLA. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. kind/enhancement Enhancement, improvement, extension size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant