Enhance max capacity metrics

[RadaBDimitrova]

How to categorize this PR?

/area monitoring
/kind enhancement

What this PR does / why we need it:
Improves the max-capacity metrics and consolidates the "max capacity reached" logic:

• Adds a new max_capacity_reached Gauge that reports how many currently targeted PVCs sit at their configured max capacity. Unlike the existing max_capacity_reached_total counter, this is a live snapshot that rises and falls as PVCs enter and leave the max-capacity state. It is reset to 0 when there are no PVCAs to reconcile.
• Moves the "already at max capacity" check into shouldResizePVC, so a PVC at max is detected up front and skipped cleanly (with the MaxCapacityReached warning event and resizing condition) instead of being handled inside resizePVC, where only actually resizable PVCs should be able to enter.
• Fixes the semantics of the max_capacity_reached_total counter so it increments once when a resize of a PVC leads to it reaching max capacity, rather than on every reconcile while the PVC continues to sit at max. Now it only increases if the max capacity is reached or if the max capacity was augmented and was reached again.

Which issue(s) this PR fixes:
Part of #88

Special notes for your reviewer:
go.mod picks up github.com/kylelemons/godebug as an indirect dependency.
/cc @Kostov6 @plkokanov

Release note:

Add a new `max_capacity_reached` Gauge reporting how many targeted PVCs are currently at their configured max capacity. The `max_capacity_reached_total` counter now increments only once per resize-to-max instead of on every reconcile.

[gardener-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign radabdimitrova for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Kostov6]

Initial review + a question:
Should we adapt the e2e test to check for those metrics if they are correctly calculated?
Maybe if it requires a lot of complexity this check is not worth it

[Kostov6]

Where does this come from?

[RadaBDimitrova]

From my understanding the changes in metrics.go that use testutils helpers directly from github.com/prometheus/client_golang/prometheus/testutil (which requires godebug) trigger this change. Since the package transitively needs godebug and became part of the build graph, go mod tidy started recording it explicitly in go.mod with the // indirect marker, even though we already have the direct dependency.

[RadaBDimitrova]

Should we adapt the e2e test to check for those metrics if they are correctly calculated?
Maybe if it requires a lot of complexity this check is not worth it

You're right, it would be too complex - the e2e scripts spin up kind and check actual PVC resize behavior. Asserting metric values there would mean scraping the metrics endpoint from inside the cluster, adding polling/timing flakiness for a value the existing unit tests already cover. Not worth it to add the complexity (and slowing down of the e2e tests) imo.