feat: add Prometheus metrics to Grove operator controllers

[AsadShahid04]

Summary

• Adds a new internal/metrics package with five grove_operator_* Prometheus metrics covering reconcile count, latency, in-flight gauge, sub-operation latency, and 409 Conflict rate
• Wraps all three controllers (PodCliqueSet, PodClique, PodCliqueScalingGroup) with ObservedReconciler via Complete() in each RegisterWithManager — zero changes to reconcile logic
• Times reconcile_spec and reconcile_status sub-phases in each controller via StartOperation / done() pairs

Problem

Grove currently has zero custom Prometheus metrics. During large-scale Dynamo inference graph deployments (e.g. 5 services × 10 replicas = 50+ PodClique Pods), operators have no visibility into reconcile duration, sub-operation latency, 409 Conflict rates, or controller saturation — making it impossible to diagnose performance degradation.

Solution

operator/internal/metrics/metrics.go — new package:

• ObservedReconciler wraps any reconcile.Reconciler; records duration, in-flight count, result label (success/error/requeue/requeue_after), and conflict count automatically
• StartOperation(controller, operation) returns a done() closure for timing named sub-operations
• All five metrics are registered on the controller-runtime global registry (sigs.k8s.io/controller-runtime/pkg/metrics) so they appear on the existing /metrics endpoint without any additional configuration

Controller wiring (no logic changes):

• podcliqueset/register.go: Complete(grovemetrics.NewObservedReconciler(controllerName, r))
• podclique/register.go: same
• podcliquescalinggroup/register.go: same
• Each reconciler.go: doneSpec := grovemetrics.StartOperation(..., "reconcile_spec"); ...; doneSpec()

Metrics exposed:

Metric	Type	Labels
grove_operator_reconcile_total	Counter	controller, result
grove_operator_reconcile_duration_seconds	Histogram	controller
grove_operator_in_flight_reconciles	Gauge	controller
grove_operator_operation_duration_seconds	Histogram	controller, operation
grove_operator_conflict_total	Counter	controller

Testing

• go build ./...: passed
• Unit tests (go test ./internal/metrics/... ./internal/controller/...): passed — 6 new tests covering success, error, conflict, requeue, requeue_after, and sub-operation histogram population

Closes #498

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[shayasoolin]

Hi @AsadShahid04 thanks for the PR. I think there's some confusion as while you follow the original issue #498, there is a follow-up GREP issue #499 which is the one with the details and full discussion. There you can see the actual (two) metrics we agreed can be implemented.
Please check that and revisit the PR contents accordingly.

[AsadShahid04]

Thanks for pointing me to PR #499, @shayasoolin — I had based the implementation on the original issue #498 and missed that the GREP proposal had narrowed scope significantly after review.

Pushed a revision (2fef9e2) that reduces to the two metrics agreed in PR #499:

• grove_operator_status_update_conflict_total{controller, target_kind} — recorded at each Status().Update() / Status().Patch() call site when a 409 Conflict occurs, with target_kind set to the Grove resource kind (PodCliqueSet / PodClique / PodCliqueScalingGroup)
• grove_operator_reconcile_errors_total{controller, error_type} — recorded at the Reconcile() exit path, categorized into: conflict, not_found, invalid, transient, server_error, unknown

Removed: ObservedReconciler wrapper, StartOperation / sub-operation histogram, and the three reconcile-level metrics that duplicate controller_runtime_reconcile_total / controller_runtime_reconcile_time_seconds / controller_runtime_active_workers.

[shayasoolin]

Thanks for iterating on this and for narrowing the PR toward the two-metric direction discussed in GREP-498. The overall shape is much closer now: no wrapper, no duplicate reconcile-level metrics, and no sub-operation histogram. I found a few implementation gaps that should be addressed before this lands. Process note: GREP-498 / PR #499 is not approved or merged yet, so there may still be requested changes later if that proposal changes before landing; the items below are based on the current proposal text and discussion, not on a finalized contract.

1. grove_operator_status_update_conflict_total is currently recorded only in the three reconcileStatus methods, but the controllers also write the same Grove status subresources from spec/update-progress/error-recording paths. For example, podcliqueset/reconcilespec.go updates the generation hash and observed generation, podclique/reconcilespec.go patches update progress and observed generation, podcliquescalinggroup/reconcilespec.go does the same for PCSG, and controller/common/reconcileerrorrecorder.go patches LastErrors. Conflicts from these status writes would not be counted, so the metric would under-report the contention signal this proposal is trying to expose. Please record the conflict metric on all PCS/PCLQ/PCSG Status().Update() / Status().Patch() error paths where apierrors.IsConflict(err) is true.

2. grove_operator_reconcile_errors_total is not recorded on all reconcile error exits. Some early returns bypass RecordReconcileError, including initial Get* failures and deletion-flow failures in the reconcilers. Since this is intended to be a reconcile-loop error metric, each returned reconcile error should increment exactly one error_type series. A small helper around the final (ctrl.Result, error) return values in each Reconcile method may make this easier to keep consistent.

3. The error_type label set does not match the current GREP text in PR #499. That proposal currently describes bounded Kubernetes classes: not_found, conflict, already_exists, timeout, forbidden, invalid, rate_limited, and internal. The implementation currently uses transient, server_error, and unknown, and does not classify AlreadyExists or Forbidden. Please align the classifier and tests with the current GREP direction, unless the GREP changes before this implementation lands.