Improve the watchable control-plane metrics: watchable_depth is always 0, and subscribe duration buckets are too coarse below 10s

[zhaohuabing]

Two follow-ups from #9773, both about the watchable_* control-plane metrics being hard to read. They are independent and can be done separately.

• 1. watchable_depth is always 0, and the dashboard graphs it

watchable_depth is recorded as len(snapshots) (internal/message/watchutil.go:135), where snapshots is the channel returned by watchable.Map.Subscribe(). That channel is unbuffered — downstream := make(chan Snapshot[K, V]) in the watchable library — so the gauge can only ever be 0.

This is not just dead weight. The shipped Grafana dashboard graphs it, so operators see a flat zero and can reasonably read it as "no backlog" while the control plane is in fact seconds behind. That misreading happened during a real incident investigation.

Removing the metric is not straightforward, because the dashboard also uses it as a template variable source, so the $Runner and $Namespace dropdowns would break and take every filtered panel with them:

charts/gateway-addons-helm/dashboards/envoy-gateway-global.json
  277   "expr": "sum by(runner) (watchable_depth{runner=~\"$Runner\", namespace=\"$Namespace\"})"
  3046  "definition": "label_values(watchable_depth,namespace)"
  3078  "definition": "label_values(watchable_depth,runner)"

Suggested order of work:

• Repoint the depth panel at something meaningful. watchable_debounce_pending (added in performance: add opt-in debounce for resource updates #9773) reports how many updates were merged per flush, but only when debouncing is enabled, so a panel that works in both modes may need a different signal.

• Move the two label_values queries onto a metric that is always present with the same runner/namespace labels — watchable_subscribe_total would do.

• Only then decide whether to deprecate and remove the gauge, or redefine what it measures. Either is a breaking change for anyone with their own dashboards or alerts, so it wants a deprecation note.

• 2. watchable_subscribe_duration_seconds has poor resolution below 10s

The buckets are {0.001, 0.01, 0.1, 1, 5, 10, 30, 60, 120} (internal/message/metrics.go:31). #9773 appended 30, 60, 120 so the tail is no longer hidden, but deliberately did not touch the existing boundaries, since removing them is breaking.

The gaps that remain are 0.1 → 1 (10x) and 1 → 5 (5x). A control plane whose translations take ~2s puts every observation in (1, 5], and because histogram_quantile interpolates within a bucket, p50/p95/p99 all come back somewhere in 1–5s with no discriminating power — the same numbers whether things are healthy or steadily degrading, until they cross 5s. A sub-second control plane has the same problem in (0.1, 1].

This repo already has better-spaced buckets for the same kind of measurement, from the k8s rest client (internal/metrics/restclient/metrics.go:24):

[]float64{0.005, 0.025, 0.1, 0.25, 0.5, 1.0, 2.0, 4.0, 8.0, 15.0, 30.0, 60.0}

Adopting that spacing, extended to 120s, would give roughly 2–2.5x steps throughout. It removes the 0.001, 1 and 5 boundaries, so anything referencing those le values breaks — hence a separate change with a breaking-change note.

[somaz94]

Opened #9860 for item 1 (draft).

It records len(snapshot.Updates) instead of len(subscription). The library documents Snapshot.Updates as the mutations that piled up since the previous snapshot, so its length is the backlog from the last handler pass, and it's there whether or not debouncing is on.

That leaves the dashboard alone. The name and labels don't change, so the label_values template variables keep working and the depth panel starts showing something real instead of needing to be repointed.

One open question in the PR: I used the pre-coalesce count. The coalesced count would measure remaining work rather than arrival, happy to switch if you'd prefer that.