Cluster metrics show almost no history on autoscaled clusters

[vchepkov]

Describe the bug

The Cluster overview CPU/Memory charts build their PromQL node selector from the
nodes alive at query time, but take the time window from the metrics range
picker. On clusters with high node churn (Karpenter, Cluster Autoscaler, spot)
nearly every node from earlier in the window has been replaced, so its samples
are filtered out even though Prometheus still holds them.

Selecting a 5-day range gives a chart that is empty for ~4 days and only rises
near the right edge — which reads as lost history or an idle cluster. The step
ramp at the edge is just the current nodes coming into existence one by one.

cluster-metrics.injectable.ts snapshots the live node list, then
request-cluster-metrics-by-node-names.injectable.ts joins it into the
node=~"..." selector in the providers. The node filter is "now" while
start/end can be days back.

Reproduction steps

1. Use a cluster that recycles nodes frequently (ours replaces them several
   times a day) with Prometheus retention longer than a node's lifetime (5d).
2. Cluster page > Worker > CPU, metrics range 5 days.
3. Data appears only for roughly the age of the oldest surviving node.

Same 6-day window queried directly, filtered to the currently-live nodes as
Freelens does, vs. unfiltered:

sum(rate(node_cpu_seconds_total{node=~"<live nodes>", mode=~"user|system"}[5m]))
  -> 53 of 124 hourly points, meaningful only for the last ~18h
sum(rate(node_cpu_seconds_total{mode=~"user|system"}[5m]))
  -> 124 of 124, full window

475 distinct nodes appear in that window; 13 are alive now, the oldest 19h old.
Retention is healthy (prometheus_tsdb_size_retentions_total 0, no gaps in
count(up)), so this is purely the selector.

Expected behavior

A 5-day chart should show 5 days, including terminated nodes — the panel
aggregates with sum() over the cluster, so their past contribution is part of
the answer.

Expressing the master/worker split in PromQL (joining node label metrics) rather
than as a snapshot of names would make the filter evaluate per timestamp, which
is what the time axis already implies. Dropping the filter for the cluster-wide
aggregate is simpler but loses the split.

Screenshots

metrics exists for 5 days

Freelens version

1.10.3

Operating system type and version

macOS 26.5.2

Installation method

DMG

Kubernetes cluster version used with Freelens

1.35 (EKS)

Logs

No response

Kubeconfig

No response

Additional context

In-cluster prometheus-community/prometheus chart (Prometheus v3.12.0), queried
directly — no Grafana in the path.

The time-range picker from #1962 is what surfaced this; before long ranges were
selectable the live-node snapshot was close enough to hide it. Probably affects
upstream Lens too, unverified.

[waterWang]

I have submitted a PR (#2415) that addresses this issue. The fix changes the cluster overview metrics to not filter by currently-alive node names, instead using direct metric name matching so that historical data from replaced nodes is preserved in the chart.

Approach: Pass an empty node list from the cluster overview (indicating "all nodes, no filter"), and modify all 4 PromQL providers (lens, helm, operator, stacklight) to omit the node=~"..." selector when no node list is provided. The per-node detail view still uses the node filter as before.

Please let me know if you have any feedback!